Broken

It’s all broken.

The TV was destroyed by a raging four year old and Wii remote.

My phone isn’t receiving iMessages over cell.

The lone PC in the house was (on purpose) upgraded to Windows 10.

My RAID DAS is flashing red just two days after I copied all of my data to it.

Bacteria has invaded my sinuses and lungs.

On the plus side, I’ve got a wonderful wife and family. I’ve got a job to go to tomorrow. I got a sweet new Santa Cruz patch from Ollie Crate and today I installed Grafana and InfluxDB … so I guess it’s not all bad.

Share Button

Change Everything

Change everything.

That’s what I seem to do. On this day, one year ago, Mary and I were preparing for the arrival of our baby. I had taken one week off of work and would go back to a new job once that leave was over.

Everything changed.

I was done having kids. Until this one. I quit a job I had been at for over eleven years. It was scary. It was exciting. These were the best decisions I ever made.

This year I’m doing something similar. Mary and I just married on the 17th and on the 23rd I’ll be leaving the best job I’ve had. It is scary. It is exciting. These may be the best decisions I ever make.

After eleven years at one job things become boring. Some of the job was so rote that I was starting to make stupid mistakes. I wasn’t performing as well as I knew I could or should. I’m also at the age where technical proficiency is coming to its last gasp. I knew if I didn’t start to push myself, I would fall behind.

Taking the job at the National Weather Service pushed me. I knew going in that I didn’t have some of the required skills. I’d never worked on a high traffic website. I’d never worked with some of the networking gear we used. More than that, I’d never worked at such a large organization.

The department I was in was mercifully small and it felt like a small dot-com. The learning curve was dramatic as. I was hired in as the Sr. System Administrator and the previous Sys Admin was gone except for on-call. There was Jr. Admin who was difficult to work with. This was a puzzle.

I had around two hours total with the previous Admin. He sketched out the main pieces. Networks. What servers did what. The basics.

Those first few months were some of the hardest but most rewarding months of my life. Put into the sink-or-swim position, I knew I had to swim.

Learning the networks. Learning to tune Apache and Memcached. Learning to work with developers using agile methods. These were phenomenal experiences.

After a few months there was some reorganization. The little dot-com was swallowed up by a larger piece of the organization. The fast, light, agile methods were replaced with meetings, forms, and authorizations. Release cycles were delayed. Changes were carefully worded so as not to fall under certain controls. It was an exercise in frustration

Tools that should make the life of a Sys Admin easier were implemented in a way that didn’t fit with a modern website. So I found something else and quit.

I’m sad. The mission was noble. The people were great. But I was feeling ineffective. Time was spent mulling over how to get changes made, not making changes.

I’m moving on. I’ll be facing a whole new slew of technologies that I’m rusty on. There will be challenges. There will be legacy systems. There will be resistance.

There will also be learning. Staying relevant in a field that changes fast. New relationships. New contacts.

My last round of “change everything” was driven by a new life. A baby that I needed to support. This change everything is driven my a completion of that family. Marriage, kids, step-kids. All driving me forward. Without them I wouldn’t have made this move. I’ll be forever thankful.

Share Button

Quick Access to Forecast on Weather.Gov

You can quickly get your forecast by going to weather.gov/your zip code. For Blue Springs, I go to http://www.weather.gov/64015 and it takes me right to the forecast page.

Another nicety, forecast.weather.gov uses adaptive design so the page elements arrange themselves to fit your device. The “mobile version” is the desktop version, the things are just rearranged to fit better.

Share Button

Installing Nvidia Drivers on RHEL or CentOS 7

Most users of Nvidia graphics cards prefer to use the drivers provided by Nvidia. These more fully support the capabilities of the card when compared to the nouveau driver that is included with the distribution. These are the steps to install the Nvidia driver and disable the nouveau driver.

  • yum -y update
  • yum -y groupinstall “GNOME Desktop” “Development Tools”
  • yum -y install kernel-devel
  • Edit /etc/default/grub. Add “rd.driver.blacklist=nouveau nouveau.modeset=0” to “GRUB_CMDLINE_LINUX”
  • grub2-mkconfig -o /boot/grub2/grub.cfg
  • List available kernels: awk -F\’ ‘$1==”menuentry ” {print $2}’ /etc/grub2.cfg
  • Use grub2-set-default to set the updated kernel to the default. The kernels are enumerated starting at 0
  • Reboot
  • Download the appropriate Nvidia driver for your card
  • Switch from graphical to text mode: systemctl isolate multi-user.target
  • Run the Nvidia driver installer
  • Remove the nouveau driver: yum remove xorg-x11-drv-nouveau
  • Backup old the old initramfs: mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r)-nouveau.img
  • Create new initramfs: dracut /boot/initramfs-$(uname -r).img $(uname -r)
  • Reboot
Share Button

Why I Hate the Linux Frame Buffer

This is probably pretty petty, and it’s easily fixable, but damn … it’s annoyingly slow at scrolling text.

Share Button

All the FLOPs

19 TFLOPs
19 TFLOPs

When I get asked what I do (well, I should really say I’m a musician or a father or something like that) I say I’m in computers … then the questions start. Then I have to explain I don’t work with computers like your laptop, I work on high performance computing (HPC) clusters. HPC clusters are what scientists use to run HUGE problems on.

The image above is the output of some monitoring and testing of a system I just installed and the performance that it’s achieving. This particular system has 94 distinct servers that contain a total of 5,888 GB of RAM and 1128 processors all connected with a network running at 40 Gbps (that would be like 40 connections from Google Fiber all at once). The performance is measure in floating point operations per second (FLOPs). This system is running at 19 TFLOPs or 19,000 GFLOPs or 19,000,000,000,000 distinct operations on a floating point number PER SECOND. For comparison, my new MacBook Air will do around .0020 TFLOPs.

Share Button

RPMS for TORQUE with Nvidia GPU Support

Getting TORQUE built into RPMs with GPU support was considerably more frustrating than I expected. I’m really not a fan of TORQUE as I seem to often run in to silly problems or serious limitations since pbs_sched is so simplistic that it’s really not the best fit for most users … but I still have to support it, so here goes.

First we need to install CUDA. Thankfully, Nvidia has added a yum repo so this whole process has gotten a little bit easier. The Getting Started Guide has all of the info, but it’s a bit to wade through since they tackle multiple distros. The basic process is to enable the EPEL repository, enable the Nvidia repository (install the appropriate RPM from the CUDA Downloads page), then install the cuda and gpu-deployment-kit packages with yum.

yum -y install cuda gpu-deployment-kit

Download the source for TORQUE from Cluster Resources/Adaptive Computing. I used version 4.2.7 and all of the examples will reference this.

Untar the TORQUE source and run configure with a few options. The annoying option is the --with-default-server options since omitting this makes the clients connect only the localhost instead of the the actual pbs_server process. No amount of config file changes or environment settings change this behavior.

./configure --with-default-server=head.cluster --enable-nvidia-gpus --with-nvml-lib=/usr/lib64/nvidia --with-nvml-include=/usr/include/nvidia/gdk

Now, you would think that would add all of the correct options and building would just go smoothly. NOPE! None of the GPU stuff gets added to the torque.spec file. Fancy! So edit torque.spec and look for the %configure section. It will look like:

%configure --includedir=%{_includedir}/%{name} --with-default-server=%{torque_server} \
--with-server-home=%{torque_home} %{ac_with_debug} %{ac_with_libcpuset} \
--with-sendmail=%{sendmail_path} %{ac_with_numa} %{ac_with_memacct} %{ac_with_top} \
--disable-dependency-tracking %{ac_with_gui} %{ac_with_scp} %{ac_with_syslog} \
--disable-gcc-warnings %{ac_with_munge} %{ac_with_pam} %{ac_with_drmaa} \
--disable-qsub-keep-override %{ac_with_blcr} %{ac_with_cpuset} %{ac_with_spool} %{?acflags}
%{__make} %{?_smp_mflags} %{?mflags}

Change this to:

%configure --includedir=%{_includedir}/%{name} --with-default-server=%{torque_server} \
--enable-nvidia-gpus --with-nvml-lib=/usr/lib64/nvidia --with-nvml-include=/usr/include/nvidia/gdk \
--with-server-home=%{torque_home} %{ac_with_debug} %{ac_with_libcpuset} \
--with-sendmail=%{sendmail_path} %{ac_with_numa} %{ac_with_memacct} %{ac_with_top} \
--disable-dependency-tracking %{ac_with_gui} %{ac_with_scp} %{ac_with_syslog} \
--disable-gcc-warnings %{ac_with_munge} %{ac_with_pam} %{ac_with_drmaa} \
--disable-qsub-keep-override %{ac_with_blcr} %{ac_with_cpuset} %{ac_with_spool} %{?acflags}
%{__make} %{?_smp_mflags} %{?mflags}

Now we can run make -j then make rpm and your RPMs will be joyfully created in ~/rpmbuild/RPMS/x86_64.

Share Button

5 years, 2 batteries, Not Bad

Service Battery. Damn.
Service Battery. Damn.

This 2008 Macbook is just about done with its second battery. Damn.

 

Share Button

Focus for the Lazy or How I Got to Inbox 0

nothing to see here
nothing to see here

I am lazy. I was born lazy and every action I take is a true internal struggle to actually get myself in to action. There’s always something more immediately rewarding that I could be doing (hello Reddit, Facebook, booze …).

Like many I confuse busy with productive. I use to have everything notify me all the time. Email, IM, text, iTunes changed a song, a website updated … It was too much. I was trying so hard to be busy that I just chased notifications around in circles.

My first step to getting to 0 was to stop the notifications. Not all of them, I keep text messages and some email accounts but all the other ones are gone and I don’t miss them one bit. This is a critical step in getting to 0 because this is part of retraining yourself to prioritize your items. I know a text is either someone close, like a friend or family, or it’s something important enough to warrant a text from a co-worker like a power circuit went down at the office.

To the inbox. With mailing lists, office chatter, and automated messages my inbox was under assault. Hundreds of messages per day of content that was only marginally useful. Some of it was duplicated due to other habits I incorporated into my schedule. I tried filtering some out, but then I did miss some important emails. It was imperative that I see them all, but then get rid of the ones I don’t need.

To do this I had to use the “Select All” feature. Yes, it was terrifying. I went through the first few pages of email in my Gmail accounts and took note of the ones that needed some attention: a reply, some action, etc. I took notes of their content so I could search for them later, pressed select all and hit “Archive”. Instantly to 0. But this wasn’t truly 0.

Once everything was archived I searched for the messages that I took note of earlier. This was good exercise in using the search to really hone in on a message.

So now you have a clean inbox, nothing was deleted, and you have a few actionable items left in your inbox. Perfect. Take action on those few items now that you don’t see some egregious number of messages in your inbox.

The hard part is keeping it clean. It’s truly taken months to get to the point where I maintain 0 on all of the accounts for a reasonable amount of time.

Since email is constant, first process the new messages. A message comes in, does it need to be read or is it automated and you know what to do with it? Read it or archive it (or delete if it’s a mailing list or automaton). If it’s something that you need to do now … DO IT! If it needs to be done later, leave it in your inbox.

When you start to type in your browser for some lovely cat videos, hit your inbox first. Pick something, get it done, then archive the damned message! You now have communication and task management in one place, the emails are just TODO items. Send your self emails to remind of tasks and toss them once done. Have that automated message from your dentist that it’s time for a cleaning, call your dentist and toss the message.

Like I said, this is not easy. You will truly feel a sense of loss when you archive everything. It’s also not an easy change and relapse is just a few hours away. You’ll slip and your inbox will grow and then you’ll have to fight your way back down.

Share Button