Moved to my own domain

The PhDOps blog has moved to my own domain, as using Blogger with embedded source code was just too annoying.
New posts will be found at the new location.

Installing a valid SSL certificate on HP ILO3

UPDATE 2014-11-06: Can't use an IP address for the CN, fixed the text accordingly

After not having to manage my ILO3based HP servers for a while, today I did feel the need for remote console access. Unfortunately, Firefox 33 decided that the SSL certificate shipped out of the box is invalid. So, for my future reference and for everybody running into a similar issue, here's how to fix access by creating a certificate sign request (CSR) on the ILO, creating an SSL certification authority (CA) on your work machine and using the CA to sign the CSR. The bonus of doing this is that you don't have to bother with browser warnings about the SSL cert anymore.

Start by logging into the ILO with a less picky browser (Chromium worked for me) and under Administration > Security > SSL Certificate, fill out the required information. I initially tried using the IP address to avoid the "certificate name/host mismatch" warning, but it turns out that while ILO3 claims to load a certificate when the CN is an IP, it just falls back to the old invalid certificate instead. So use the ILO's host name and then set up that host name to access the ILO (DNS or /etc/hosts). Click the "Generate CSR" button. This will take a while to generate the CSR, so in the meantime, get the CA started on your work machine.

# generate the CA key, using a 4096 bit RSA key with an AES256 passphrase
openssl genrsa -aes256 -out rootCA.key 4096
...enter password twice when prompted, don't forget this password...

# Create and self-sign the root CA certificate
openssl req -x509 -new -nodes -key rootCA.key -days 1024 -out rootCA.pem

# copy the rootCA certificate to the distro's CA certificate store
cp rootCA.pem /etc/ssl/certs/   # might be different for your distro

In the meantime, hopefully the generation of the CSR in the ILO has finished (make sure to wait 10 minutes), so click the "Generate CSR" button again to get a pop-up with the base64-encoded CSR. Copy&paste that into a file on your work machine, I'll assume ilo.csr.

# optional: Check if the CSR looks sane
openssl req -text -noout -verify -in ilo.csr

# sign the ILO CSR using the rootCA key
openssl x509 -req -in ilo.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out ilo.crt -days 500
...enter password when prompted...

Now click the "Import certificate" button in the ILO and copy&paste the ilo.crt file contents into the text box. Click import and wait for an ILO restart.
While the ILO is restarting, open your browser settings and import your /etc/ssl/certs/rootCA.pem file as trusted CA.

Raspberry Pi Temperature Monitor, Mark II

A while ago I set up a Raspberry Pi based temperature monitoring system, to monitor temperatures for the equipment room and the server rack and freezers located in there. I still need to write a post detailing the software and hardware used for this, hopefully I'll get around to that soon. In short, I'm using a bunch of 1-wire DS18B20 digital temperature sensors connected to the Raspberry Pi. Today, though, I spent my time upgrading the case and fixation of the temperature monitor.

Mark I had a simple plastic back and a cardboard front, and was taped to the network/electricity rail in the room, like so:

While expedient to get set up and move the mess of cable out of the way, this setup has some problems. First, the Raspberry Pi tended to crash a lot. Not the software, but the hardware. That's because this yellow tape sucks, and that's why eventually I brought some proper duct tape and "fixed it". Still, it was clear that eventually even duct tape would succumb to the pull of gravity. Also, we'll lose power for a scheduled test of the emergency power system on Thursday, and I wanted to get my little Pi into the server rack so I could hook it up to the UPS.

For the Mark II case, I wanted to get a proper plastic case, front and back. Fortunately, when you work in a lab, there are a lot of petri dishes around. We also have the large square ones. Two of those, a drill, a hand saw and half an hour later, the Mark II case was ready to go.

It's attached to the rack door mesh with cable ties (yay, cable ties), and nicely out of the way of the 19" slots. Also visible is the custom board with audio jacks that the sensors are connected to. The falcon tube cap on the right is a stand-off so the add-on board won't push down too far, that still feels a bit hacked. Oh well, we're getting there.

Update 2013-09-18, 08:32 UTC: Turns out there is one unexpected feature with mounting the "in-rack" sensor on the door as well. It is now possible to track when I open and close the door, and of course the temperatures are off in that case.

The anatomy of a freezer failure

A while ago I built a system to monitor temperatures in my server and freezer room. Yes it's the same room, but I could only get the space and air conditioning once. I still need to write the blog post on how I set up the temperature monitor system, but basically it's a Raspberry Pi with a number of DS18B20 1-wire digital temperature sensors. I'm using statsd and graphite to graph the data, and a separate set of scripts to send email alerts when specific sensors go above their configured temperatures.

Today was the first time the system was put to use, and I'm pretty happy with how it all worked out. I can even reconstruct what probably happened. Looking at my graphite temperature graph for that freezer, the following story unfolds:

A bit after 11:00, someone opened the freezer. They used the top drawer, which is pretty full with boxes holding Eppendorf cups. These boxes come in a low and a high version. The inner ceiling of the freezer is lower in the back than it is in the front, and care must be taken that the last row of boxes in the drawer only consists of the low boxes. If you place a high box there, the drawer will not slide in all the way but will stick out about 5 mm. These additional 5 mm will prevent the door from creating a seal when closed. At around 11:05, the temperature control of the freezer notices that the temperature is too high and starts to cool more. Note that my own temperature sensor is sitting right in the cold air vent rigth on the top of the freezer, so it's probably measuring more extreme temperatures than the built-in sensor.
After running the compressor at full whack for about 100 minutes, the cooling finally fails and the freezer starts to thaw. At 12:45, the temperature rises above -10 °C, triggering an alert email. Thus alerted, yours truly walks over to the freezer room and notices that the door doesn't close correctly. After shuffling some boxes around, the door closes correctly again and the temperature begins to fall.
Unfortunately, the controller of the freezer must have crashed at 12:40 as well, because the freezer does not resume cooling. The rising temperature triggers a second warning email at 13:06. This time, it takes a bit longer to realize what the actual problem is about, but eventually I realize that the cooling isn't doing anything and toggle power on the freezer. The cooling comes back on, and the situation is improving again.

The first live failure has shown that for our purposes the temperature monitoring systems is working nicely. While the freezer did fail during the day, the controller got stuck somehow and didn't trigger the acoustic alarm we should have gotten in case of a failure. The email-based system works, and the response time during the day is sufficient to avoid serious problems. For a cost of about 100 EUR in electronics, the Raspberry Pi / DS18B20 combination is a cost-effective way to monitor a room full of freezers.

PhDOps is now a thing.

Since March 23rd 2013, I've been using the #PhDOps as a tag on twitter for posts relating to the work I do maintain the antiSMASH web service. As I've been collecting blog posts that I needed to write about running an academic web service while having a full time job getting a PhD, I figured this would be a good name for a blog to collect said posts.

Stay tuned for a post on using Docker to deploy Linux container (LXC) based application containers.