You may or may not know that I write another blog called “The K-12 Tech Guy”. This is a blog that I wrote as part of my job in the Hopkins School District. The big news is that my last day in Hopkins is Friday, March 25, which means that the K-12 Tech Guy will come to an end as a separate blog. I’m going to import the entries I wrote as part of that blog into this blog.

I plan to continue to post “geek stuff” from time to time as part of this blog.

Speaking of “geek stuff”, we’ve had a tough couple of days at work with one of our servers. We run an issue tracking (or “ticketing”) system called Request Tracker. Up until two days ago, RT was running on server class hardware running the testing version of Debian Linux. I wasn’t involved with setting up the box originally, but at the time Debian was chosen because we had some “extra” x86 hardware sitting idle and the guy who set it up had a good amount of Debian.experience.

Over the last year, the server has run pretty well, but in the last month or so we started noticing intermittent problems with it. On Wednesday afternoon it finally died. When I went to check on what was going on with the server I discovered it was kernel panicked and waiting to be rebooted. I rebooted it, and as expected, it complained that it hadn’t done a clean shutdown and automatically ran fsck on the drives. Keep in mind that the root file system partition was not set up on a journaled file system because a stable one wasn’t available a the time we originally set it up. Initially fsck found errors, but it was able to repair them. The problem was, that after it repaired the problems and we rebooted it, it still failed to start. UGH! We did lots of Googling and posted some messages to the Twin Cities Linux Users Group mailing list, but came up with no resolution to fix the software. The general feeling was that one or more of the config files got munched up when the kernel panic happened and there was no easy way to figure out which one.

So we bit the bullet, backed up our data (which was on a journaled partition), and began the process of rebuilding the server. It has been two days since we started and we are still working on it. We made the switch from Debian to Red Hat Enterprise Linux because several of the utilities, such as backup daemons and UPS software are better supported on Redhat. We also made sure we are using a journaled file system for the boot partition this time.

The basic server software is rebuilt, and we are now in the process of installing Request Tracker. This has proven to be challenging as well. RT is very particular about which versions Perl and Perl modules it uses and the stock set that comes with RHEL is not what RT is optimized around. We’ve decided to “roll our own” Perl and install in /opt and then we’ll point RT to that instead. This creates a bit more work, but she make RT rock solid. Sound like fun yet?

–Pete