Being a school district, we don’t always have the latest and greatest technology as compared to an enterprise environment. When I first started with the district our server room was a small room the size of a large closet. It had no four post racks, water sprinklers for fire suppression, and a window air conditioner to keep it cool. It was not exactly an ideal location for mission critical servers to be housed, but it was all we had, so we made due with it.

Two years ago we finally had the opportunity to create a room a bit better suited for housing servers. The new room has a gas fire suppression system, six four post racks that are bolted to the floor, a separate heating and air conditioning system, and a large UPS system for providing backup power. The server room has worked well for us, but it hasn’t been without its teething problems. Friday was one of the problems.

At this time of the year in Minneapolis, the high temperature is usually quite cool — in the 40s and 50s. Consequently, the air conditioning system in our server room is not usually running. We’ve usually switched to heat or at the very least, piping in outside air. Our air handling system is handled by a computer and so we don’t usually have to think about. This past Friday was a fluke. The high was 75 degrees in Minneapolis. As a result, the air handling system should have turned on the air conditioning system. Unfortunately, because of a mechanical problem, that didn’t happen. It didn’t take long for the temperature to begin to soar in that room. Unfortunatley for us, we didn’t realize it for quite a while because of a critical mistake we made in the new server room. The mistake: we don’t have a device monitoring the temperature in that room. The result, of course, was that we were not notified that there was a problem until the servers themselves began to notify us that they were getting hot. Not an ideal situation!

Because the situation got very hot very quickly, we ended up shutting down the majority of our servers to prevent them from producing additional heat and to prevent them from overheating. Fortunately, this happened late on Friday afternoon, so it wasn’t a big deal to shut them down — we were lucky! The air conditioning was fixed on late Friday evening and everything is back up and running now.

What did we learn? We need to have something monitoring the temperature of that room so we can be notified if there is a problem way before the servers themselves start reporting a problem. I’ll be ordering something to do this job sometime tomorrow. It seems like such a simple thing to have that. I’m surprised that none of us thought of this sooner!

–Pete

UPDATE 11-13-04: Our new temperature/humidity sensors arrived this week. I’ll be installing them as soon as I get the chance.