The electron, smallest of the three particles that make up the atom, how often do we take this plucky little lepton for granted? But let them stop flowing through the wires leading into our computers, and we quickly realize just how dependent we are on them. This lesson was brought home to me last night, as a late season snow storm took out the power to my house for 4 hours.
This isn’t the first time that PSNH (Power Shutdowns Noticeably Happen) has failed in their contractual obligation to keep the electrons flowing to my house. In fact, we lost power for 3 days in late January, and for 4 days in 1998 during the ice storms. I have pretty much everything in the house on a UPS (7 in all), and have a auto-starting generator on the budget for this year.
But once we take care of something, we tend to put it out of our mind. ” No need to worry about power failures, I have a UPS…” But we forget that most solutions come with new problems of their own. I got a practical demonstration of this fact when I started to hear a loud annoying chirping coming out of my rack a couple of times a day, lasting a minute. With that much stuff in my rack, there’s a lot of things that can make noise, but I surmised right off the bat that one of my UPSi was trying to tell me something.
Sure enough, one of them had it’s “replace battery” light on. Rather than replace the battery, I took the opportunity to order a 1500 VA rack mount UPS, since traditional “cinderblock” UPSi don’t really work well in a rack. Besides, the rack mount ones look cooler. That UPS is due to arrive via, well, UPS today. Unfortunately, when I needed it last night, I was stuck with the old one. It immediately started to chirp in a rapid, panic-inducing manner. Luckily, that UPS doesn’t power either of the two systems in the rack, so I didn’t need to worry about an abrupt shutdown on a PC. It did however power my brand new 24″ Gateway monitor, which began to flash on and off about once a second, as the UPS failed in an interesting mode.
There are two lessons for SysAdmins to take away from this. Firstly, a UPS is not a buy-once and forget item. You’re going to need to plan in advance to replace the batteries as they age. In a medium-size datacenter (one large enough to have lots of racks, but small enough not to have centralized conditioned power), it probably makes sense to standardize on a single model of UPS, and keep spare batteries around. I also learned something I already knew but had forgotten, you should have all of the critical items for a PC attached to the same UPS as the PC. Because the USB hub that my keyboard and mouse were attached to was on the flaky UPS, I had to scramble to attach them directly into the back of the PC so that I could shut it down.