Every once in awhile you’ll read another story about stolen backup tapes–with millions of confidential records–that are lost forever. Will someone steal your identity? Is the security of our nation compromised? We never know what happens to those tapes unless they’re miraculously recovered. I’d like to imagine that the unwitting thieves more likely destroy their ill-gotten booty because they tried just a little too hard to jam an LTO tape into a DLT drive (or worse yet, a VCR).

It’s human nature to believe criminals are stupid, which is never a reasonable assumption to make. Some people might write off these thefts as smash and grabs, when there is also consider the possibility that the systems administrator (and those backup tapes), were the intended target all along. The thieves break in, steal the tapes, but they also take your Cuisinart, your Xbox360 and your Veronica Mars DVDs to divert suspicion. The sysadmin is then busy working with an insurance claim, and nobody is reviewing the inventory of what was on those tapes.

There will always be sysadmins out there that take the backup tapes home. I’m not saying it’s outright bad, but it’s an understandable behavior. The most likely reason is that transporting the tapes from the tape library to an off site location is out of the way or inconvenient. If you’re at the data center with tapes in hand, and your house is 3 miles away, while the off site storage facility is 20 miles away; what would you choose?

There’s a policy at my current employer that attempts to reduce the chances of this happening. Tape changes can only occur before lunch, and never on a Friday. Eliminate the incentive to just drive home, and the tapes are more likely to end up at the proper storage facility. That works well in our situation, but your mileage may vary.

I worked at another IT shop, where off site storage was the sysadmin’s home. That’s okay if your employer is one of those 1 to 5 man garage startup wannabes. Just be sure to invest a few extra dollars and do it right; buy a safe. Make sure it’s fireproof, and large enough to accommodate at least 30 tapes. Then find out what your homeowners insurance will and won’t cover; a typical policy probably won’t cover the cost of the tapes in contrast to what you would get at a professional storage site.

The two technological solutions I’ll bring up include tape encryption, and remote mirroring. Encrypted tapes are less likely to be compromised by a thief, and remote mirroring of data to a separate facility eliminates the human transport factor altogether.

Tape backups and archives will always be susceptible to theft. The media is physically small enough, and valuable enough to always make it a tempting target. The sooner you come to terms with this, the better off you’ll be when it comes to setting policies and procedures to protect them.

There’s a Computerworld article out in the wild about the New York Stock Exchange (Euronet) building out 200 new Linux servers. Articles like this are usually puff-pieces, so technology people are always wondering about the real details behind decisions. I had a few questions about 2 interesting quotes made by CIO Steve Rubinow.

The first interesting comment was the desire to achieve “technology independence”. First, what are the benefits of technology independence to the NYSE? More to the point, how can an organization buy 200 servers of any architecture and remain technology independent?

True, there could be growth or a change in the direction, but if you’ve already committed yourself to 200 servers, you should have a good idea of what your future paths are.

The other comment was on Linux being “polished enough”, in comparison to Unix operating systems that have a 20 year history. What does polished enough mean? It almost suggests that the other OS offerings are better, but you’re settling for Linux ? What Linux features (if any) convinced you to use it?

Is there something Linux vendors could do to make a better OS offering? Or does it even matter since you’ve already committed yourself down this technology path? Should Linux contributors be happy that there is more business use, or should they be concerned that the reasoning behind using Linux may not come across as a glowing endorsement?

Here is my next poll about logs: Who looks at logs at your organization?

Vote here!

Also, my past polls and analysis are here.

Let’s say you have evidence of network errors. Here’s the symptoms that you see:

1. A lot of TCP retransmits (layer4)
2. No Ethernet frame errors, dropped packets, or CRC errors (layer 2)
3. No ICMP errors, or IP level errors. Pings report no lag or dropped packets. (layer 3)
4. Failures are only reported on two nodes in your network, but no errors on the switch between the two nodes.

Given the above evidence, would you look at the wiring between the two nodes, including the patch panel ports? If so, why?

No wrong answers, just trying to bring about an open discussion of opinions.

If you happen to be one of those lucky individuals dating somebody working in the computer forensics department of a law enforcement crime lab, there’s only one gift you need to give. It’s inexpensive, thoughtful, and it really comes in handy. That one gift is the plastic, waterproof, roll-up keyboard.

There are a few varieties out there, and they’re carried by multiple vendors, including Adesso. They can easily be found in the keyboard aisles of MicroCenter, Fry’s, and maybe Best Buy if you’re lucky. They’re compact, easy to carry and indispensable when it comes to working on a crime scene. It takes awhile to get used to the typing on soft rubber keys, but once they get the hang of it, it’s pretty simple to switch back and forth between this and a regular keyboard.

Every once in awhile, your forensic investigator/significant other may come across a computer they have to administer, but there is no keyboard readily available; or there is a keyboard, but it hasn’t been fingerprinted yet. In some cases, the keyboard may be a biohazard if it’s covered in blood, mucus, or other bodily fluids. Maybe the original keyboard was damaged by gunfire, or maybe the cold body of a white collar criminal is sprawled across it waiting to be attended to by a coroner.

With a roll-up keyboard, the investigator can quickly plug in through a USB or PS/2 port and do whatever necessary things he or she needs to get done. They can issue a shutdown command, or stop that last desperate attempt of performing a DoD wipe of a hard drive. Once the work is done, they can roll up the keyboard into a biohazard bag if it needs to be cleaned up later.

The only feature that this keyboard lacks is a pass-through USB port to attach a cheap $2 mouse. But considering the other benefits of these keyboards I’m willing to overlook this. The average price I found for these in stores is roughly $20. Don’t you think the forensic investigator in your life is at least worth that much?

Brendan Gregg wrote up a performance comparison between the utilities Top and Prstat. And just for good measure, he throws in a lot of Dtrace example code to show how he came to his conclusions.

Here’s some advice if you notice your system may be having a performance problem. Run the ls of your /proc filesystem. This will tell you how many active processes are running on your system.

If you’re dealing with a large process count, consider running either command with a short number of iterations, otherwise your monitoring attempts will only contribute to the overall load. I would argue that in a lot of cases, people running Top or Prstat in a background window are actually contributing to any performance problems that they eventually discover through the use of those tools.