This doesn’t look good, right?

Most open source monitoring tools do filesystem health checking by comparing the current percentage of used space against a set value. If it’s is 90% full, send out a warning page; if it’s 89%, send the all clear.

Notice that I said filesystem, and not actual disk. A single disk that’s 90% full can be a bad thing, because there are fewer free blocks available for writing, which leads to longer write times and file fragmentation. Not all filesystems are restricted to a single disk: there may be a back-end RAID solution, or the filesystem may be a shared filesystem served over NFS.

Unfortunately, you could be the receiver of flapping alert pages where a filesystem sits between 90% and 89%, but it still performs fine. Unlike a broken Ethernet cable, the resolution for a filesystem threshold may not be so easy. Sometimes there are files that can’t be deleted, or there may not be any additional storage to allocate. You may have a filesystem that sits at 91% full for months simply because a new disk shelf won’t arrive until the next budget cycle.

Everything comes down to disk blocks, even SAN and NAS solutions. That brings back the concern regarding fragmentation and performance. But what if your filesystem is a read-only OS image? Or what if it turns out 10% equates to 500 gigabytes on a huge disk appliance? If the filesystem is never being written to, or if the amount of writes equates to 0.001% of the entire filesystem, then where’s the fire?

What about the inverse? What if your filesystem never reaches 90% full? Can there still be problems?

In the above graph, nobody would have been paged by Nagios or other tools, because the filesystem never reached 90%. For the past few months it averaged 40% full, shot up to 75%, and then went back down. A newly released application was behaving incorrectly, and the issue was caught by the programmer. The next morning he stealthily re-released the application and corrected the issue. Nobody in systems administration noticed until the graph was checked in relation to another issue. If the programming error was never discovered, the filesystem would have filled up, probably at the most inconvenient time possible for a systems administrator.

I would like to recommend to people developing filesystem or disk monitoring solutions change their way of thinking about filesystem health. Hard limits on allocated space may still be required, but those warnings should be optional. Measuring fullness makes assumptions about block structure that may not be correct.

At the same time, the monitoring system should compare the standard deviation for the filesystem percentage over the past 24 hours, and compare it to the standard deviation for the past hour. Actually, you’d probably want to compare the first 23 hours out of 24, grab that standard deviation, and compare it to the deviation of the last hour.

If those two deviations aren’t close, then there could be radical changes made to your filesystem that need to be addressed. Maybe files are being added or deleted, either way, it may warrant an investigation. For large filesystems in the terrabyte/petabyte range, using the percentage value may not be granular enough, so you will need to work with the actual value of free kilobytes or blocks.

I take it back. This isn’t a recommendation to monitoring developers, this is a challenge. The first major open source monitoring guy that puts this solution together will have my undivided attention.

I’m working with a product that includes this disclaimer in their support documentation:

“Virtual environments, such as VMWare (and others) are not recommended, and thus not supported.”

I can almost see their point. It’d be pretty daunting to gauge a benchmark if a customer described the running host as “1/13th of two dual core processors, 3.1 gigs of memory, and a 27 gigabyte filesystem disk”. True, that’s a pretty extreme situation, but I wouldn’t doubt it if there was the occasional bad provisioning by virtual system installers.

Anyone who implements virtualization is implicitly trusting the VM solution to do the right thing, and when we see the operating system up and running, we just assume everything works perfectly. But let’s be honest: almost every VM solution creates some overhead, so you’re missing out on a few resources. That loss shouldn’t amount to much, but it could mean a lot to an application. And while CPU and memory can be partitioned, device IO such as hard disks are a little sketchy.

To the developers of the above unnamed application, I know it’s going to be a big hassle, but five years from now, you’re not going to be able to avoid virtualization. Instead of the blanket disclaimers, increase your virtualization knowledge base, and create more test suites. Find out what works, what doesn’t, and why. It’s still okay to set guidelines on usage, but a wholesale avoidance of virtualization will hurt in the long run.

I just bought a new MacBook. A co-worker of mine just bought a new MacBook Pro. To put it another way, we both bought new Apple Laptops….one week before the new models came out.

After Tuesday’s announcement I looked at the Apple website with a hint of despair. I missed out on a larger hard drive, a faster CPU, and more graphics memory. Catherine missed out on a faster CPU, more memory and storage, plus the holy grail of Multi-Touch. At the time of Apple’s announcement, my laptop was only five days old; and Catherine’s was six.

Both of our systems were bought in person at the same Apple store. We both knew what we wanted when we walked in so our sales encounters were short. Catherine’s sales rep was a short, clean cut guy, that looks like every Mac user walking down the street. Mine was a big, serious guy, who kind of looked like Dr. Artz. Both sales people were professional, courteous, and well informed on the products we asked about.

When the news of the new models hit Engadget, co-workers convinced us it wouldn’t hurt to go to the store and see what they would do for us. I didn’t have the laptop or receipt with me, but I figured I could just ask. During my lunch break I went out to that physical behemoth representing all that is good and capitalistic in this country; the Mall of America.

I found a manager and I explained my plight. To win over his sympathy, I made up a sob story about kids in an orphanage who only wanted the laptop so they could play a scratched up copy of Myst that somebody found in a dirty alley.

The manager explained my options: pay a restocking charge of 10% ($149), or get a refund of $200. The restocking charge was because I’d be returning an open/used laptop to Apple that they wouldn’t be able to sell. I told them I’d think about it and return the next day. Deep inside. I was kicking myself for selfishly taking my laptop out of the box and using it the very day I purchased it. I should have known better!

Since Catherine had the MacBook Pro her restocking charge would have been $250, but her refund would be $600. That’s a cost swing of $250 for me, and $850 for her. We both know we’re never going to be 100% ahead of the technology curve, but we didn’t expect to be hit with new models this quickly. To be fair, we’ll also admit that our gripes sound like a perfect candidate for White Whine.

Maybe the complaint is due to Apple’s notoriety in keeping new products secret. I can understand keeping the MacBook Air secret, because it’s a new product line. But why keep hardware upgrades secret? The reviews of the new models report that the units are faster, but there’s nothing really new or innovative. If the sales person had told us that new models was coming out, we both would have waited, and we would have been happier customers. Unfortunately, sales people at the Apple store have no idea when new models are coming, so they’re just as powerless as we are.

Both of us took the refund money and ran. We’re happy with what we have, but a little disappointed in how the product release cycles can make someone regret their purchase. The funny thing is, they’re both still good notebook computers. But the bleeding edge Mac culture (and to the PC culture as well) embraces newness, and shuns obsolescence.

Sam wrote a blog post about the cost of SMS messages. I admire the effort, but I’m not in 100% agreement with his conclusion.

I think the goal the author was trying to make is that SMS messages are overpriced, and consumers should be outraged. To support his arguments, he compares a price per byte breakdown between a SMS message, an email message, and a printed document with thousands of characters (whether binary, hexadecimal or base64, I couldn’t tell you). Unfortunately, the comparisons seem a little weak, and a real cost breakdown between these two technologies is not fair.

Let’s take a look at 3 points the author tries to make.

1. What is the value (and the cost) of an SMS message?

A SMS message typically originates from one user’s cell phone and arrives at another user’s cell phone. People use them for quick messages that either don’t need an immediate reply, or do not require the receiver to have 100% of their attention on their cell phone. Everyone has sent one of these at one time or another: “I’ll be late for lunch”, “OMG! TTYL?”, or “I never want to see your cheating face again”.

You don’t need to boot up a PC, and you have a higher expectation that someone will notice the message quicker, because almost everyone carries their cell phone with them. These aspects of SMS are features that carry a dollar value. Twenty cents may seem high per message when the technology cost is almost zero, but technology is not the only expense by a carrier.

First and foremost, people are needed to maintain the service. That includes systems administrators, engineers, and customer support personnel. Those costs need to be in balance with the total number of SMS messages that pass through a network. Every carrier out there has already measured their own cost per SMS message, and that includes transport, personnel costs, and billing costs. If SMS isn’t profitable, people could lose their jobs, or the per message price could go up.

2. Ah, but SMS shouldn’t have any cost, because the infrastructure is already in place, right?

Not exactly.

A per unit SMS charge could mean that you’re actually paying less for your base service. If every customer is expected to send 25 messages a month, they could reduce the base price to be competitive, knowing they’ll recover that cost with the add ons. If you send under 25 messages, the carrier hopes that the emotional teenager down the street will send 50 messages to cover your slack.

Alternatively, a carrier could reduce operating costs by offering unlimited messages for a base price, like $10 for unlimited messages; but that creates the risk that consumers would just forgo SMS altogether.

3. Apples and Oranges (Is email cheaper than SMS?)

The other aspect of this article that confuses me is the comparison between SMS and email. To put this in a better perspective, I’ll make my own argument.

If I were to fly from Minneapolis to Denver, the price of a one way ticket would be $290.23 with a flight time of approximately 2.5 hours. If I were to drive the exact same trip, my gas cost would be $180 (assuming that I can get 350 miles for every full tank of gas), and I would get there within 17 hours.

With either solution, I end up in Denver. Why does the airline in this case feel justified in charging such a higher cost? And if you think about it, that plane is going to Denver anyways, so I should be able to just ride it for free.

My comparison is a fail of epic proportions, because both methods of transportation have different operating costs. Airplanes cost more than cars, automobile gasoline has a tax for maintaining roads, flight attendants need paychecks. In both cases I’m paying different amounts of money, with different service expectations, but getting the exact same result.

The comparison between email and SMS isn’t fair because the author admits that there is no per email message cost. No ISP would ever want to deal with billing per email message, because the tracking of incoming and outgoing messages would only increase the price. You can use your bandwidth for web surfing, email, online games, or anything. SMS messaging does not offer these features.

Next, let’s discuss the idea of sending a single MP3 file (much less 2,560) over the SMS protocol. This is totally unreasonable action due to the size limits of SMS messages. SMS was never designed to transfer files, so why compare a file transfer? Most cell phone offer other methods, such as Bluetooth, or dedicated data networks for sending files. And while I’m at it, email isn’t the best protocol for file transfers either. If I had 2,560 individual emails of 4gb each, I would be looking at one mbox file of 10gb. Managing that mailbox would kill most mail clients, and probably a couple of IMAP servers as well.

Finally, the author is incensed that the person receiving the SMS message may also have to pay a surcharge. Unfortunately, he fails to point out that the recipient of the email message will very likely have an ISP charge as well.

Conclusion

If I’m coming across as harsh, it’s not my intention. This analysis is simply a counterpoint to the claim that text messages are expensive. Yes, they do cost a consumer money, and they probably make a profit for the carriers. I do not think this means SMS messages are bad, or exploitative.

The best way for a consumer to determine the cost of SMS messaging is to see what benefits the service gives you. If they save time, improve communication, or reduce confusion, there’s a value to that. If SMS does not do any of these, then you have the option of not using it.

I just installed XMLRPC::Lite on a Linux host. You may not have seen this module used too often, but it actually comes bundled in the SOAP::Lite distribution.

Installation time of one perl module in a 5.8.8 environment? One hour. Was it slow Internet bandwidth? Am I counting the meeting time where we discussed the need for this code? Neither. The install took one hour due to the need to satisfy all of the code dependencies.

On a host with a default perl installation, I also had to install the dependencies TimeDate, HTML-Tagset, HTML-Parser, MailTools, MIME-Types, Email-Date-Format, Test-Pod, Pod-Escapes, libwww-perl, Crypt-SSLeay, XML-Parser, Compress-Raw-Zlib, IO-stringy, File-Temp, IO-Compress-Zlib, IO-Compress-Base, Pod-Simple, MIME-Tools, FCGI, and Compress-Zlib. I also had to install the openssl-devel and expat-devel RPMs to satisfy the build requirements of some of the modules. And I didn’t get one nice full list of 20 dependencies to be satisfied; I had a list that seemed to grow with each module distribution that I installed.

I didn’t want to make the job too difficult, because this was a special case server outside of our normal support environment. No RPM packages were readily available, so going through a build process was still the quickest option.

I’m not angry, but I’ll admit to some frustration and confusion. The first question that will come to my mind is why aren’t some of these modules shipped by default yet? Or, out of all of these modules, how many of them are actually being used?

Is there anyone using Compress::Raw::Zlib that wouldn’t also want to install Compress::Zlib? Why bundle these modules separately? Are the extra Test and Pod modules really necessary for the end user, or does it just make the developers job easy? And is Email::Date::Format the only module out there that outputs the time in the RFC 2822 time format?

Again, no harm’s done; but I hope that perl authors and distributors keep this anecdote in mind. The less effort that I need to go through to install your code, the easier my job gets.

I just have to start with this quote from Rich Mogul: “… Legions of armchair futurists slobber over their keyboards, spilling obvious dribble that they either predict every year until it finally happens or is so nebulous that they claim success if a butterfly flaps its wings in Liechtenstein.” :-) Amen to that, Rich. Onwards to my 2008 predictions!

So, just as in 2006 and 2007, I am coming up with security predictions that cover both technology and market. I just posted a review of my last’s year’s prediction where I mostly erred on the conservative side. I promise to be more ‘extreme’ this year, while still keeping the old wisdom of Richard Feynman in mind: if you predict the status quo, you are more likely to be correct…

Here is my ‘twitter-style’ (I guess what used to be called telegraph-style :-)) view of predictions in no particular order:

Platform security:

• Vista makes us secure = no. People start to actually use it (in large numbers) = maybe. And then get 0wned = yes! The volume of Vista hacking (and then Win 2008 hacking) will increase as the year progresses.
• Increase in Mac hacking = yes. The story is that Vista drives Mac adoption -> Mac increase in popularity will drive a new wave of Mac “0wnership”
• Web application hacking still on the growth path = yes. As they say, ‘it will get worse before it gets better.’ I am predicting that 2008 is still the year when it continues to be getting worse.

Vulnerabilities:

• 0days use becomes mundane = yes. This will be especially true for those browser-hacking folks who “need” to earn some cash off phishing and other data theft. Thus, “0day use” will no longer constitute news!

Hacking, data theft, etc:

• Loss of trust towards legitimate Internet sites = yes. This is manifested by things like this point by the WS guys - more 0wned than malicious sites are used to spread malware. Even now I shudder from the thought that ANY site I visit might be displaying a malicious banner ad which is either bought or “hacked in” by the attackers. The implications of this are pretty horrifying!
• Major utility/SCADA hack = no (not yet). Everybody predicts this one forever (as Rich mentions), but I am guessing we would need to wait another year or so for this …
• Cyber-terrorism = no (again, not yet!) Will it be a reality in the future? You bet! Just not now …
• A massive data theft to dwarf TJX = yes. And it will include not some silly credit card number (really, who cares? :-)), but full identity - SSN and all.

Malware:

• The year of mobile malware = no (not yet, if you insist!). As I discussed here, mobile malware is “a good idea” (for attackers) provided there is something valuable to steal (not the case yet in the US)
• More fun bots = yes. Bots are here to stay: they follow an overall trend for IT automation (seriously!). Think of bot infrastructures as “shadow IT” with their own SLAs, business model innovation, performance optimization tactics, etc
• Fewer worms and viruses = yes (why write one if you can make money off bots?) As the share of “conventional” viruses and worms in the whole malware universe decreases, so will the popularity of “legacy” AV vendors …
• Facebook malware/malicious app = yes . This one will be fun to see (others agree), and current malware defenses will definitely not stop this “bad boy.”On the flip side, there is not that much to steal off Facebook accounts …

Compliance:

• PCI DSS continues its march = yes. In fact, I bet PCI DSS frenzy will spread downmarket - there is sooooo much more Level 3s and Level 4s compared to Level 1 merchants. They all take CCs, they are all insecure - thus, they will all be 0wned! And then hopefully fined :-)
• ISO17799, ITIL, COBIT frameworks = maybe (again); they likely won’t be ‘hot,’ at least not in the US; ad hoc approach (with some use of ideas from the above frameworks) to security management will still rule.

Risk management:

• Will we know what risk management actually is in the context of IT security = no. Some people (e.g here) might, but not the majority. And don’t even get me started on security ROI :-) This part of security realm will continue to be occupied mostly by loudmouths who will spout, but never define; rant, but never explain; blab, but never clearly state. Sorry to those who are not like this, but you will continue to be in the minority in 2008.

Security technologies:

• eVoting security will flare up = yes. Expect big and bad stories about evoting in preparation to the US elections. Maybe another “chad story”, but with an “e-” added to it? Fun, fun, fun! :-)
• Full disk encryption becomes popular = no. In fact, I predict that in 2008 encryption would be “the new firewall” - more and more people will hide from reality behind “we have encryption - we are safe now!” (check out my piece on encryption mistakes, while you are at it)
• NAC= huh. Huh? The451Group said it best: “NAC has been the ‘next big thing’ for about four years now - that’s a long time in the IT world.” Others just say “NAC fallout has started.” NAC vs insider attacks? Gimme a break… :-)
• More whitelisting for host and network security = yes (but combined with blacklisting, which is certainly not going away!) As malware landscape becomes even more diverse, application whitelisting for security will start to shine even more.
• Academic security research stays ridiculous = yes. Wrong problems, wrong solutions, wrong speed (as in: solving solved problems of day before yesterday…). There will be some exceptions: for example, some of the Project Honeynet academic participants deliver a punch!
• Secure coding becomes mainstream = no (definitely, ‘not yet’ on this one) It pains me to say that that I think that while this ball definitely started rolling (e.g. SANS is pushing it hard now) it won’t be hurtling down the highway at full speed. 2009? Sure, may be!
• IPv6 = no (while most think ‘not yet’, some start thinking ‘not ever’) In other words, Internet ’secure by design’ = pipe dream in 2008.

Security market:

• Mid-market and SMB security = yes! I think 2008 is the year when smaller organizations will start buying the types of security solutions that were only looked at by the large enterprises before. After all, they have the same problems to solve! They have compliance too. They lose data
• More security SaaS (software as a service) = yes. It is not just Qualys anymore … More companies will figure out ways to sell security software as a service. This is especially true due to the SMB security spending increase predicted above!
• ‘Consolidation’ = no. Whaaaaat? You just said ‘no’ to consolidation in security market? :-) Well, Vendor X might buy Vendor Z and Vendor N might go down in flames, but I predict that we will celebrate 2009 with just as many security vendors as we have today …

Logging and log management:

• Database logging = yes. 2008 is the year when database logs will be collected and analyzed just as Unix syslog, Windows event logs and firewall logs are collected and analyzed today by just about everybody.
• Application logging will start = yes. People will start collecting (at least collecting at first) application logs, not just firewall and server OS logs (and database logs, as mentioned above). Maybe ERP, CRM logs, maybe other large enterprise applications will lead the way. Major ‘application logging waterfall’ will occur later, however …
• Now that collection and management are ‘taken care of’ in many organizations, log analysis will (again…) come to the forefront = yes. In the end of 2008, we will be doing log analysis in a large number of fun, new ways - it won’t just be about rule-based correlation and keyword searching anymore (Andrew agrees)

Last year’s drag-ons :-) and ongoing trends:

• Some things make dumb predictions since they are so pitifully obvious and have been going on for years already. Thus, I pile them in this section…
• So, client vs server exploitation: it started a few years back and will continue, for sure: more client vulnerabilities will be used to 0wn more desktops. Similarly, application vulnerabilities will beat platform ones. And targeted, commercially-driven attacks will overtake indiscriminate ones (another “no-brainer” that some try to sell as a prediction…)
• Both of the above will power further evolution of network and system security into data and broader information security (it will be happening for another 3-5 years)
• More fun “web 2.0″ threats will come our way, but then again, this is true about most of the technologies that are being actively adopted …

Dark horses, that will influence security in a major but unknown way in 2008:

• Virtualization = people talk about hypervisor security and virtual security appliances as well as other fun stuff (e.g. this), but, in all honesty, we can’t yet fathom the impact that the coming virtualization wave will have on information security.
• Privacy = I predict that privacy issues, also privacy laws and public outcry due to privacy violations will impact the world of information security in 2008. However, my crystal ball is refusing to share the details on how exactly, citing “privacy concerns” :-)

Come back in Jan 2009 to see how I did!

Any comments? Additional predictions?

Even though these posts are from my main blog ( see “Security Warrior” blog) and not from this one, the top posts would still be of interest to my readers here. So, enjoy!

These are my top popular “Security Warrior” blog posts for 2007! To make this a competition of posts, I am removing the links to the main blog, search labels (e.g. log management, which was indeed one of the most popular resources on the blog) as well as grouping posts together in theme clusters.

1. Same as during past few months, the “fallout” from being featured on a high-profile programming site continues to drive humongous loads of traffic which made this set of posts the most popular, even for the year. The topic that got such a huge boost was anti-virus efficiency. The posts are: Answer to My Antivirus Mystery Question and a “Fun” Story, More on Anti-virus and Anti-malware, Let’s Play a Fun Game Here … A Scary Game, The Original Anti-Virus Test Paper is Here!, Protected but Owned: My Little Investigation as well as a final entry about my own switch away from mainstream major-vendor anti-virus tool: A Bit More on AV and Closure (Kind of) to the Anti-Virus Efficiency/Effectiveness Saga.
2. Next by rank is a set of my Top11 lists: Top 11 Reasons to Collect and Preserve Computer Logs and Top 11 Reasons to Look at Your Logs (the third list, Top 11 Reasons to Secure and Protect Your Logs, was not quite that popular - I have long argued that, sadly, few people care about log security yet).
3. Wow! I love, love, love the fact that my blog readers made my first Common Event Expression (CEE), post introducing this emerging log standard, (official site now live!) one of the most popular: Finally, Common Event Expression (CEE) is Out!!!. My other CEE-related posts are labeled here.
4. Hurray to database logging (finally!) My posts related to database logging top the charts. Specifically, How to Do Database Logging/Monitoring “Right”? as well as its “prequels” :-) Full Paper on Database Log Management Posted and On Database Logging and Auditing (Teaser + NOW Full Paper).
5. Finally, security ROI saga that flared up mid-year is also among the most popular. Indeed, Security ROI Pile-Up! post made it into Top5 (the related posts are: The Entire Security ROI Blood Trail and ROI, ROSI, RROI and Harry Potter Tales). The rest of my ROI-related posts are labeled here.
6. At the risk of destroying my math credibility, I will add an item #6 to my Top 5 list, again. This little post called On Open Source in SIEM and Log Management have also generated a lot of traffic and discussion. Indeed, log management vs SIEM as well as reasons for a lack of a popular and complete open source log management solution are fun topics!

See you in 2009! :-)

Possibly related posts:

• Monthly Blog Round-Up - December 2007
• Monthly Blog Round-Up - November 2007
• Monthly Blog Round-Up - October 2007
• Monthly Blog Round-Up - September 2007
• Monthly Blog Round-Up - August 2007

Every once in awhile you’ll read another story about stolen backup tapes–with millions of confidential records–that are lost forever. Will someone steal your identity? Is the security of our nation compromised? We never know what happens to those tapes unless they’re miraculously recovered. I’d like to imagine that the unwitting thieves more likely destroy their ill-gotten booty because they tried just a little too hard to jam an LTO tape into a DLT drive (or worse yet, a VCR).

It’s human nature to believe criminals are stupid, which is never a reasonable assumption to make. Some people might write off these thefts as smash and grabs, when there is also consider the possibility that the systems administrator (and those backup tapes), were the intended target all along. The thieves break in, steal the tapes, but they also take your Cuisinart, your Xbox360 and your Veronica Mars DVDs to divert suspicion. The sysadmin is then busy working with an insurance claim, and nobody is reviewing the inventory of what was on those tapes.

There will always be sysadmins out there that take the backup tapes home. I’m not saying it’s outright bad, but it’s an understandable behavior. The most likely reason is that transporting the tapes from the tape library to an off site location is out of the way or inconvenient. If you’re at the data center with tapes in hand, and your house is 3 miles away, while the off site storage facility is 20 miles away; what would you choose?

There’s a policy at my current employer that attempts to reduce the chances of this happening. Tape changes can only occur before lunch, and never on a Friday. Eliminate the incentive to just drive home, and the tapes are more likely to end up at the proper storage facility. That works well in our situation, but your mileage may vary.

I worked at another IT shop, where off site storage was the sysadmin’s home. That’s okay if your employer is one of those 1 to 5 man garage startup wannabes. Just be sure to invest a few extra dollars and do it right; buy a safe. Make sure it’s fireproof, and large enough to accommodate at least 30 tapes. Then find out what your homeowners insurance will and won’t cover; a typical policy probably won’t cover the cost of the tapes in contrast to what you would get at a professional storage site.

The two technological solutions I’ll bring up include tape encryption, and remote mirroring. Encrypted tapes are less likely to be compromised by a thief, and remote mirroring of data to a separate facility eliminates the human transport factor altogether.

Tape backups and archives will always be susceptible to theft. The media is physically small enough, and valuable enough to always make it a tempting target. The sooner you come to terms with this, the better off you’ll be when it comes to setting policies and procedures to protect them.

There’s a Computerworld article out in the wild about the New York Stock Exchange (Euronet) building out 200 new Linux servers. Articles like this are usually puff-pieces, so technology people are always wondering about the real details behind decisions. I had a few questions about 2 interesting quotes made by CIO Steve Rubinow.

The first interesting comment was the desire to achieve “technology independence”. First, what are the benefits of technology independence to the NYSE? More to the point, how can an organization buy 200 servers of any architecture and remain technology independent?

True, there could be growth or a change in the direction, but if you’ve already committed yourself to 200 servers, you should have a good idea of what your future paths are.

The other comment was on Linux being “polished enough”, in comparison to Unix operating systems that have a 20 year history. What does polished enough mean? It almost suggests that the other OS offerings are better, but you’re settling for Linux ? What Linux features (if any) convinced you to use it?

Is there something Linux vendors could do to make a better OS offering? Or does it even matter since you’ve already committed yourself down this technology path? Should Linux contributors be happy that there is more business use, or should they be concerned that the reasoning behind using Linux may not come across as a glowing endorsement?

While it could vaguely be considered system/application news, Nintendo announced a new version of “Photo Channel”, will be released in December. For those that own a Wii, Photo Channel is that icon that is mostly ignored, right next to “Wii Shop Channel”.

The update includes the comment that they’ll be dropping MP3 support for AAC support, and it has a few Nintendo users a little upset. The rumor was that the change was brought about due to licensing costs for the MP3 codec, but nobody knows for sure.

I’m one of the few people I know that’s made the switch from MP3s into AAC files, and burn CDs directly into AAC. I didn’t gain anything in quality from the MP3 to AAC conversion, but there was a slight savings in hard drive space. I still have my music, and it can still be played on the iPod, the Zune, or other players put out by Creative, SanDisk, Sony, etc, etc.

So my question is, is the MP3 file format really more popular than AAC; or does it only appear to be more popular because the term MP3 is slowly becoming a lost trademark in the same vein of Xerox and Coke?

AAC is superior to MP3, since it is the successor format; but do people actually use the extended features? Or is this a case where Betamax is technically superior, but for some reasons the markets prefer VHS?

Mark Wade wrote a firsthand account on tracking a purchase that was originally solicited from a SPAM email.

There are a couple of points where the article is vague on technical details, but I’m probably not the intended audience. I found the article to be informative, but it also left me with more questions.

First, I wanted more details on the domain registrations that were encountered along the way. Are the domains still active, or are they on administrative hold? I’m wondering if spammers pay for domains with stolen credit cards, knowing they may only have a couple of days to use it; or if they keep a cache of domains that they cycle through, and just move the site contents from domain to domain.

Second, the article mentions that somewhere along the chain of purchasing his item that a SSL certificate was encountered. A trusted SSL certificate is going to be a little harder for a spammer to get compared to a domain name. Remember, the whole point of SSL certificates are to make Internet commerce safe, and to give consumers confidence with a system of verifiable trust. Unless, the cert is free, payment for a certificate is up front; and the certificate itself should have accurate owner information. The author never mentions who the signing authority was, nor did he mention whether or not there were any signing errors.

Third, he mentions a tracking log of the package, but does not reveal which shipping company it was. Since the package never arrived, I’m left wondering whether the entire log was a fake. Usually, a shipping company cannot provide tracking information for a package that goes through other shippers. For example, a tracking number for something shipped via Canada Post isn’t too helpful once the parcel is handed off to the US Postal System for its final delivery. The log shown in the article mentions multiple hops in China, and multiple hops in Virginia. Was the comment about the package being lost in the USPS a real loss by the post office, or sarcastic wit?

Compromising hosts to send spam is pretty easy; but setting up an actual commerce website to handle transactions risks more exposure. If a spammer really wants to make money this way, there has to be a path leading from the victim to the criminal.

At the same time, there are other companies involved in the purchasing process that create more leads. Domain registrars, web hosting providers, certificate providers, shippers, and credit card processors. As a consumer, it would be more interesting to me to know which of these vendors have more extensive track records of dealing with spammers. Are these companies being duped, or are they more swayed by easy money compared to building consumer confidence?

The original title of Mr. Wade’s article was “Following the SPAM”, and in that regard he seems to have done a good job. As a follow-up, I would recommend a more detailed investigation, and to repeat the advice once given to Woodward and Bernstein: “Follow The Money.”

The legal maneuvering between two of my more favored tech companies just kicked it up a notch. Johnathan Schwartz looks like he’s not going to take NetApp’s lawsuit lying down.

I’m not even going to pretend I’m a lawyer, but a counter-suit is probably a suitable response to the original NetApp suit. In the end, the best hope for everyone (customer wise), is a mutual understanding between these two behemoth companies.

There are too many companies out there that have major investments in NFS, ZFS, and Netapp filers. For enterprise users of any of these technologies (or all of them), a one sided victory probably sounds pretty scary.

There’s an interesting post on the Skype forums where a Linux user asks why Skype is reading his /etc/passwd file.

Since it was posted during the weekend, and there are Skype developers out there dealing with other issues, I’m guessing he won’t get a reasonable answer from a Skype developer until at least Monday.

It’s not my job to cover for Skype, but I’ll try to give a reassuring answer. More than likely, Skype is only reading /etc/passwd in order to get information about the userid it is running under. It may need to do this to determine its home directory, or to learn more information about the user.

If the programmers are doing it right, they’re using the getpwent() system call to grab the password entry through the name service switch. That’s because you may not be using a local password file at all. If you’re using a workstation in a large environment, your authentication information could be stored in LDAP, NIS, or another global password map. Ironically, when a program uses getpwent(), it doesn’t even know if you have a valid /etc/passwd file.

If you traced the call even further, you would see that the program is reading all of the entries in the /etc/passwd file. Since the file is sequential, there’s no way around that. Nor is there an easy way to tell what happens to the data once it’s read. Maybe it ignores the entries that it doesn’t care about, or maybe it emails them to an insidious hacker.

The /etc/passwd file can be read by any user, so it carries the bare minimum amount of security information. The actual passwords are encrypted and stored in the /etc/shadow file, which is then protected by the operating system by making it read-only from the root account. Reading /etc/passwd might give someone a slight insight into what accounts to possibly compromise, but it won’t offer any special insight on how to compromise them.

On a final note, although I encourage the testing and using of security tools, like AppArmor, I think its use in this case is unneeded. Chris Brown said it best in Protecting your applications with AppArmor:

AppArmor is not intended to provide protection against execution of ordinary tools run by ordinary users. You already have the classic Linux security model in place to constrain the activities of such programs.

Any regular user on a Unix host, should rely on the host operating system security model. I’ll admit, I feel more comfortable saying that about a Unix/Linux/Mac host than I would for a Windows host. If Skype–or any application–tries to perform an action that it isn’t allowed to do, the operating system should prevent it.

The recent change to German law to implement the EU Framework Decision on Attacks against Information Systems (enacted in Paragraph 202c of the German Penal Code) has caused many security researchers based in Germany to look to move elsewhere, or to remove previously available research findings.

The change in the law, which went into effect on August 10, criminalises the production, distribution, possession, and sale of tools that can be used to commit cybercrimes. Unfortunately, a strict interpretation of the changes would make possession of tools that could be used maliciously (such as nmap or Nessus for instance) illegal. While in reality, legal opinions are that the courts would differentiate between a cracker and a security researcher based on their intent, noone (unsurprisingly) seems to want to be the first test case.

The content for a number of projects have all but disappeared, such as the recent Month of PHP bugs, and the well known THC (The Hackers Choice) group, as well as smaller projects such as BtCrawler. Others are saying farewell to Germany and reestablishing themselves elsewhere such as the KisMac wifi scanner for OSX and the Phenoelit group.

All in all a hard strike against a country which has produced much valuable security research and expertise.

Every once in awhile I will have maintenance windows that need to start at Midnight. When announcements are made, there’s always going to be one person out there that gets the day wrong. If I say the system will be unavailable Wednesday at Midnight, I will be asked if I mean Tuesday night leading into Wednesday morning, or Wednesday night leading into Thursday morning.

Technically, Midnight is the beginning of a new day. If you’re used to military time, it’s easier to understand, since you’re starting over at 0000. But most people have always used Midnight in the context of a late evening, as in, I’m going to stay up until Midnight; so it’s pretty common to accidentally associate it with the previous day.

To overcome confusion, we announce all Midnight maintenance windows as having a start time of 12:01 AM. It’s close enough to Midnight, and for some reason, everybody seems to understand it. Sure, Midnight is ambiguous, but 12:01 on a Wednesday is clearly early understood as Wednesday morning. I’m not going to complain about a one minute compromise if it means avoiding ten minutes of writing emails to clarify the schedule.

Most data center operators require all their servers to have static IP addresses (or they pin a dynamic address to a specific hardware address, which is essentially the same thing), and most also require them to have meaningful names. This makes a lot of sense for a number of reasons.

First of all, you don’t want someone to be able to come along and plug in any old device into your data center network. After all, this is the pipe that’s most likely to be pumping sensitive information between databases, mainframes, and web servers. By requiring a static IP or a whitelisted MAC address, you make it less likely (but not nearly impossible) for a rogue device to be able to access all the network configuration it needs.

Second, it’s important that consumers of data center services always know how to connect to those services. Granted, this is becoming less important today as these services are more likely to be clustered and hidden behind switches. But a good data center operator always wants to know how to get directly to any of her boxes from a secure shell.

Third, speaking of secure shells, if a server ever does change its IP address or other aspects of its identity, you’ll get an annoying but serious error message.

I’m sure there are many other reasons. And I hope some of my readers will share them with me.

The new thing is that resources in the dynamic data center can be created, provisioned, deprovisioned, and destroyed with no intervention from an operator; old policies about naming and addressing no longer apply. In the dynamic data center, virtual machines appear, do some work, and disappear on the fly. Depending on the virtualization system, a new VM may be assigned a random MAC address just before being loaded for the first time. There is no opportunity for an operator to assign a static IP address to a virtual machine or to configure a newly-created virtual MAC address in a DHCP whitelist.

Data center operators are going to need to change their view of the world. Systems management software vendors need to propose a better way.

I work in academia. I’m a sysadmin. However, I took a rather non-traditional route to sysadmin-hood. The very brief version of the story goes like this:

I started as a lowly database reporting geek. I found that I liked databases, and the database guys took me under their wing and made a DBA out of me. Then I went to work for Sybase and became a full-blown data snob. However, on various client sites I found that the people I inevitably needed to interact with to get my work done were the sysadmins.

As time went on I gained their trust and they started giving me more privileges on the machines where the database servers were running. It was a great help, and I was able to spread my sysadmin wings in an environment quite close to what would be called “production”.

After about a year of Perl scripting on the database end and for the sysadmin tasks I needed to perform, I decided that what I really wanted to do was build out these huge end-to-end systems. I wanted to do everything. I wanted to much with network gear, environmental monitors, power distribution, UNIX, databases, LDAP, Apache, and whatever else I could get my hands on. My next job was for a consulting firm that put me through tons of systems training, and I was on my way.

When I wasn’t at work, I was reading books about Linux and UNIX (I think I read all of the ones available at that time, actually), pounding the Linux forums, and setting up services on Linux boxes I had set up at home. I could rattle off ipchains rulesets in my sleep, recite Apache rewrite rules verbatim, quote error strings and tell you what they meant and how long you had before your disk just completely failed. I could set up quad-boot machines, run Linux on old SPARCs, and I had already written code to handle most of the basic admin tasks, as well as some basic monitoring.

In short, I was determined. I had given up any notion of a life, hacked day and night, read Phrack, 2600, the llama book, bat book and more, grokked perl, php, sed, awk, tr, ed, vi, ksh and bash, and had gotten myself ready to sit for the SCSA, SCNA, CCNA, and CheckPoint certifications after working in IT, mostly in database administration and development, for about 3 years.

I still felt like a flaming idiot. Heck, there are plenty of occasions where I feel pretty dopy now! Luckily, my sponge-like brain has shown no signs of becoming saturated.

My main goals now boil down to doing my best to fight specialization. I don’t want to stop coding Python, or doing database development, or maintaining LDAP servers, or building beowulf clusters, or maintaining VMWare servers. Aside from these goals, I also try to share as much of the knowledge I have with others who might be where I was 10 years ago by putting it here, or on my blog, or on my Linux admin site, or other sites, or on IRC, or the forums, or in magazine articles, or in slide presentations for LUGs. Oh - I once put some of my knowledge in a book, too!

What I want to know now, though, is this: How did you get here? Did you major in CS and choose systems work? Did you do something non-technical and became your company’s all ’round IT guy? Did you fight your way out of the phone support farm? Let me know! Share your story!

Imagine you bought something.

• You rely on it with your business, with your very livelihood. Sometimes even with your life.
• There is no warranty whatsoever on what you bought.
• And you don’t know what’s inside the box.
• Also, you cannot look inside the box, in fact, it is illegal.
• You might not have have heard about the seller before and you have no particular reason to trust him.

Are you totally and irreversibly mad? How can you do it?! If you are not mad, aren’t you criminally negligent? Or just very, very, very stupid.

However, we all are. We all bought software at least once in our lives …

This blurb is inspired by some discussions I had at CONFidence 2007 Conference (where I presented on “Log Forensics” in front of about 180 people). Another related fun thought I picked there is that the most scary cyber-criminal of the future is not a spammer, a scammer, a phisher or a pharmer, and not even a good ole “cracker” - it is an unethical software engineer, who changes the code slightly to introduce a weakness (or a full-blown backdoor or a logic bomb) and later uses or sells this knowledge. In light of the above characteristics of software purchases, think billions stolen in one shot, think ruined companies, think stock market manipulation, think direct physical damage (and, yes, real cyberterror), etc. We do live in interesting times …

The electron, smallest of the three particles that make up the atom, how often do we take this plucky little lepton for granted? But let them stop flowing through the wires leading into our computers, and we quickly realize just how dependent we are on them. This lesson was brought home to me last night, as a late season snow storm took out the power to my house for 4 hours.

This isn’t the first time that PSNH (Power Shutdowns Noticeably Happen) has failed in their contractual obligation to keep the electrons flowing to my house. In fact, we lost power for 3 days in late January, and for 4 days in 1998 during the ice storms. I have pretty much everything in the house on a UPS (7 in all), and have a auto-starting generator on the budget for this year.

But once we take care of something, we tend to put it out of our mind. ” No need to worry about power failures, I have a UPS…” But we forget that most solutions come with new problems of their own. I got a practical demonstration of this fact when I started to hear a loud annoying chirping coming out of my rack a couple of times a day, lasting a minute. With that much stuff in my rack, there’s a lot of things that can make noise, but I surmised right off the bat that one of my UPSi was trying to tell me something.

Sure enough, one of them had it’s “replace battery” light on. Rather than replace the battery, I took the opportunity to order a 1500 VA rack mount UPS, since traditional “cinderblock” UPSi don’t really work well in a rack. Besides, the rack mount ones look cooler. That UPS is due to arrive via, well, UPS today. Unfortunately, when I needed it last night, I was stuck with the old one. It immediately started to chirp in a rapid, panic-inducing manner. Luckily, that UPS doesn’t power either of the two systems in the rack, so I didn’t need to worry about an abrupt shutdown on a PC. It did however power my brand new 24″ Gateway monitor, which began to flash on and off about once a second, as the UPS failed in an interesting mode.

There are two lessons for SysAdmins to take away from this. Firstly, a UPS is not a buy-once and forget item. You’re going to need to plan in advance to replace the batteries as they age. In a medium-size datacenter (one large enough to have lots of racks, but small enough not to have centralized conditioned power), it probably makes sense to standardize on a single model of UPS, and keep spare batteries around. I also learned something I already knew but had forgotten, you should have all of the critical items for a PC attached to the same UPS as the PC. Because the USB hub that my keyboard and mouse were attached to was on the flaky UPS, I had to scramble to attach them directly into the back of the PC so that I could shut it down.

I read an article on Slashdot the other day about an newly released open source application. I read a few of the comments, and I found this one (slightly paraphrased):

$ apt-get fooapp
"You have searched for packages named fooapp in all distributions....Can't find that package."
Sorry, I'm not interested.

The comment suggested that since he can’t try the software in a pre-built distribution then it isn’t worth trying.

Unlike a few years ago when every Linux user ran configure by hand, the speed and convenience of installing packages has put the compiler on the back burner. Packages aren’t a bad thing, but I think it’s a poor reflection of an administrator’s skill set if they shun the development tools that are available for every Unix environment.

I’m not saying that packages should be avoided. I build them myself for software that I have compiled and tested manually. Once packaged, they’re pushed out during the remote installation process. After that, I am considered the distributor for certain applications in the infrastructure, and the primary support contact. I’ll also concede that it doesn’t make much sense to recompile Gnome or KDE if my OS vendor provides a pre-built package, along with support and regular upgrades. I won’t be too optimistic about installing packages that aren’t built or approved by the author of the software.

When I interview an administrator, I usually throw in a couple of programming interview questions, such as, “How do you determine which shared libraries a program requires?” or “What steps would you take to compile the Apache webserver?”. I don’t expect them to be a full fledged C programmer, but I think it’s important to know how to build software. The candidates that have demonstrated these skills have also been more proficient in debugging, tracing system calls, and identifying performance problems ahead of time.

Ok, so I’m not completely sold yet. I still have a boatload of Perl code floating about, and for certain things I’m still writing *new* Perl code. However, I was coerced into using Python for a project I’m working on, and I have to say that I think Python is coming on to me.

I try to ignore the furtive glances, and those times when I could swear it’s actually winking at me. I don’t acknowledge the beguiling smiles and greetings I get from Python when I open my laptop. I just get down to the business of coding and pretend none of it ever happened. That pretending is getting harder by the day.

Here’s the thing. I’ve been doing all of my sysadmin scripting in perl, awk and shell (sometimes together) for a decade. After 10 years, Perl still doesn’t even say “hello” to me. It seems to stand ready to spit my own code all over me whenever I try to talk to it. And just when I’m about ready to call it my friend, just when I think I know it, it completely changes. Well, I’m tired of it. I’m tired of the schizophrenia. I’m tired of the attitude. I’m tired of feeling like a Perl n00b after using it for 10 years.

I’m leaving.

Of course, like lots of relationships, it’s complicated. Over the years, Perl and I have spawned offspring that aren’t going to just disappear because I decide I don’t like Perl anymore. I promise to care for them and keep them up to date.

But from this day forward, I’m going with Python in those places where I can. I *want* to feel confident with a language. I *want* to take advantage of code reuse, self-documenting code, and OO design principles. I *want* to have readable, concise code. I *want* to solve problems that are larger than the every day “please change my shell” requests. I want to build tools. I want to architect solutions. I want to solve some of the problems sysadmins face, but I had to solve my own big problem first: namely, being a self-hating Perl slinger who was never particularly comfortable with how Perl, at a very high level, works.

If you’re an admin using Python on a regular basis for your admin scripting, let me know how you think it compares to Perl for equivalent tasks. If you’re a Perl coder who has tried and *not* used Python for reasons besides the lack of curly braces, fill us in! If you’re a religious zealot for one language and have never used the other, feel free to move on!

If you’re not managing your time, it’s almost guaranteed that you’re misusing it. I’ve heard, and even used, tons of excuses in the past to not manage my time, but eventually you’ll come to the realization that saying you don’t have time to figure out a system to manage your time is just ridiculous. For those of you who feel this way, I recently blogged about the system I’m building to manage my time in a completely digital way.

I used the old Franklin planner for quite some time, but I eventually realized that since I have a blackberry, and a way to synchronize in a “round trip” fashion from the blackberry to a task list, to a web-based calendar, and finally back to a local calendar, there would seem to be no gaps left for the dead-tree planner to fill.

If you found other tools useful or you’re still hanging on to the Personal Analog Assistant, please share with us your thoughts on the tools you’re using or the gaps that you still haven’t found a digital solution for.