Earlier this year, without any logical reason, the wireless on my wife’s MacBook went flaky.

We had been using it for over a year with no problems whatsoever, but after installing new recessed lighting in my kitchen, the MacBook now has intermittent problems getting a signal from the the Linksys WRT54G router in the basement. No other laptops have any problem connecting, and the MacBook has no problem connecting to other wireless networks.

I used to be a Mac pro, but my Mac OS X chops are a little flabby. I’ve examined the logs to see if there is some explanation for this problem, but I can’t figure it out. There are basically two or three areas of the house where I am more or less guaranteed to receive a signal; the rest of the house is a big, intermittent dead zone.

Call me naive, but I still enjoy traveling. Maybe it’s because I have not been seriously affected by delays or cancellations (pace my trips to Kelowna), but there’s still little I like better than getting on a plane and going somewhere new. Especially when, as is true now, it’s been a number of weeks since I’ve been out of town.

This week it’s New York City for two nights. I’ll be manning the Novell booth at Interop New York on Wednesday and Thursday, and I will be taking time to meet up with friends who are attending the O’Reilly Web 2.0 conference next door.

Come by and say hello at booth 439 if you’re coming to Interop, or give me a shout if you’re in town and want to meet up.

I had a really fantastic time at OSCON this year. When I went to OSCON in 2006, I was a little put off by the fact that there seemed to be very little focus on systems administration, and meanwhile systems administrators are responsible for huge swaths of open source software growth in businesses large and small. I’m happy to report that this year’s OSCON had quite a bit of focus on topics of interest to systems folks, including myself. Thanks, ORA!

But on the 5-day flight back to the east coast (ok, maybe it just felt like 5 days), I had a chance to think more about what I had seen and heard. I had a lot of conversations, mostly with developers, about cloud computing initiatives like AppEngine, Amazone Web Services, BigTable, and the like. My take on this is that Amazon and Google are providing developer-centric interfaces to help them solve traditional systems administration problems…. and they appear to be doing it with some success.

So one might ask… “where does this leave the lowly sysadmin?”

It leaves you in an extremely fast-paced, ever-changing technological landscape with a set of needs, tools, and technologies that never stop evolving, and cause a lot of perceived community fragmentation while everyone scrambles to figure out which direction is “the way to go”. Sound familiar? It should. It’s exactly where you’ve been for your entire career. Some would say it’s the fact that things never seem to stagnate that makes them love system administration in the first place!

What I view as being pretty exciting (and I hope this continues - call me a blasphemer) is that, because of these developer-centric systems interfaces, there’s a bit of a forced convergence: developers have no choice but to have some understanding of what’s happening under the hood, because they’re going to have to write tools to essentially “rope in the cloud” — to manage all of this stuff. On the other hand, systems administrators would probably do well to take this opportunity to do more interesting stuff with code than the typical pushing out of account information, watchdog scripts, custom log parsers, system tool wrappers, and the like.

It’s a great time to pick up a new language, too, if you have an interest. Lots of sysadmins have picked up Ruby and/or Python as a means of broadening their horizons. If you use and like Perl, there’s no reason you can’t use it, but seeing what the hubbub is about surrounding newer stuff that looks like it’s more than a fad by now can’t hurt. I personally chose Python as my primary language because I never liked Perl, and Ruby *looks* like Perl to me (though I still dabble with it just to be famiiliar). If you *like* Perl, check out Ruby. If you only use Perl because you have to, give Python a shot! There are client libraries for a lot of these new services available in Ruby, Python, Perl, and PHP.

Dive into the cloud! The water’s fine!

“(Twitteriffic/Urban Spoon/Where) would like to use your current location”

Once I select “Ok” two times while running the program, the iPhone no longer asks; it assumes the program has carte-blanche to know my location. I haven’t been prompted for any other security issues yet, but I’ve only used about six or seven applications so far.

Granting privilege escalation on an application by application basis is good, but I’d like to make a couple of recommendation for the next release of the iPhone OS.

1. From the “Settings” application, give me a master list of all installed applications, so I can say in advance whether an app can have, cannot have, or must always ask for the privilege it is requesting.

2. Clearly identify that the OS is prompting for privilege escalation, and not the application itself.

3. Create more privileges. For example, Twitteriffic needs permission to know where I am; but MyStreets doesn’t need permission to read and sort all of the contacts on my phone. And with VoiceRecord, I was never prompted for permission for the application to listen to my microphone. That would suggest that any application could just read the contact info or use the microphone at any time. Maybe I’m wrong, or maybe Apple’s application screeners check this out beforehand.

I came, I saw, I conquered, and somehow, I also managed to activate my new 3G iPhone. And I seem to be the only guy I know in my area that has made it past that final step.

Early this morning, Two co-workers, assorted friends, and myself, found each other at different points in the line. The following is a transcript of the events, recreated from SMS messages and Twitter posts

6:00 Woke up late.
6:41 (incoming SMS from Clay) Where are you?
6:42 (reply to Clay) Sorry, woke up late. ETA to MoA = 12 minutes
6:55 (to Clay) Fyckinf light rail car blocking mall entrance!
6:56 (reply from Clay) LOL!
7:00 Parked at west entrance, standing in line.
7:20 I download an app to my old EDGE phone to pass the time.
7:35 I can’t believe she cut in front of me and 8 other people. Is she married to that guy?
7:55 Curtains are down
8:00 Line’s moving
8:03 People cheer and applaud at the first sale. (this is starting to make my cynical)
8:30 In store
8:35 With personal concierge. Handled upgrade of At&t account.
8:40 Sign-in to iTunes failed.
8:45 Bought hard plastic case from Contour Design. Unlike the old metal phone, the plastic case could never look badass.
8:47 Heading in to the office
8:50 Neither phone works. EDGE unavailable on old phone, but Wi-fi works on both phones.
9:03 Try plugging new phone into PC, and running iTunes. No joy.
9:39 Receive SMS page from At&t.
10:13 Receive second SMS page from At&t. Unexpected incoming call arrives.
10:14 Confirmed that phone is working.

My activation process never was fully completed, but it seems to work. This was my first time camping for an Apple product, so I’ll give the experience a 6 on a scale of 1-10 for excitement. The store purchase flow was smooth (despite the iTunes unresponsiveness), employees were helpful and friendly, and they managed to handle the crowd pretty quickly.

New York and the East Coast had a one hour advantage over us mid-westerners, so there was already a strain on the sale process by the time we were let in. I’m guessing if you are currently holding a brick, it’ll either self activate, or you’ll have better luck with iTunes this afternoon.

Thwarting DDoS attacks is good for tier 1 content providers - and not for the reason you’re probably thinking. I was recently talking with someone who recently ejected from the content provider world, and he told me an interesting story about how peering relationships are actually impacted by DDoS (distributed denial of service) attacks. Let’s say for a moment that you’re providing hosting for some of the largest companies in the world. Chances are you are pushing a lot more packets out to users on the Internet than you are receiving.

It turns out that tier 1 peering agreements are set up in such a way that they want it to be as close to a 1 to 1 relationship as possible in terms of ingress and egress packets. If you as a content provider slip too far below a certain threshold you can get re-negotiated to a tier 2 or tier 3 status, giving you far worse rates, worse service level agreements and less quality of service. But the problem is if you are a content provider, you generally receive about 1K of data inbound and push out dozens or hundreds of packets on the outbound for each request. Hardly a 1 to 1! Kiss your tier 1 status goodbye!

So in comes a DDoS attack. Generally speaking DDoS attacks try to consume bandwidth or system resources. In either case generally they create a great deal of outbound packets as the systems try to respond to the DDoS attack. This would make the ratio of inbound to outbound packets worse if left unchecked, let alone the other more obvious negatives for your consumers in not being able to reach your sites!

Off you go to find yourself an anti-DDoS vendor to fit your needs and after implementing them you suddenly realize you are now no longer responding to the DDoS attack with packets from your side. Instead the ratio of packets becomes closer to equilibrium - as many inbound packets as outbound. Your inbound packets are a mix of good and bad traffic, of course, but maintaining that tier 1 relationship is key for many large content providers, so that same DDoS attack (and thwarting it) became a huge business asset suddenly.

Now, that leads us to the next obvious leap of logic - where any inbound automated traffic that can be blocked might be worth blocking, to a point, so that your levels of inbound and outbound traffic are stabilized. Web application firewalls fit into that mold, where they are able to send significantly less traffic if they block traffic that is not destined to a real person. Blocking the spiders and robots that won’t affect your marketing efforts with the search engines seems like an easy way to solve some of the massive outbound packet issues that cause content providers so much pain with their peering agreements.

Clearly the benefits are visible to anyone who penny pinches regarding their bandwidth bills, but this was an interesting new take on how to justify the cost of your security devices, if you needed another bullet in your justification to your boss.

This doesn’t look good, right?

Most open source monitoring tools do filesystem health checking by comparing the current percentage of used space against a set value. If it’s is 90% full, send out a warning page; if it’s 89%, send the all clear.

Notice that I said filesystem, and not actual disk. A single disk that’s 90% full can be a bad thing, because there are fewer free blocks available for writing, which leads to longer write times and file fragmentation. Not all filesystems are restricted to a single disk: there may be a back-end RAID solution, or the filesystem may be a shared filesystem served over NFS.

Unfortunately, you could be the receiver of flapping alert pages where a filesystem sits between 90% and 89%, but it still performs fine. Unlike a broken Ethernet cable, the resolution for a filesystem threshold may not be so easy. Sometimes there are files that can’t be deleted, or there may not be any additional storage to allocate. You may have a filesystem that sits at 91% full for months simply because a new disk shelf won’t arrive until the next budget cycle.

Everything comes down to disk blocks, even SAN and NAS solutions. That brings back the concern regarding fragmentation and performance. But what if your filesystem is a read-only OS image? Or what if it turns out 10% equates to 500 gigabytes on a huge disk appliance? If the filesystem is never being written to, or if the amount of writes equates to 0.001% of the entire filesystem, then where’s the fire?

What about the inverse? What if your filesystem never reaches 90% full? Can there still be problems?

In the above graph, nobody would have been paged by Nagios or other tools, because the filesystem never reached 90%. For the past few months it averaged 40% full, shot up to 75%, and then went back down. A newly released application was behaving incorrectly, and the issue was caught by the programmer. The next morning he stealthily re-released the application and corrected the issue. Nobody in systems administration noticed until the graph was checked in relation to another issue. If the programming error was never discovered, the filesystem would have filled up, probably at the most inconvenient time possible for a systems administrator.

I would like to recommend to people developing filesystem or disk monitoring solutions change their way of thinking about filesystem health. Hard limits on allocated space may still be required, but those warnings should be optional. Measuring fullness makes assumptions about block structure that may not be correct.

At the same time, the monitoring system should compare the standard deviation for the filesystem percentage over the past 24 hours, and compare it to the standard deviation for the past hour. Actually, you’d probably want to compare the first 23 hours out of 24, grab that standard deviation, and compare it to the deviation of the last hour.

If those two deviations aren’t close, then there could be radical changes made to your filesystem that need to be addressed. Maybe files are being added or deleted, either way, it may warrant an investigation. For large filesystems in the terrabyte/petabyte range, using the percentage value may not be granular enough, so you will need to work with the actual value of free kilobytes or blocks.

I take it back. This isn’t a recommendation to monitoring developers, this is a challenge. The first major open source monitoring guy that puts this solution together will have my undivided attention.

I’m working with a product that includes this disclaimer in their support documentation:

“Virtual environments, such as VMWare (and others) are not recommended, and thus not supported.”

I can almost see their point. It’d be pretty daunting to gauge a benchmark if a customer described the running host as “1/13th of two dual core processors, 3.1 gigs of memory, and a 27 gigabyte filesystem disk”. True, that’s a pretty extreme situation, but I wouldn’t doubt it if there was the occasional bad provisioning by virtual system installers.

Anyone who implements virtualization is implicitly trusting the VM solution to do the right thing, and when we see the operating system up and running, we just assume everything works perfectly. But let’s be honest: almost every VM solution creates some overhead, so you’re missing out on a few resources. That loss shouldn’t amount to much, but it could mean a lot to an application. And while CPU and memory can be partitioned, device IO such as hard disks are a little sketchy.

To the developers of the above unnamed application, I know it’s going to be a big hassle, but five years from now, you’re not going to be able to avoid virtualization. Instead of the blanket disclaimers, increase your virtualization knowledge base, and create more test suites. Find out what works, what doesn’t, and why. It’s still okay to set guidelines on usage, but a wholesale avoidance of virtualization will hurt in the long run.

I just bought a new MacBook. A co-worker of mine just bought a new MacBook Pro. To put it another way, we both bought new Apple Laptops….one week before the new models came out.

After Tuesday’s announcement I looked at the Apple website with a hint of despair. I missed out on a larger hard drive, a faster CPU, and more graphics memory. Catherine missed out on a faster CPU, more memory and storage, plus the holy grail of Multi-Touch. At the time of Apple’s announcement, my laptop was only five days old; and Catherine’s was six.

Both of our systems were bought in person at the same Apple store. We both knew what we wanted when we walked in so our sales encounters were short. Catherine’s sales rep was a short, clean cut guy, that looks like every Mac user walking down the street. Mine was a big, serious guy, who kind of looked like Dr. Artz. Both sales people were professional, courteous, and well informed on the products we asked about.

When the news of the new models hit Engadget, co-workers convinced us it wouldn’t hurt to go to the store and see what they would do for us. I didn’t have the laptop or receipt with me, but I figured I could just ask. During my lunch break I went out to that physical behemoth representing all that is good and capitalistic in this country; the Mall of America.

I found a manager and I explained my plight. To win over his sympathy, I made up a sob story about kids in an orphanage who only wanted the laptop so they could play a scratched up copy of Myst that somebody found in a dirty alley.

The manager explained my options: pay a restocking charge of 10% ($149), or get a refund of $200. The restocking charge was because I’d be returning an open/used laptop to Apple that they wouldn’t be able to sell. I told them I’d think about it and return the next day. Deep inside. I was kicking myself for selfishly taking my laptop out of the box and using it the very day I purchased it. I should have known better!

Since Catherine had the MacBook Pro her restocking charge would have been $250, but her refund would be $600. That’s a cost swing of $250 for me, and $850 for her. We both know we’re never going to be 100% ahead of the technology curve, but we didn’t expect to be hit with new models this quickly. To be fair, we’ll also admit that our gripes sound like a perfect candidate for White Whine.

Maybe the complaint is due to Apple’s notoriety in keeping new products secret. I can understand keeping the MacBook Air secret, because it’s a new product line. But why keep hardware upgrades secret? The reviews of the new models report that the units are faster, but there’s nothing really new or innovative. If the sales person had told us that new models was coming out, we both would have waited, and we would have been happier customers. Unfortunately, sales people at the Apple store have no idea when new models are coming, so they’re just as powerless as we are.

Both of us took the refund money and ran. We’re happy with what we have, but a little disappointed in how the product release cycles can make someone regret their purchase. The funny thing is, they’re both still good notebook computers. But the bleeding edge Mac culture (and to the PC culture as well) embraces newness, and shuns obsolescence.

Sam wrote a blog post about the cost of SMS messages. I admire the effort, but I’m not in 100% agreement with his conclusion.

I think the goal the author was trying to make is that SMS messages are overpriced, and consumers should be outraged. To support his arguments, he compares a price per byte breakdown between a SMS message, an email message, and a printed document with thousands of characters (whether binary, hexadecimal or base64, I couldn’t tell you). Unfortunately, the comparisons seem a little weak, and a real cost breakdown between these two technologies is not fair.

Let’s take a look at 3 points the author tries to make.

1. What is the value (and the cost) of an SMS message?

A SMS message typically originates from one user’s cell phone and arrives at another user’s cell phone. People use them for quick messages that either don’t need an immediate reply, or do not require the receiver to have 100% of their attention on their cell phone. Everyone has sent one of these at one time or another: “I’ll be late for lunch”, “OMG! TTYL?”, or “I never want to see your cheating face again”.

You don’t need to boot up a PC, and you have a higher expectation that someone will notice the message quicker, because almost everyone carries their cell phone with them. These aspects of SMS are features that carry a dollar value. Twenty cents may seem high per message when the technology cost is almost zero, but technology is not the only expense by a carrier.

First and foremost, people are needed to maintain the service. That includes systems administrators, engineers, and customer support personnel. Those costs need to be in balance with the total number of SMS messages that pass through a network. Every carrier out there has already measured their own cost per SMS message, and that includes transport, personnel costs, and billing costs. If SMS isn’t profitable, people could lose their jobs, or the per message price could go up.

2. Ah, but SMS shouldn’t have any cost, because the infrastructure is already in place, right?

Not exactly.

A per unit SMS charge could mean that you’re actually paying less for your base service. If every customer is expected to send 25 messages a month, they could reduce the base price to be competitive, knowing they’ll recover that cost with the add ons. If you send under 25 messages, the carrier hopes that the emotional teenager down the street will send 50 messages to cover your slack.

Alternatively, a carrier could reduce operating costs by offering unlimited messages for a base price, like $10 for unlimited messages; but that creates the risk that consumers would just forgo SMS altogether.

3. Apples and Oranges (Is email cheaper than SMS?)

The other aspect of this article that confuses me is the comparison between SMS and email. To put this in a better perspective, I’ll make my own argument.

If I were to fly from Minneapolis to Denver, the price of a one way ticket would be $290.23 with a flight time of approximately 2.5 hours. If I were to drive the exact same trip, my gas cost would be $180 (assuming that I can get 350 miles for every full tank of gas), and I would get there within 17 hours.

With either solution, I end up in Denver. Why does the airline in this case feel justified in charging such a higher cost? And if you think about it, that plane is going to Denver anyways, so I should be able to just ride it for free.

My comparison is a fail of epic proportions, because both methods of transportation have different operating costs. Airplanes cost more than cars, automobile gasoline has a tax for maintaining roads, flight attendants need paychecks. In both cases I’m paying different amounts of money, with different service expectations, but getting the exact same result.

The comparison between email and SMS isn’t fair because the author admits that there is no per email message cost. No ISP would ever want to deal with billing per email message, because the tracking of incoming and outgoing messages would only increase the price. You can use your bandwidth for web surfing, email, online games, or anything. SMS messaging does not offer these features.

Next, let’s discuss the idea of sending a single MP3 file (much less 2,560) over the SMS protocol. This is totally unreasonable action due to the size limits of SMS messages. SMS was never designed to transfer files, so why compare a file transfer? Most cell phone offer other methods, such as Bluetooth, or dedicated data networks for sending files. And while I’m at it, email isn’t the best protocol for file transfers either. If I had 2,560 individual emails of 4gb each, I would be looking at one mbox file of 10gb. Managing that mailbox would kill most mail clients, and probably a couple of IMAP servers as well.

Finally, the author is incensed that the person receiving the SMS message may also have to pay a surcharge. Unfortunately, he fails to point out that the recipient of the email message will very likely have an ISP charge as well.

Conclusion

If I’m coming across as harsh, it’s not my intention. This analysis is simply a counterpoint to the claim that text messages are expensive. Yes, they do cost a consumer money, and they probably make a profit for the carriers. I do not think this means SMS messages are bad, or exploitative.

The best way for a consumer to determine the cost of SMS messaging is to see what benefits the service gives you. If they save time, improve communication, or reduce confusion, there’s a value to that. If SMS does not do any of these, then you have the option of not using it.

I just installed XMLRPC::Lite on a Linux host. You may not have seen this module used too often, but it actually comes bundled in the SOAP::Lite distribution.

Installation time of one perl module in a 5.8.8 environment? One hour. Was it slow Internet bandwidth? Am I counting the meeting time where we discussed the need for this code? Neither. The install took one hour due to the need to satisfy all of the code dependencies.

On a host with a default perl installation, I also had to install the dependencies TimeDate, HTML-Tagset, HTML-Parser, MailTools, MIME-Types, Email-Date-Format, Test-Pod, Pod-Escapes, libwww-perl, Crypt-SSLeay, XML-Parser, Compress-Raw-Zlib, IO-stringy, File-Temp, IO-Compress-Zlib, IO-Compress-Base, Pod-Simple, MIME-Tools, FCGI, and Compress-Zlib. I also had to install the openssl-devel and expat-devel RPMs to satisfy the build requirements of some of the modules. And I didn’t get one nice full list of 20 dependencies to be satisfied; I had a list that seemed to grow with each module distribution that I installed.

I didn’t want to make the job too difficult, because this was a special case server outside of our normal support environment. No RPM packages were readily available, so going through a build process was still the quickest option.

I’m not angry, but I’ll admit to some frustration and confusion. The first question that will come to my mind is why aren’t some of these modules shipped by default yet? Or, out of all of these modules, how many of them are actually being used?

Is there anyone using Compress::Raw::Zlib that wouldn’t also want to install Compress::Zlib? Why bundle these modules separately? Are the extra Test and Pod modules really necessary for the end user, or does it just make the developers job easy? And is Email::Date::Format the only module out there that outputs the time in the RFC 2822 time format?

Again, no harm’s done; but I hope that perl authors and distributors keep this anecdote in mind. The less effort that I need to go through to install your code, the easier my job gets.

I just have to start with this quote from Rich Mogul: “… Legions of armchair futurists slobber over their keyboards, spilling obvious dribble that they either predict every year until it finally happens or is so nebulous that they claim success if a butterfly flaps its wings in Liechtenstein.” :-) Amen to that, Rich. Onwards to my 2008 predictions!

So, just as in 2006 and 2007, I am coming up with security predictions that cover both technology and market. I just posted a review of my last’s year’s prediction where I mostly erred on the conservative side. I promise to be more ‘extreme’ this year, while still keeping the old wisdom of Richard Feynman in mind: if you predict the status quo, you are more likely to be correct…

Here is my ‘twitter-style’ (I guess what used to be called telegraph-style :-)) view of predictions in no particular order:

Platform security:

• Vista makes us secure = no. People start to actually use it (in large numbers) = maybe. And then get 0wned = yes! The volume of Vista hacking (and then Win 2008 hacking) will increase as the year progresses.
• Increase in Mac hacking = yes. The story is that Vista drives Mac adoption -> Mac increase in popularity will drive a new wave of Mac “0wnership”
• Web application hacking still on the growth path = yes. As they say, ‘it will get worse before it gets better.’ I am predicting that 2008 is still the year when it continues to be getting worse.

Vulnerabilities:

• 0days use becomes mundane = yes. This will be especially true for those browser-hacking folks who “need” to earn some cash off phishing and other data theft. Thus, “0day use” will no longer constitute news!

Hacking, data theft, etc:

• Loss of trust towards legitimate Internet sites = yes. This is manifested by things like this point by the WS guys - more 0wned than malicious sites are used to spread malware. Even now I shudder from the thought that ANY site I visit might be displaying a malicious banner ad which is either bought or “hacked in” by the attackers. The implications of this are pretty horrifying!
• Major utility/SCADA hack = no (not yet). Everybody predicts this one forever (as Rich mentions), but I am guessing we would need to wait another year or so for this …
• Cyber-terrorism = no (again, not yet!) Will it be a reality in the future? You bet! Just not now …
• A massive data theft to dwarf TJX = yes. And it will include not some silly credit card number (really, who cares? :-)), but full identity - SSN and all.

Malware:

• The year of mobile malware = no (not yet, if you insist!). As I discussed here, mobile malware is “a good idea” (for attackers) provided there is something valuable to steal (not the case yet in the US)
• More fun bots = yes. Bots are here to stay: they follow an overall trend for IT automation (seriously!). Think of bot infrastructures as “shadow IT” with their own SLAs, business model innovation, performance optimization tactics, etc
• Fewer worms and viruses = yes (why write one if you can make money off bots?) As the share of “conventional” viruses and worms in the whole malware universe decreases, so will the popularity of “legacy” AV vendors …
• Facebook malware/malicious app = yes . This one will be fun to see (others agree), and current malware defenses will definitely not stop this “bad boy.”On the flip side, there is not that much to steal off Facebook accounts …

Compliance:

• PCI DSS continues its march = yes. In fact, I bet PCI DSS frenzy will spread downmarket - there is sooooo much more Level 3s and Level 4s compared to Level 1 merchants. They all take CCs, they are all insecure - thus, they will all be 0wned! And then hopefully fined :-)
• ISO17799, ITIL, COBIT frameworks = maybe (again); they likely won’t be ‘hot,’ at least not in the US; ad hoc approach (with some use of ideas from the above frameworks) to security management will still rule.

Risk management:

• Will we know what risk management actually is in the context of IT security = no. Some people (e.g here) might, but not the majority. And don’t even get me started on security ROI :-) This part of security realm will continue to be occupied mostly by loudmouths who will spout, but never define; rant, but never explain; blab, but never clearly state. Sorry to those who are not like this, but you will continue to be in the minority in 2008.

Security technologies:

• eVoting security will flare up = yes. Expect big and bad stories about evoting in preparation to the US elections. Maybe another “chad story”, but with an “e-” added to it? Fun, fun, fun! :-)
• Full disk encryption becomes popular = no. In fact, I predict that in 2008 encryption would be “the new firewall” - more and more people will hide from reality behind “we have encryption - we are safe now!” (check out my piece on encryption mistakes, while you are at it)
• NAC= huh. Huh? The451Group said it best: “NAC has been the ‘next big thing’ for about four years now - that’s a long time in the IT world.” Others just say “NAC fallout has started.” NAC vs insider attacks? Gimme a break… :-)
• More whitelisting for host and network security = yes (but combined with blacklisting, which is certainly not going away!) As malware landscape becomes even more diverse, application whitelisting for security will start to shine even more.
• Academic security research stays ridiculous = yes. Wrong problems, wrong solutions, wrong speed (as in: solving solved problems of day before yesterday…). There will be some exceptions: for example, some of the Project Honeynet academic participants deliver a punch!
• Secure coding becomes mainstream = no (definitely, ‘not yet’ on this one) It pains me to say that that I think that while this ball definitely started rolling (e.g. SANS is pushing it hard now) it won’t be hurtling down the highway at full speed. 2009? Sure, may be!
• IPv6 = no (while most think ‘not yet’, some start thinking ‘not ever’) In other words, Internet ’secure by design’ = pipe dream in 2008.

Security market:

• Mid-market and SMB security = yes! I think 2008 is the year when smaller organizations will start buying the types of security solutions that were only looked at by the large enterprises before. After all, they have the same problems to solve! They have compliance too. They lose data
• More security SaaS (software as a service) = yes. It is not just Qualys anymore … More companies will figure out ways to sell security software as a service. This is especially true due to the SMB security spending increase predicted above!
• ‘Consolidation’ = no. Whaaaaat? You just said ‘no’ to consolidation in security market? :-) Well, Vendor X might buy Vendor Z and Vendor N might go down in flames, but I predict that we will celebrate 2009 with just as many security vendors as we have today …

Logging and log management:

• Database logging = yes. 2008 is the year when database logs will be collected and analyzed just as Unix syslog, Windows event logs and firewall logs are collected and analyzed today by just about everybody.
• Application logging will start = yes. People will start collecting (at least collecting at first) application logs, not just firewall and server OS logs (and database logs, as mentioned above). Maybe ERP, CRM logs, maybe other large enterprise applications will lead the way. Major ‘application logging waterfall’ will occur later, however …
• Now that collection and management are ‘taken care of’ in many organizations, log analysis will (again…) come to the forefront = yes. In the end of 2008, we will be doing log analysis in a large number of fun, new ways - it won’t just be about rule-based correlation and keyword searching anymore (Andrew agrees)

Last year’s drag-ons :-) and ongoing trends:

• Some things make dumb predictions since they are so pitifully obvious and have been going on for years already. Thus, I pile them in this section…
• So, client vs server exploitation: it started a few years back and will continue, for sure: more client vulnerabilities will be used to 0wn more desktops. Similarly, application vulnerabilities will beat platform ones. And targeted, commercially-driven attacks will overtake indiscriminate ones (another “no-brainer” that some try to sell as a prediction…)
• Both of the above will power further evolution of network and system security into data and broader information security (it will be happening for another 3-5 years)
• More fun “web 2.0″ threats will come our way, but then again, this is true about most of the technologies that are being actively adopted …

Dark horses, that will influence security in a major but unknown way in 2008:

• Virtualization = people talk about hypervisor security and virtual security appliances as well as other fun stuff (e.g. this), but, in all honesty, we can’t yet fathom the impact that the coming virtualization wave will have on information security.
• Privacy = I predict that privacy issues, also privacy laws and public outcry due to privacy violations will impact the world of information security in 2008. However, my crystal ball is refusing to share the details on how exactly, citing “privacy concerns” :-)

Come back in Jan 2009 to see how I did!

Any comments? Additional predictions?

Even though these posts are from my main blog ( see “Security Warrior” blog) and not from this one, the top posts would still be of interest to my readers here. So, enjoy!

These are my top popular “Security Warrior” blog posts for 2007! To make this a competition of posts, I am removing the links to the main blog, search labels (e.g. log management, which was indeed one of the most popular resources on the blog) as well as grouping posts together in theme clusters.

1. Same as during past few months, the “fallout” from being featured on a high-profile programming site continues to drive humongous loads of traffic which made this set of posts the most popular, even for the year. The topic that got such a huge boost was anti-virus efficiency. The posts are: Answer to My Antivirus Mystery Question and a “Fun” Story, More on Anti-virus and Anti-malware, Let’s Play a Fun Game Here … A Scary Game, The Original Anti-Virus Test Paper is Here!, Protected but Owned: My Little Investigation as well as a final entry about my own switch away from mainstream major-vendor anti-virus tool: A Bit More on AV and Closure (Kind of) to the Anti-Virus Efficiency/Effectiveness Saga.
2. Next by rank is a set of my Top11 lists: Top 11 Reasons to Collect and Preserve Computer Logs and Top 11 Reasons to Look at Your Logs (the third list, Top 11 Reasons to Secure and Protect Your Logs, was not quite that popular - I have long argued that, sadly, few people care about log security yet).
3. Wow! I love, love, love the fact that my blog readers made my first Common Event Expression (CEE), post introducing this emerging log standard, (official site now live!) one of the most popular: Finally, Common Event Expression (CEE) is Out!!!. My other CEE-related posts are labeled here.
4. Hurray to database logging (finally!) My posts related to database logging top the charts. Specifically, How to Do Database Logging/Monitoring “Right”? as well as its “prequels” :-) Full Paper on Database Log Management Posted and On Database Logging and Auditing (Teaser + NOW Full Paper).
5. Finally, security ROI saga that flared up mid-year is also among the most popular. Indeed, Security ROI Pile-Up! post made it into Top5 (the related posts are: The Entire Security ROI Blood Trail and ROI, ROSI, RROI and Harry Potter Tales). The rest of my ROI-related posts are labeled here.
6. At the risk of destroying my math credibility, I will add an item #6 to my Top 5 list, again. This little post called On Open Source in SIEM and Log Management have also generated a lot of traffic and discussion. Indeed, log management vs SIEM as well as reasons for a lack of a popular and complete open source log management solution are fun topics!

See you in 2009! :-)

Possibly related posts:

• Monthly Blog Round-Up - December 2007
• Monthly Blog Round-Up - November 2007
• Monthly Blog Round-Up - October 2007
• Monthly Blog Round-Up - September 2007
• Monthly Blog Round-Up - August 2007

Every once in awhile you’ll read another story about stolen backup tapes–with millions of confidential records–that are lost forever. Will someone steal your identity? Is the security of our nation compromised? We never know what happens to those tapes unless they’re miraculously recovered. I’d like to imagine that the unwitting thieves more likely destroy their ill-gotten booty because they tried just a little too hard to jam an LTO tape into a DLT drive (or worse yet, a VCR).

It’s human nature to believe criminals are stupid, which is never a reasonable assumption to make. Some people might write off these thefts as smash and grabs, when there is also consider the possibility that the systems administrator (and those backup tapes), were the intended target all along. The thieves break in, steal the tapes, but they also take your Cuisinart, your Xbox360 and your Veronica Mars DVDs to divert suspicion. The sysadmin is then busy working with an insurance claim, and nobody is reviewing the inventory of what was on those tapes.

There will always be sysadmins out there that take the backup tapes home. I’m not saying it’s outright bad, but it’s an understandable behavior. The most likely reason is that transporting the tapes from the tape library to an off site location is out of the way or inconvenient. If you’re at the data center with tapes in hand, and your house is 3 miles away, while the off site storage facility is 20 miles away; what would you choose?

There’s a policy at my current employer that attempts to reduce the chances of this happening. Tape changes can only occur before lunch, and never on a Friday. Eliminate the incentive to just drive home, and the tapes are more likely to end up at the proper storage facility. That works well in our situation, but your mileage may vary.

I worked at another IT shop, where off site storage was the sysadmin’s home. That’s okay if your employer is one of those 1 to 5 man garage startup wannabes. Just be sure to invest a few extra dollars and do it right; buy a safe. Make sure it’s fireproof, and large enough to accommodate at least 30 tapes. Then find out what your homeowners insurance will and won’t cover; a typical policy probably won’t cover the cost of the tapes in contrast to what you would get at a professional storage site.

The two technological solutions I’ll bring up include tape encryption, and remote mirroring. Encrypted tapes are less likely to be compromised by a thief, and remote mirroring of data to a separate facility eliminates the human transport factor altogether.

Tape backups and archives will always be susceptible to theft. The media is physically small enough, and valuable enough to always make it a tempting target. The sooner you come to terms with this, the better off you’ll be when it comes to setting policies and procedures to protect them.

There’s a Computerworld article out in the wild about the New York Stock Exchange (Euronet) building out 200 new Linux servers. Articles like this are usually puff-pieces, so technology people are always wondering about the real details behind decisions. I had a few questions about 2 interesting quotes made by CIO Steve Rubinow.

The first interesting comment was the desire to achieve “technology independence”. First, what are the benefits of technology independence to the NYSE? More to the point, how can an organization buy 200 servers of any architecture and remain technology independent?

True, there could be growth or a change in the direction, but if you’ve already committed yourself to 200 servers, you should have a good idea of what your future paths are.

The other comment was on Linux being “polished enough”, in comparison to Unix operating systems that have a 20 year history. What does polished enough mean? It almost suggests that the other OS offerings are better, but you’re settling for Linux ? What Linux features (if any) convinced you to use it?

Is there something Linux vendors could do to make a better OS offering? Or does it even matter since you’ve already committed yourself down this technology path? Should Linux contributors be happy that there is more business use, or should they be concerned that the reasoning behind using Linux may not come across as a glowing endorsement?

While it could vaguely be considered system/application news, Nintendo announced a new version of “Photo Channel”, will be released in December. For those that own a Wii, Photo Channel is that icon that is mostly ignored, right next to “Wii Shop Channel”.

The update includes the comment that they’ll be dropping MP3 support for AAC support, and it has a few Nintendo users a little upset. The rumor was that the change was brought about due to licensing costs for the MP3 codec, but nobody knows for sure.

I’m one of the few people I know that’s made the switch from MP3s into AAC files, and burn CDs directly into AAC. I didn’t gain anything in quality from the MP3 to AAC conversion, but there was a slight savings in hard drive space. I still have my music, and it can still be played on the iPod, the Zune, or other players put out by Creative, SanDisk, Sony, etc, etc.

So my question is, is the MP3 file format really more popular than AAC; or does it only appear to be more popular because the term MP3 is slowly becoming a lost trademark in the same vein of Xerox and Coke?

AAC is superior to MP3, since it is the successor format; but do people actually use the extended features? Or is this a case where Betamax is technically superior, but for some reasons the markets prefer VHS?

Mark Wade wrote a firsthand account on tracking a purchase that was originally solicited from a SPAM email.

There are a couple of points where the article is vague on technical details, but I’m probably not the intended audience. I found the article to be informative, but it also left me with more questions.

First, I wanted more details on the domain registrations that were encountered along the way. Are the domains still active, or are they on administrative hold? I’m wondering if spammers pay for domains with stolen credit cards, knowing they may only have a couple of days to use it; or if they keep a cache of domains that they cycle through, and just move the site contents from domain to domain.

Second, the article mentions that somewhere along the chain of purchasing his item that a SSL certificate was encountered. A trusted SSL certificate is going to be a little harder for a spammer to get compared to a domain name. Remember, the whole point of SSL certificates are to make Internet commerce safe, and to give consumers confidence with a system of verifiable trust. Unless, the cert is free, payment for a certificate is up front; and the certificate itself should have accurate owner information. The author never mentions who the signing authority was, nor did he mention whether or not there were any signing errors.

Third, he mentions a tracking log of the package, but does not reveal which shipping company it was. Since the package never arrived, I’m left wondering whether the entire log was a fake. Usually, a shipping company cannot provide tracking information for a package that goes through other shippers. For example, a tracking number for something shipped via Canada Post isn’t too helpful once the parcel is handed off to the US Postal System for its final delivery. The log shown in the article mentions multiple hops in China, and multiple hops in Virginia. Was the comment about the package being lost in the USPS a real loss by the post office, or sarcastic wit?

Compromising hosts to send spam is pretty easy; but setting up an actual commerce website to handle transactions risks more exposure. If a spammer really wants to make money this way, there has to be a path leading from the victim to the criminal.

At the same time, there are other companies involved in the purchasing process that create more leads. Domain registrars, web hosting providers, certificate providers, shippers, and credit card processors. As a consumer, it would be more interesting to me to know which of these vendors have more extensive track records of dealing with spammers. Are these companies being duped, or are they more swayed by easy money compared to building consumer confidence?

The original title of Mr. Wade’s article was “Following the SPAM”, and in that regard he seems to have done a good job. As a follow-up, I would recommend a more detailed investigation, and to repeat the advice once given to Woodward and Bernstein: “Follow The Money.”

The legal maneuvering between two of my more favored tech companies just kicked it up a notch. Johnathan Schwartz looks like he’s not going to take NetApp’s lawsuit lying down.

I’m not even going to pretend I’m a lawyer, but a counter-suit is probably a suitable response to the original NetApp suit. In the end, the best hope for everyone (customer wise), is a mutual understanding between these two behemoth companies.

There are too many companies out there that have major investments in NFS, ZFS, and Netapp filers. For enterprise users of any of these technologies (or all of them), a one sided victory probably sounds pretty scary.

There’s an interesting post on the Skype forums where a Linux user asks why Skype is reading his /etc/passwd file.

Since it was posted during the weekend, and there are Skype developers out there dealing with other issues, I’m guessing he won’t get a reasonable answer from a Skype developer until at least Monday.

It’s not my job to cover for Skype, but I’ll try to give a reassuring answer. More than likely, Skype is only reading /etc/passwd in order to get information about the userid it is running under. It may need to do this to determine its home directory, or to learn more information about the user.

If the programmers are doing it right, they’re using the getpwent() system call to grab the password entry through the name service switch. That’s because you may not be using a local password file at all. If you’re using a workstation in a large environment, your authentication information could be stored in LDAP, NIS, or another global password map. Ironically, when a program uses getpwent(), it doesn’t even know if you have a valid /etc/passwd file.

If you traced the call even further, you would see that the program is reading all of the entries in the /etc/passwd file. Since the file is sequential, there’s no way around that. Nor is there an easy way to tell what happens to the data once it’s read. Maybe it ignores the entries that it doesn’t care about, or maybe it emails them to an insidious hacker.

The /etc/passwd file can be read by any user, so it carries the bare minimum amount of security information. The actual passwords are encrypted and stored in the /etc/shadow file, which is then protected by the operating system by making it read-only from the root account. Reading /etc/passwd might give someone a slight insight into what accounts to possibly compromise, but it won’t offer any special insight on how to compromise them.

On a final note, although I encourage the testing and using of security tools, like AppArmor, I think its use in this case is unneeded. Chris Brown said it best in Protecting your applications with AppArmor:

AppArmor is not intended to provide protection against execution of ordinary tools run by ordinary users. You already have the classic Linux security model in place to constrain the activities of such programs.

Any regular user on a Unix host, should rely on the host operating system security model. I’ll admit, I feel more comfortable saying that about a Unix/Linux/Mac host than I would for a Windows host. If Skype–or any application–tries to perform an action that it isn’t allowed to do, the operating system should prevent it.

The recent change to German law to implement the EU Framework Decision on Attacks against Information Systems (enacted in Paragraph 202c of the German Penal Code) has caused many security researchers based in Germany to look to move elsewhere, or to remove previously available research findings.

The change in the law, which went into effect on August 10, criminalises the production, distribution, possession, and sale of tools that can be used to commit cybercrimes. Unfortunately, a strict interpretation of the changes would make possession of tools that could be used maliciously (such as nmap or Nessus for instance) illegal. While in reality, legal opinions are that the courts would differentiate between a cracker and a security researcher based on their intent, noone (unsurprisingly) seems to want to be the first test case.

The content for a number of projects have all but disappeared, such as the recent Month of PHP bugs, and the well known THC (The Hackers Choice) group, as well as smaller projects such as BtCrawler. Others are saying farewell to Germany and reestablishing themselves elsewhere such as the KisMac wifi scanner for OSX and the Phenoelit group.

All in all a hard strike against a country which has produced much valuable security research and expertise.

Every once in awhile I will have maintenance windows that need to start at Midnight. When announcements are made, there’s always going to be one person out there that gets the day wrong. If I say the system will be unavailable Wednesday at Midnight, I will be asked if I mean Tuesday night leading into Wednesday morning, or Wednesday night leading into Thursday morning.

Technically, Midnight is the beginning of a new day. If you’re used to military time, it’s easier to understand, since you’re starting over at 0000. But most people have always used Midnight in the context of a late evening, as in, I’m going to stay up until Midnight; so it’s pretty common to accidentally associate it with the previous day.

To overcome confusion, we announce all Midnight maintenance windows as having a start time of 12:01 AM. It’s close enough to Midnight, and for some reason, everybody seems to understand it. Sure, Midnight is ambiguous, but 12:01 on a Wednesday is clearly early understood as Wednesday morning. I’m not going to complain about a one minute compromise if it means avoiding ten minutes of writing emails to clarify the schedule.

Most data center operators require all their servers to have static IP addresses (or they pin a dynamic address to a specific hardware address, which is essentially the same thing), and most also require them to have meaningful names. This makes a lot of sense for a number of reasons.

First of all, you don’t want someone to be able to come along and plug in any old device into your data center network. After all, this is the pipe that’s most likely to be pumping sensitive information between databases, mainframes, and web servers. By requiring a static IP or a whitelisted MAC address, you make it less likely (but not nearly impossible) for a rogue device to be able to access all the network configuration it needs.

Second, it’s important that consumers of data center services always know how to connect to those services. Granted, this is becoming less important today as these services are more likely to be clustered and hidden behind switches. But a good data center operator always wants to know how to get directly to any of her boxes from a secure shell.

Third, speaking of secure shells, if a server ever does change its IP address or other aspects of its identity, you’ll get an annoying but serious error message.

I’m sure there are many other reasons. And I hope some of my readers will share them with me.

The new thing is that resources in the dynamic data center can be created, provisioned, deprovisioned, and destroyed with no intervention from an operator; old policies about naming and addressing no longer apply. In the dynamic data center, virtual machines appear, do some work, and disappear on the fly. Depending on the virtualization system, a new VM may be assigned a random MAC address just before being loaded for the first time. There is no opportunity for an operator to assign a static IP address to a virtual machine or to configure a newly-created virtual MAC address in a DHCP whitelist.

Data center operators are going to need to change their view of the world. Systems management software vendors need to propose a better way.

I work in academia. I’m a sysadmin. However, I took a rather non-traditional route to sysadmin-hood. The very brief version of the story goes like this:

I started as a lowly database reporting geek. I found that I liked databases, and the database guys took me under their wing and made a DBA out of me. Then I went to work for Sybase and became a full-blown data snob. However, on various client sites I found that the people I inevitably needed to interact with to get my work done were the sysadmins.

As time went on I gained their trust and they started giving me more privileges on the machines where the database servers were running. It was a great help, and I was able to spread my sysadmin wings in an environment quite close to what would be called “production”.

After about a year of Perl scripting on the database end and for the sysadmin tasks I needed to perform, I decided that what I really wanted to do was build out these huge end-to-end systems. I wanted to do everything. I wanted to much with network gear, environmental monitors, power distribution, UNIX, databases, LDAP, Apache, and whatever else I could get my hands on. My next job was for a consulting firm that put me through tons of systems training, and I was on my way.

When I wasn’t at work, I was reading books about Linux and UNIX (I think I read all of the ones available at that time, actually), pounding the Linux forums, and setting up services on Linux boxes I had set up at home. I could rattle off ipchains rulesets in my sleep, recite Apache rewrite rules verbatim, quote error strings and tell you what they meant and how long you had before your disk just completely failed. I could set up quad-boot machines, run Linux on old SPARCs, and I had already written code to handle most of the basic admin tasks, as well as some basic monitoring.

In short, I was determined. I had given up any notion of a life, hacked day and night, read Phrack, 2600, the llama book, bat book and more, grokked perl, php, sed, awk, tr, ed, vi, ksh and bash, and had gotten myself ready to sit for the SCSA, SCNA, CCNA, and CheckPoint certifications after working in IT, mostly in database administration and development, for about 3 years.

I still felt like a flaming idiot. Heck, there are plenty of occasions where I feel pretty dopy now! Luckily, my sponge-like brain has shown no signs of becoming saturated.

My main goals now boil down to doing my best to fight specialization. I don’t want to stop coding Python, or doing database development, or maintaining LDAP servers, or building beowulf clusters, or maintaining VMWare servers. Aside from these goals, I also try to share as much of the knowledge I have with others who might be where I was 10 years ago by putting it here, or on my blog, or on my Linux admin site, or other sites, or on IRC, or the forums, or in magazine articles, or in slide presentations for LUGs. Oh - I once put some of my knowledge in a book, too!

What I want to know now, though, is this: How did you get here? Did you major in CS and choose systems work? Did you do something non-technical and became your company’s all ’round IT guy? Did you fight your way out of the phone support farm? Let me know! Share your story!

Imagine you bought something.

• You rely on it with your business, with your very livelihood. Sometimes even with your life.
• There is no warranty whatsoever on what you bought.
• And you don’t know what’s inside the box.
• Also, you cannot look inside the box, in fact, it is illegal.
• You might not have have heard about the seller before and you have no particular reason to trust him.

Are you totally and irreversibly mad? How can you do it?! If you are not mad, aren’t you criminally negligent? Or just very, very, very stupid.

However, we all are. We all bought software at least once in our lives …

This blurb is inspired by some discussions I had at CONFidence 2007 Conference (where I presented on “Log Forensics” in front of about 180 people). Another related fun thought I picked there is that the most scary cyber-criminal of the future is not a spammer, a scammer, a phisher or a pharmer, and not even a good ole “cracker” - it is an unethical software engineer, who changes the code slightly to introduce a weakness (or a full-blown backdoor or a logic bomb) and later uses or sells this knowledge. In light of the above characteristics of software purchases, think billions stolen in one shot, think ruined companies, think stock market manipulation, think direct physical damage (and, yes, real cyberterror), etc. We do live in interesting times …

The electron, smallest of the three particles that make up the atom, how often do we take this plucky little lepton for granted? But let them stop flowing through the wires leading into our computers, and we quickly realize just how dependent we are on them. This lesson was brought home to me last night, as a late season snow storm took out the power to my house for 4 hours.

This isn’t the first time that PSNH (Power Shutdowns Noticeably Happen) has failed in their contractual obligation to keep the electrons flowing to my house. In fact, we lost power for 3 days in late January, and for 4 days in 1998 during the ice storms. I have pretty much everything in the house on a UPS (7 in all), and have a auto-starting generator on the budget for this year.

But once we take care of something, we tend to put it out of our mind. ” No need to worry about power failures, I have a UPS…” But we forget that most solutions come with new problems of their own. I got a practical demonstration of this fact when I started to hear a loud annoying chirping coming out of my rack a couple of times a day, lasting a minute. With that much stuff in my rack, there’s a lot of things that can make noise, but I surmised right off the bat that one of my UPSi was trying to tell me something.

Sure enough, one of them had it’s “replace battery” light on. Rather than replace the battery, I took the opportunity to order a 1500 VA rack mount UPS, since traditional “cinderblock” UPSi don’t really work well in a rack. Besides, the rack mount ones look cooler. That UPS is due to arrive via, well, UPS today. Unfortunately, when I needed it last night, I was stuck with the old one. It immediately started to chirp in a rapid, panic-inducing manner. Luckily, that UPS doesn’t power either of the two systems in the rack, so I didn’t need to worry about an abrupt shutdown on a PC. It did however power my brand new 24″ Gateway monitor, which began to flash on and off about once a second, as the UPS failed in an interesting mode.

There are two lessons for SysAdmins to take away from this. Firstly, a UPS is not a buy-once and forget item. You’re going to need to plan in advance to replace the batteries as they age. In a medium-size datacenter (one large enough to have lots of racks, but small enough not to have centralized conditioned power), it probably makes sense to standardize on a single model of UPS, and keep spare batteries around. I also learned something I already knew but had forgotten, you should have all of the critical items for a PC attached to the same UPS as the PC. Because the USB hub that my keyboard and mouse were attached to was on the flaky UPS, I had to scramble to attach them directly into the back of the PC so that I could shut it down.

I read an article on Slashdot the other day about an newly released open source application. I read a few of the comments, and I found this one (slightly paraphrased):

$ apt-get fooapp
"You have searched for packages named fooapp in all distributions....Can't find that package."
Sorry, I'm not interested.

The comment suggested that since he can’t try the software in a pre-built distribution then it isn’t worth trying.

Unlike a few years ago when every Linux user ran configure by hand, the speed and convenience of installing packages has put the compiler on the back burner. Packages aren’t a bad thing, but I think it’s a poor reflection of an administrator’s skill set if they shun the development tools that are available for every Unix environment.

I’m not saying that packages should be avoided. I build them myself for software that I have compiled and tested manually. Once packaged, they’re pushed out during the remote installation process. After that, I am considered the distributor for certain applications in the infrastructure, and the primary support contact. I’ll also concede that it doesn’t make much sense to recompile Gnome or KDE if my OS vendor provides a pre-built package, along with support and regular upgrades. I won’t be too optimistic about installing packages that aren’t built or approved by the author of the software.

When I interview an administrator, I usually throw in a couple of programming interview questions, such as, “How do you determine which shared libraries a program requires?” or “What steps would you take to compile the Apache webserver?”. I don’t expect them to be a full fledged C programmer, but I think it’s important to know how to build software. The candidates that have demonstrated these skills have also been more proficient in debugging, tracing system calls, and identifying performance problems ahead of time.

Ok, so I’m not completely sold yet. I still have a boatload of Perl code floating about, and for certain things I’m still writing *new* Perl code. However, I was coerced into using Python for a project I’m working on, and I have to say that I think Python is coming on to me.

I try to ignore the furtive glances, and those times when I could swear it’s actually winking at me. I don’t acknowledge the beguiling smiles and greetings I get from Python when I open my laptop. I just get down to the business of coding and pretend none of it ever happened. That pretending is getting harder by the day.

Here’s the thing. I’ve been doing all of my sysadmin scripting in perl, awk and shell (sometimes together) for a decade. After 10 years, Perl still doesn’t even say “hello” to me. It seems to stand ready to spit my own code all over me whenever I try to talk to it. And just when I’m about ready to call it my friend, just when I think I know it, it completely changes. Well, I’m tired of it. I’m tired of the schizophrenia. I’m tired of the attitude. I’m tired of feeling like a Perl n00b after using it for 10 years.

I’m leaving.

Of course, like lots of relationships, it’s complicated. Over the years, Perl and I have spawned offspring that aren’t going to just disappear because I decide I don’t like Perl anymore. I promise to care for them and keep them up to date.

But from this day forward, I’m going with Python in those places where I can. I *want* to feel confident with a language. I *want* to take advantage of code reuse, self-documenting code, and OO design principles. I *want* to have readable, concise code. I *want* to solve problems that are larger than the every day “please change my shell” requests. I want to build tools. I want to architect solutions. I want to solve some of the problems sysadmins face, but I had to solve my own big problem first: namely, being a self-hating Perl slinger who was never particularly comfortable with how Perl, at a very high level, works.

If you’re an admin using Python on a regular basis for your admin scripting, let me know how you think it compares to Perl for equivalent tasks. If you’re a Perl coder who has tried and *not* used Python for reasons besides the lack of curly braces, fill us in! If you’re a religious zealot for one language and have never used the other, feel free to move on!

If you’re not managing your time, it’s almost guaranteed that you’re misusing it. I’ve heard, and even used, tons of excuses in the past to not manage my time, but eventually you’ll come to the realization that saying you don’t have time to figure out a system to manage your time is just ridiculous. For those of you who feel this way, I recently blogged about the system I’m building to manage my time in a completely digital way.

I used the old Franklin planner for quite some time, but I eventually realized that since I have a blackberry, and a way to synchronize in a “round trip” fashion from the blackberry to a task list, to a web-based calendar, and finally back to a local calendar, there would seem to be no gaps left for the dead-tree planner to fill.

If you found other tools useful or you’re still hanging on to the Personal Analog Assistant, please share with us your thoughts on the tools you’re using or the gaps that you still haven’t found a digital solution for.

I’m lucky to be in a position where I am not forced to specialize on a single technology. I have always made a habit of keeping up with the job market, and it seems the trend is that the bigger the company you wind up at, the more likely you are to be staring at the same thing day in and day out.

There are people even less specialized than me - my buddy is a one-man IT shop. He’s in an environment that could probably use just about one more IT guy. He’s probably wishing for the luxury of slightly more specialization that might be afforded to him if there was someone else around to help out.

There are people more specialized than me, too. A family member of mine is just about the hardest of hard core WebSphere admins. Oh, he’s a UNIX admin, but he got a job at $HUGECORP and I get the impression that he hasn’t seen so much as a zone file ever since. It’s this level of specialization I’m fighting.

Last week, I spent the week doing database development (culminating in a bit of SQL goodness you can read about on my blog) This week, I’m replacing an aging print (CUPS/Samba) server with a virtual machine. After that, I’m afraid I have a little PHP code to write for Moodle, and then I’m back into the back end on a data warehousing project. Before February started, I was upgrading (read: PXE kickstarting) a 30-node teaching lab, helping debug some weird issues on a beowulf cluster, and retiring the first beowulf cluster I ever deployed.

Over this coming summer, I hope to migrate more services from our old NIS service to our new LDAP service, which I deployed using Fedora Directory Server about two months after it was released to the world. I’ve blogged here that I was happy about that. I still am :-)

…And I’m still a little bitter that I rarely get to touch our networking gear.

Oh yeah, one other way I’ve been able to fight off specialization is by consulting. I have clients that need things set up that I have no use for or have outgrown in my day-job environment. This experience has even provided a couple of useful experiences that I brought back with me to my day job, and certainly keeps me grounded in the realities of how other organizations go about making technology decisions, and how those decisions affect the business side of things.

What about you? Are you able to fight specialization in your work? Have you suffered or been saved by specializing/not specializing? Have you turned down bigger paychecks to avoid specialization? Are you sorry you did/didn’t take a highly specialized job? I’m interested in your views on this! Share! :-)

There has been quite a bit of fall-out over my post on the lack of evolution in system administration. Most of it’s about what I expected — “right on!”, I should have been more rigourous, there was too much hyperbole, I’m wrong, I’m insane — but it was still interesting watching it all happen. I’ll go into a bit more detail below the fold.

There’s a lot of good, interesting discussion happening on this blog as of late. The state of system administration tools, in general, is pretty poor. “Taking care of business” is still very dependent on the ingenuity and creativity of the individual administrator. Maybe being able to be creative is one of the things you like about administration, but wouldn’t it be nice to solve a problem more interesting than, say, installing emacs on 30 machines without working up a sweat?

And sure, more sysadmins could probably participate (or initiate) more projects, or provide more feedback. And more sysadmins could probably publish more of the bits of code they write than they do. And more of us could probably be more productive if we could find a way to interact with more admins, thereby realizing that we’re really *not* the only people who could ever make use of that 20-line piece of perl mastery. I agree on all of these points. However, I think much of this dances around a few large,
multi-colored elephants in the room. Flame if you must, but allow me to point them out for you….

Since fellow O’Reilly blogger Luke Kanies mentioned me in his excellent piece Why Isn’t System Administration Evolving? I thought I should clear up some misstatements.

My post on the fact that I don’t think system administration is evolving seems to have ticked some people off on the lopsa-discuss list.

Sure, I was a bit flippant in my post, but I think many of the critics are falling into the common trap of thinking that tools vary in functionality but not in fundamentals. Paul Graham does a great job discussing this in his article about beating the averages; it is about languages but it’s just as valid here. Just like languages, tool power varies considerably.

I don’t just mean tool functionality, that is, the specific features of the tool; I mean the overall power. I think of this variance in power as deriving from difference in models: Tools with low-level models are not very powerful.

For instance, take one of the big differences between cfengine and Puppet; cfengine focuses mostly on managing textfiles, while Puppet manages semantically more powerful constructs like users, services, and packages. Thus, Puppet has higher-level models than cfengine, and if you accept my premise that model power derives from how high-level it is, that generally makes Puppet a more powerful tool than cfengine.

Note that that doesn’t make it more functional — it could be less stable, there could be lots of interesting things it can’t do, whatever.

This is one of the things I think are really lacking in the sysadmin world: High-level models. Do you think in terms of /etc/passwd, or users? If you think about /etc/passwd, then LDAP is fundamentally unrelated, but if you think in terms of users, then they’re functionally equivalent.

It’s not just about how powerful the modeling of the tool is, though, it’s about the power of the models it creates.

Take dependencies and relationships between resources. Are there any sysadmin tools that effectively model these relationships? Of those tools, do any of them make the information exportable to other tools, so you’re encouraged to spend a lot of time developing these comprehensive models? No, not really. Sure, some of the monitoring apps allow you specify basic dependency information, but they’re extremely limited, and you can never export them so they’re useful elsewhere.

What really sets Puppet apart from Cfengine isn’t that it models higher-level resources instead of files, it’s that it allows you to model the relationships between those resources. Puppet’s lowest layer is responsible for the resource modeling (e.g., what it means to be a User on Solaris vs. OS X), but its language handles the high-level relationships between those resources, including how they’re grouped (e.g., they’re all part of the class of resources that makes apache work) and how they relate (e.g., apache should restart if its configuration file changes, and changes to the file should happen before the service is checked).

This is why better tools are so important: The models our tools use set an upper limit of the size of the problems we can comfortably work with. Unless we develop a whole slew of new tools with more powerful models, we’ll be forever stuck thinking about bits on disk.

That means it’s not just about configuration management, it’s about everything on your network. If my configuration management system can’t model the network, then I can’t set up IP addresses or firewall ports; if it can’t model the backup system, then I can’t provide guarantees of data availability or perform automated restores; if it can’t model change, then I have to do all migrations manually.

These models are critical, and they’re all in our heads instead of in our tools. This fact is a severe limitation on our field’s ability to move forward.

Now that I’ve gotten the introduction out of the way, I can get to the meat of why I’m blogging here in the first place. This is a long one, but I think you’ll find it’s worth it.

I’m not writing Puppet because I think I’m right or whatever; I’m writing it out of desperation, because no one else is even trying. Not only are people not trying to make better tools than we have available today, they’re not even using the crappy ones we do have available, which is just sad. Imagine if the computing world had just refused to write any code until C (or better yet, Ruby) showed up; where would we be now?

I’m less interested in why sysadmins don’t use the existing tools, though, and more interested in why they don’t publish their own. Most of the rest of the technical world seems to have figured out how to solve their problems using code and then how to turn that code into a self-sustaining project, either open-source or commercial. Sure, this stuff was complicated fifteen years ago, but it’s pretty straightforward now; and yet, instant messaging tools have larger development communities than sysadmin tools, and the majority of sysadmins spend their days toiling with a bunch of little one-off scripts that no one else will ever see or use and that the next sysadmin will gratefully /dev/null as soon as possible.

I’ve heard all of the standard excuses — we don’t have enough time, we can’t risk it, we spend all day doing computers and don’t want to do it at night, my company won’t let me, etc. Every software project that has ever evolved out of an internal project has exactly these same excuses, and yet they have somehow succeeded. Why have so few sysadmin tools evolved this way? Why are sysadmins so willing to believe their excuses?

I don’t jump on over-crowded bandwagons very often; because everyone out there has thrown out their view on what the unseen iPhone will be like. I’ll admit that there is one aspect of the iPhone that I’m curious about, but nobody seems to have commented on.

If you run OS X, you have the Address Book application, iChat, and iSync. All of these programs perform some form of contact management and centralization of information. If you run Windows, you have a few PIM applications to choose from, but there’s nothing that really stands out as the end-all, be-all contact manager.

If there really is an iPhone, I’m guessing that it will also come bundled with a really good phone management application. It will provide contact management and syncing, management of other information such as images, ring tones, application settings. It might also provide the ability to control the phone directly from the computer. Whatever this application is, it will be released for both the Macintosh and Windows environments. Very few of the current cell phones come with syncing applications, so this would set them apart from everyone else in the market.

If the phone comes with a camera, and an iPod 30 pin connector, then you open the door to having a usable web camera for video conferencing applications like iChat. That prediction might be a bit of a stretch, but I’d have a hard time imagining that nobody at Apple has never considered the idea of soft phone applications while designing a hardware phone.

That’s all the insight I have at the moment. I guess I’ll know for sure in a few hours.

This piece, even though it borders on being content-free at times :-), has a few fun insights on logging and audits (some painfully obvious to us logging people :-))

For example: “Does the fact that audit logging is enabled satisfy all the different regulatory compliance requirements? […] In all but the most generic cases, the clear answer to this is no.” But of course! Having logs is great, but looking at them is awesome (and sometimes required - see PCI Requirement 10)!

Or this “gem” - “The problem is no one looks at these logs unless something bad happens” which goes straight back to my famous (and old) “Five Mistakes of Log Analysis” paper (an update, called “Six Mistakes of Log Management” is to be published soon) Having a log management solution solves this one nicely.

Or this one: “Audit logging is not just for compliance reasons.” Guess what, every sysadmin worth his salt have known this for, say, 20 years :-)

Overall, the piece is much better at asking questions than giving answers. Seriously, is this true: “What do you log? Do you look at successes and failures? How vulnerable are your logs once they’ve been written? How long do you keep your logs? Only you can answer those questions.” No, there are pretty good answers already given to the questions above. Such answers are given in various regulations (some mentioned in the article) as well as “best practice” and IT governance frameworks, such as COBIT, ITIL and various ISO docs.

Mike Rothman kicks it as well by saying “Kevin Beaver beats the horse a bit too much about why you should log (providing the regulatory context) and not enough perspective on how you should do it.”

In any case, I liked the blurb since it helps to bring awareness of log management to those still hiding from it under their desks…

Here are the results of my security conference poll, as of 11/16/2006 (the results hasn’t changed much since). Which information security conference do you like the most?
		FIRST (46%)
		DEFCON (12%)
		BlackHat (10%)
		SANS (all) (7%)
		Other (7%)
		CanSecWest (5%)
		RSA (4%)
		Gartner IT Security Summit (1%)
		ISACA (0%)
		Security Decisions (ISD) (0%)
		CSI (all) (0%)
		MISTI (all) (0%)
		ISSA (0%)
		TechnoSecurity (0%)
		226 total votes

What are the conclusions:

1. The link to a poll was posted on a FIRST web site. I will let the reader decide the causality here :-)
2. DEFCON/Blackhat still rock!
3. SANS is great as well, presenting a close adjusted (see item 1. above) second after DEFCON/Blackhat
4. Some shows where we usually notice an abundance of fake experts presenting their rants are rated low, as they should be
5. Some Gartner folks blessed the poll with their votes :-) (Hi, Rich!)
6. Looks like I totally failed to spell out some of the popular shows, since Other category is so big (please, please, dear readers, post comments and enlighten me on this)

Obviously, the results reflect the bias of my readership selection, but, at the same time, they are not entirely unexpected…

Here is a quick poll “Which information security conference do you like the most?” Please vote, I am trying to see which conferences folks like nowadays; I promise to post the commented results here…

Now, this is one of’em philosophical posts, but with direct relevance to system administration, security and audit, thus I am reposting here (original and some discussion of it that occured elsewhere). After all, I do have to justify the “Ph” in my Ph.D., right? :-) At the same time, this post will have an unmistakable stench of a rant :-) for some of my readers.

Recently, I was involved in some fun discussions on storage security. And, in most cases, you store “stuff” to let others access it, not just for archival or - gasp!- compliance purposes. One of the storage vendors I talked to recently mentioned that every year they’ve been in business (since early 90s), they have to add one or more audit features to their information access solution to increase the level of details, performance of their audit logging or whatever other audit related feature.

My response was: “What? You didn’t build them from the very beginning?” And then I thought: why provide access without audit logging?

No, really, why have it?! Disks are cheap, bandwidth is affordable, CPUs are powerful: why provide access to any information without having an ability (at least) to log each and every successful and failed access?

Before some of you label me “a privacy Nazi”, I have to disclose that I am somewhat of a fan of Scott McNealy’s saying “You have no privacy. Get over it.” Having access audit info is useful in so many cases, that not doing it becomes inexcusable and, frankly, stupid. Some of the many uses for such information are:

• Operational troubleshooting: knowing who failed to access the info and why
• Policy audit: who accessed what, with or without authorization?
• Regulatory compliance: legal requirement to have audit data is there to stay
• Incident response: what info got stolen and by whom?
• Information access trending and performance optimization: are we providing quick and reliable access to information?

So, what about privacy? Privacy is defined (in Wikipedia, where else) as “ability of an individual or group to keep their lives and personal affairs out of public view, or to control the flow of information about themselves.” We see two completely different things here: keeping the info out of public view and controlling the info about you. The former is clearly reasonable and possible. How about the second? To be honest, it sounds like a sheer idiocy to me, because you do not control it and never did. You’ve got to a) become invisible and b) stay home all the time :-) for a fair shot - albeit not a certainty! - at controlling the info about yourself. I can still talk about you - and thus control the information flow about you - by saying “ah, that invisible guy that stays home all the time!” :-)

So, what is the connection between the above definition and my call for “no access without logging”? Logging is NOT a privacy risk; inappropriate use for collected data is. Before you object by invoking the infamous “guns don’t kill people; gaping holes in vital organs do” :-) I have to say that the above privacy definition is about access to information about people, not about the existence of said information. And, yes, Virginia, there IS a difference!

Similarly, nowadays many folks are appalled when they see stuff like this (”Fresh calls for ISP data retention laws. US attorney general cranks up the volume.”), but it actually - gasp! - seems reasonable to me, in light of the above. Admittedly, if your bandwidth is so huge that you cannot log and retain, you might be able avoid logging or at least avoid long term log retention, but that is a different story altogether.

Another thing that is tied to this is the whole “privacy vs security” debate which never made quite sense to me - until now. This is indeed the area where those who want to have logs for security and other uses will clash with those who don’t trust controls on the collected log data and would prefer for such data to never get created in the first place. But that would be a subject of a follow-up post later….

So, have doing security and especially log analysis for whatever number of years gone to my head? Or am I onto a critical trend here? Comment away!!!

tags: security, storage, access, audit, privacy

About once a week, a ticket works its way down the grapevine. User cannot upload a file by FTP. It’s always a different user, but the problem is the same. I don’t think it’s the frequency of the problem that’s frustrating, it’s tracking the source of the problem.

What are their firewall settings? What are our firewall settings? Are they coming across the VPN? Is it an active or passive data connection? Is their FTP script sending the wrong password again? Are they hitting the proftpd server or the wu-ftpd server? Hey, this FTP daemon log doesn’t report any connection details!

In the end, there’s always a different root cause. Maybe it’s a NAT translation failure, or a load balancer that needs to be rebooted. Next week it will be a totally different scenario, and a totally different solution.

FTP is an antiquated protocol, designed to address early shortcomings in the IP protocol over 25 years ago. In my humble opinion, it’s time to give up and let this protocol die. There are other protocols out there that can do the job just as well, if not better. Here are the top contenders:

SCP/SSH. Secure authentication, and data encryption to boot. SCP is available on almost every platform out there. There are even a few decent SCP GUI clients out there. The only downside is there aren’t too many tools that can script SCP uploads on a Windows host.

SVN/CVS/Other. Okay, these protocols are a little restrictive. They allow for file uploads and downloads, but their main function is versioning control. Neither of them will be helpful if you’re trying to perform simple file transfer operations outside of a centralized repository.

Jabber. Not a popular protocol, but something to keep in mind. It would be incredibly easy to set up peer to peer file exchanges between client/servers or client/client environments.

HTTP. This should be the obvious choice; I know it’s my personal favorite. HTTP has authentication mechanisms built into the protocol, SSL is available, download recovery is possible, and servers can design fancy HTML interfaces for uploads and downloads through a web page. Plus, almost every programming and scripting language out there has a HTTP client library to facilitate scripted actions.

Probably the only thing that doesn’t work well is HTTP upload through a web browser. They will upload files, but web clients like Firefox (and others) don’t bother to report upload statistics like bytes sent or time remaining. Anyone uploading files a file is stuck in feedback limbo. Is my file being sent? The blue E is just spinning.

If somebody out there develops an extension or program to make HTTP uploads easier, they will have my immediate gratitude. I’ll email the program’s webpage to every remote user I have, with a side note saying that they will no longer need to send me their entire firewall configuration to debug problems in the future.

At some point, you’re going to be disappointed in somebody that provides you with products or services. Well, maybe you might be one of the lucky ones who never has to deal with that. The odds, however, are in my favor.

I have a hardware supplier that I truly appreciate. Their equipment is top notch, reliable, and it makes my job incredibly easy. And I’ve also had great support from that company…. until last week. For the past seven days, I dealt with random reboots, unanswered tickets, and parts that were never shipped. Because of this lapse in service, I am writing a letter of complaint.

In my case, I’m not about to dump vendor X for replacement Y. I have six years of good reliability and support behind me, while this is just one bad incident. My goal is to make sure my concerns are addressed, and that the company makes steps in preventing this in the future. I’m disappointed, but I still have a degree of faith in this vendor. Ironically, I did talk to one service technician who was very reliable, and did a great job in helping me. One person did a great job, but my overall feelings were that the support system had failed.

Once my ordeal was finished, I spent about five hours putting together a letter of complaint (If you’re chromatic, you know it takes me forever to edit my own stuff). From what I have learned, and studied from my old business writing textbook, I’ll give you some advice on how to write a complaint letter.

1. Make sure you’re addressing the right person. Talk to your VAR, or your salesperson. They will usually point you in the right direction since you are a valued customer (Unless of course, you’re complaining about your VAR or salesperson). If you’re not communicating with the right people, your letter may be totally ineffective. Don’t assume that writing directly to the CEO of a company is going to give you more attention.

2. Detail everything. Try to explain the situation as accurately as possible. Provide dates, names or references when applicable. The details will act as your own retelling of events, and may provide a huge contrast compared to any internal notes or memos on the incident.

3. Explain the basics in the first 2-3 paragraphs. You might have a long winded explanation of what happened, but try to make sure your person knows the reason you are trying to communicate with him or her. You need to literally capture the attention of the person you are writing to. If your vendor’s product was determined to be the root cause of the halon discharge, that should probably be mentioned at the beginning.

4. Explain why you feel things went bad. The vendor might see things differently from your point of view. From their perspective, they may only see a single service ticket in a queue that was closed within 24 hours. They may not understand the details of what happened during that time, or how this problem has affected your business. If you’re dealing with trouble tickets, also keep in mind that they may have internal notes that you might not see if you also have access to the ticket.

5. Give praise when they did something right. If you had great support, or any positive experience, compliment them on it. While it is important to know what a company has done wrong, it is also important to know what they did right.

6. Venting frustration is not the same as all out anger. If you’re intent on maintaining a relationship with this vendor, maintain a sense of professionalism. Your needs will not be addressed if the addressee of your letter thinks your a jerk. If anything, the relationship will probably go downhill.

It’s safe to say that when someone receives a box full of rattling bits of a circuit board with a note, “Enclosed is your crappy raid controller”, that you have no desire to further pursue a relationship with this vendor.

7. Don’t forget that you are representing yourself and your company. Written correspondence is still held in high regard over email. Depending on what you write, your letter might be framed, it might be quoted in the company newsletter, or it might be tacked up to a bulletin board in the break room with red pen marks correcting grammer and spelling errors.

Always show your letter to a higher up if possible, and don’t be afraid to let somebody else proofread it.

I love free software - free as in freedom. I’ll take a free beer too. I didn’t know about those things back in 1998 when I finally got Slackware to boot on my DEC Prioris and suddenly had a fast version of , ah, UNIX. Naw, this was free .

I immediately attempted to work that server into a project in an all Microsoft shop at Cap Gemini US. Somewhere along the line one of the tech support guys grabbed it and locked it in a closet where others attempted to scavenger it. I got it back and took it home where it eventually became my workstation.

One of the cool things about Linux? Remote administration. I could log into the server from afar and do stuff. I didn’t have to go down to the data center, use a KVM switch and look at a NT GUI.

I remember remote administration, industrial strength servers running on Intel processors that scaled and lots of developers working on the core system. Those were the days when geeks were men!

Well, I just got another dose of Microsoft in the form of Fedora Linux. I started installing their free directory server and what a mess. Actually, it’s a kluge. It’s broke like Red Hat 5.0 was broke. But now, you have to use a KVM switch and look at a desktop even after you figure out how to make it work.

By the way, kluge has another spelling and here’s the definition:

(I found a definition of kluge on Wikipedia that I like better than any I have seen before)

A kludge (or kluge) is a clumsy or inelegant solution to a problem. In engineering, a kludge is a workaround using unrelated parts cobbled together. People demonstrating the force of the term often say that it takes a skilled craftsman intimate with the task, the material at hand, and the operating environment to construct a workaround clunky enough to be called a kludge.

That’s the definition of Fedora Directory Server. It reminds me of the broken NT servers on which I got certified - that’s in passing tests not crazy. Broke and in need of workarounds.

Last week, we attempted to get the console to work but to no avail. What was wrong? Here’s what the documentation said and what we did to work around the problem:

Try to start up the admin console (this is a graphic user interface, so you’ll need to be running X):

# cd /opt/fedora-ds
# ./startconsole -u admin -a http://server1.centralsoft.org:12853/

If you get this sky-is-falling output:
GC Warning: Out of Memory!  Returning NIL!
GC Warning: Out of Memory!  Returning NIL!
GC Warning: Out of Memory!  Returning NIL!
Exception in thread "main" java.lang.OutOfMemoryError
   --No stacktrace available--

then you have a Java package problem.

As this was written, the Java packaged with Fedora (gcj) did not quite work with Directory Server, even though it’s version 1.4.2. Check http://directory.fedora.redhat.com/wiki/Install_Guide to find any updates or fixes. If the problem has not yet been resolved, change one line in the start-console script. Remove the -ms8m and -mx64m options (bold in this example to help you find them) from /opt/fedora-ds/startconsole.

Of course, that didn’t work.

This week we get a new message from the folks at Fedora:

Java runtime. The JRE is required in order to use the Console. Either the Sun or the IBM JRE version 1.4.2 or later is required. Unfortunately, the console does not (yet) build and run with the open source GNU gcj/Classpath java implementation, but we are working on it. We thought that gcc/gcj 4.1 included with Fedora Core 5 would work, but it still has many problems, so your best bet is to use Sun or IBM JRE.

So, I built Java the hard way. I wound up at another URL and it said:

Fedora Core 4 users are advised not to use the Java RPM provided by Sun. It contains Provides that conflict with names used in packages provided as part of Fedora Core 4. Because of this, Sun Java might disappear from an installed system during package upgrade operations. Fedora Core 4 users should use either the RPM from jpackage.org or manually install the Sun Java tarball into /opt. Sun Java 1.5+ is recommended for stability purposes.

I won’t put you through the pain, you can go to http://fedoranews.org/mediawiki/index.php/JPackage_Java_for_FC4 and see it for yourself. I built it even though the contributor wrote the instructions for FC4. I found broken links. So, I went looking for the packages and found them on other sites.It all worked.

But here’s a rub. Openoffice.org doesn’t work. But a workaround exists:

The JPackage java RPMs support switching between java implementations using the "alternatives" system.

[localhost ~]$ sudo /usr/sbin/alternatives --config java

There are 3 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
   1           /usr/share/java/libgcj-java-placeholder.sh
   2           /usr/lib/jvm/jre-1.4.2-gcj/bin/java
*+ 3           /usr/lib/jvm/jre-1.5.0-sun/bin/java

Enter to keep the current selection[+], or type selection number: 2
[localhost ~]$ java -version
java version "1.4.2"
gij (GNU libgcj) version 4.0.0 20050519 (Red Hat 4.0.0-8)

So, you can switch to number 3 to run Fedora Directory server and back to number 2 to run Openoffice.org. Slick.

Final Thoughts

I like the command line. I’m not a big GUI guy unless no alternative exists. So with the console in a state of disrepair, I tried searching FDS from the command line. Then I went to another GUI tool, LDAP Administrator from Softerra. I got a picture of Fedora Directory Server’s DIT. Oooooh is it ugly. I like to think I’m a pretty good LDAP admin, modeler, etc. but I don’t wanna work on FDS unless they get that console working.

And if they do: Well, get a KVM switch, hope that it’s good one and you can keep your keyboard and mouse running when you change to another console and back. Cause you’re going to need to run X and you might as well get used to those admin tools in System->Admin->?

I recently got a call from a friend in Amman, Jordan who said the very large web site he manages needs a Linux system administrator. But guess what, they can’t find a single Linux sysadmin in the entire country. With plenty of Linux desktop users around, I’m starting to wonder if anyone wants to learn how to use chkconfig.

The next problem that I’m starting to sense deals with a lack of standards. A client company in Dallas-Ft. Worth (my hometown) demonstrates a global condition. A Linux system administrator goes to work and builds a nice application for this media company. He sets up the system and then like many Linux sysadmins decides he can’t stand seeing Windows XP on the desktops and in film editing bays. So he leaves.

The server needs maintenance and someone to show the users how to back up the considerable data on the server. So, we get a system admin over to the client to help out. He finds a kluge of workarounds, experiments with software raids, deprecated libraries and so on. We can’t figure out how the previous Linux guy did what he did. Of course, our predecessor left no documentation.

You got it: We mounted the file system, compressed the data files and moved them to a Win32 server. Then we install a new Linux operating system, build the data management application again and transport the data back to the server. This time, the system has documentation and a record of what we did.

Plenty of companies experience similar outcomes. Linux sounds like a good idea (and it is) but Linux slobs leave their IT departments and a mess no one can figure out. It’s not giving us a good rap, friends.

We even found a system where the sysadmin rewrote the entire init system. Try coming along after that.

Hey, I love Linux and will always advocate its use. But, I hate slobs and the arrogance in the community.

Now, I expect lots of trash in the comment section from people in denial. All I can say is to stay in denial, remember to voice your opinion that Microsoft will fail and Linux already rules. How about this: “in your dreams”.

I’ve been administering production UNIX systems since 1998. I’ve been running Linux-and-only-Linux on my desktops and laptops for almost that same period of time. In 2003, after seeing an influx of Macs into our department, my boss asked if I’d keep a Mac around just so we could replicate any issues that were reported by our userbase to help them out. I found it perfectly usable, though at the time it was real work for me to get it to be usable for all of the things I’d need it to do at work, and I never was comfortable with it. Recently, however, I had the opportunity to get a new laptop through work, and I decided to give the new MacBook Pro a shot.

I really committed to making it my primary workhorse, shunning my nice dual-lcd-monitor linux box and a perfectly usable Gateway laptop (”the monster” - 17″ screen and a full numeric keypad!!). It has been about one month since the MacBook Pro showed up, and I haven’t had to use any other machine for about 2 weeks. It took a little bit of googling to find everything I needed, and there’s still one or two pieces of the pie missing, but overall, I think I can definitely call this machine “admin ready”.

So let’s have a look at what I’ve done to my MacBook to make it worthy, and maybe some friendly readers can help me fill in some as yet missing pieces!

Between November 2003 and May 2005, I was working on the rather mammoth task of evaluating my department’s environment, analyzing and auditing our infrastructure services, seeing if we could migrate from NIS to LDAP, and then evaluating LDAP server software, writing tools to perform the migration, writing tools to maintain data consistency between NIS and LDAP (for those things that couldn’t use LDAP yet), and writing tools to administer LDAP and integrate it into our environment.

Software evaluation at the time was fairly quick: I looked at eDirectory and SunOne, both of which (at the time) allowed you to store some obscene number of objects under a no-cost license. I chose OpenLDAP at this time because it was libre software with a pretty active support mailing list.

I spent the next year learning the intricacies of OpenLDAP: which back ends might benefit my deployment, how to configure that back end, why my distribution vendor chose not to use the recommended back end (forcing me to build from source), keeping up with the rather frequent upgrades (which the support list folks demand you do, lest you be heckled rather than supported), figuring out how Access Control Lists work, figuring out why some operations were so slow while others seemed blazing fast (this goes back to choice of back end and its configuration), and the list goes on. Over the course of a year, I probably upgraded OpenLDAP 3 times, upgraded the back end at least two times (one was the advent of Sleepycat BDB 4.0, if memory serves), and began to feel like there was no end to the tweaking. I feared I’d be pigeonholed into being solely an “LDAP Admnistrator” instead of a system administrator.

Then, on June 1, 2005, Fedora Directory Server (and Red Hat Directory Server) was released to the world. It wasn’t yet completely open source, but they announced that they were committed to open sourcing the bits that weren’t yet open in the coming months. I downloaded and installed the server, got on the support IRC channel, and imported all of our data in a couple of hours. By September, just three months later, it was in production, and I haven’t looked back.

Over the weekend, this news item caught my eye. Among other things, this proposed legislation would:

Require any manufacturer of “routing” and “addressing” hardware to offer upgrades or other “modifications” that are needed to support Internet wiretapping.

Its getting to that time of year when thousands of security professionals and hackers congregate in Las Vegas for the Blackhat and Defcon conferences. My company is generously sending me and a few colleagues to Sin City to attend :-)

The Blackhat line up is pretty strong, and this year features a lot of web application security talks (as summarised here by Jeremiah Grossman). You might spot me there on the speaking list for the Oedipus talk, but I’m not actually going to be speaking this year.

As for EuSecWest earlier this year, I am going to blog a bit of detail about the talks that I make it to - its probably fair to say that these are going to lean heavilly towards the web application security side. I am also going to be attending Defcon, and am looking forward to catching up with a lot of people who don’t really ever go to something quite as commercial as Blackhat.

So, if you’re in Vegas for Blackhat or Defcon, and interested in catching up, drop me an email. I will probably be at the Shadow Bar at Caesar’s on Wednesday night meeting other members of the webappsec mailing list, and otherwise will be around from the 1st to the 7th of August.

Every time a former Bell company has merged with yet another former Bell company in the last 10 years:

SBC merges with Ameritech
SBC merges with Pacific Bell
SBC merges with AT&T
(anyone seeing a pattern here?)
Bell Atlantic merges with GTE
Bell Atlantic merges with NYNEX, becomes Verizon

When each of these mergers happened, each company would trot out its press flacks, its CEOs and other assorted fish-faces, and they would all basically say the same thing:

Disk layout planning is still a good idea. The trick is making sure your applications respect your plans.

Would you buy a computer that you had no intention of using?

So, it is often reported that since the “bad guys” share technology information (such as exploits, bot access, malware, etc), the “good guys” should ramp up their sharing efforts as well. But companies’ unwillingness to share data that might, under the circumstances, be considered sensitive is legendary – and understandable.

Thus, while I was happy to see such projects as Splunk Base which lets users upload their logs that indicate problems (yes, security problems as well) and tag the logs with descriptive tags that enable other Base users to learn from their experience, described via tagged log samples. Just sharing logs is nowhere near as useful as sharing such experiences. Either way, this is a good initiative to watch.

Specifically, CNet says (http://news.com.com/Start-up+brings+glitch+wiki+to+IT+pros/2100-7346_3-6056530.html): “Instead, Splunk has designed its software and Splunk Base to allow system administrators to submit information themselves and then classify and search the collected information of their peers. ”

Well, it brings our the standard question: if you start a community for marketing reasons (this one clearly fits such definition), how do you make sure it actually takes off and starts a life as a real community of dedicated users (sometimes ramping up to “raving fans” :-)). I was reading this book by Guy Kawasaki (”Selling the Dream”) and it seems to have some answers… In any case, there is a difference between a real community and just a free platform for sharing which might develop into a community, might get monetized or just might tank. We will see what happens to this one.

Security remains an issue as well. Passwords are not too uncommon in Unix and Apache logs (if users mistype them for a username). Other things to watch for include allowed email addresses, IP addresses of critical servers, access control rule information, types of security software used and maybe a few dozen other possible thingies… An intelligent sanitization algorithm seems very important!

My experience with Honeynet Project data tells me that sanitization is not as easy as some think. So, given you have a serious issue – that you might or might not want others to know about, and that might or might not contain sensitive data, do you want to post that data to an open forum hoping that a) someone would help you and/or b) your experience will help someone else? Just post the comments here.

Another fun thing is the “added intelligence” factor. It has to be better (make it “much better”) than simply dumping the logs on the public HTML page and having good ole Google search them…

Admins spent hours looking at logs and sorting thru various log “esoterica.” I am sure many of my readers ever exclaimed ‘there gotta be a standard’…

Here is an interesting piece from Computerworld written by Oracle CSO Mary-Ann Davidson. She indicates that NIST is taking the charge in defining a common audit log standard. Can it actually happen? Maybe, if NIST can leverage US government’s purchasing power and demand support for such standard from all kinds of log-producing device and software vendors. I would not say that the chance is very high, but - unlike failed even standard projects like IDMEF and CIEL - this one seems to have the right players in place…

Just imagine the world where all the logs look the same :-)

So, a while ago I did this poll on network intrusion prevention mishaps. What are the results and what do they tell us?

* Block and NOT alert you on the threat at all (45%)
* Alert but NOT block the threat (29%)
* Block and NOT tell you what specifically was blocked (18%)
* What is an intrusion prevention system? (5%)

So, as it was pretty obvious that a majority of respondent NIPS users (45%) will be pretty upset about the silent blocking - the first case above. And, the hidden motivation for this poll was actually a story relayed to me by a friend who was recently involved in a “major” NIPS evaluation for a “leading” magazine. It turns out that one of the common NIPS devices is afflicted by that very thing - silently dropping certain suspicious packets without any record in the logs. Just think “broken application troubleshooting” and be terrified about wasted hours of system/network administrator time…

Recently, I picked some fun discussion on system log admissibility in court. Here is a summary of several posts by others and my comments as well.

Overall, its a fun but very confusing and esoteric subject…

In an unrelated post, Richard Bejtlich stated on his blog that “Tor servers will have to run inline filters to police this sort of activity.”

This issue troubled me for a while. Somebody smart :-) told me some time ago that Tor license and legal FAQ actually prohibits such monitoring and (?) filtering. Specifically, it says:

“Q: Should I snoop on the plaintext that exits through my Tor server?
A: No. You are technically capable of monitoring or logging plaintext that exits your node if you modify the Tor source code or install additional software to enable such snooping. However, Tor server operators in the U.S. can create legal and possibly even criminal liability for themselves under state or federal wiretap laws if they affirmatively monitor, log, or disclose Tor users’ communications, while non-U.S. operators may be subject to similar laws. Do not examine the contents of anyone’s communications without first talking to a lawyer.”

My response that was that the above goes against common sense, but I was told that law and common sense have nothing to do wich each other…

Ideas? Discussion?

Here is an interesting doc, pertaining to summarize current ISP operational security practices. It even has a neat section on logging practices:

“2.7. Logging Considerations

Although logging is part of all the previous sections, it is
important enough to be covered as a separate item. The main issues
revolve around what gets logged, how long are logs kept and what
mechanisms are used to secure the logged information while it is in
transit and while it is stored.”

The weird part is that the document advocates “exception logging”, rather than a full audit logging of network connections. Is that because those ISP usually have huge network pipes? Or is there some legal requirements to not have discoverable data on connectivity?

A very fun paper on bots (a type of malware, in case you did not know). Two quotes of note are below:

About the botnet owner: “The young hacker doesn’t have much sympathy for his victims. ‘All those people in my botnet, right, if I don’t use them, they’re just gonna eventually get caught up in someone else’s net, so it might as well be mine,’ 0×80 says. ‘I mean, most of these people I infect are so stupid they really ain’t got no business being on [the Internet] in the first place.’”

About one of the victims: “He eventually opted to buy a new PC rather than spend the time and money to repair the infected one. ‘It just made more sense for me to get a new $300 Dell that came with a free monitor that was better than the one I had,’ he says.”

Here is a fun poll that I want people to answer: Inline Network Intrusion Prevention poll: “What is the worst thing your inline Network Intrusion Prevention system can do? ”

Treat this as a puzzle right now, I will explain why I am asking it when I get a semplance of a representative set (maybe 50-100 votes).

Sorry for cross-blogging :-), but its a fun bit so I figured I’d post it here too (I want to test the new platform as well). In this blog post I discuss what was one of the important lessons of CME-24 (you do know what it is, don’t you?)