This fun article covers some of the mistakes that often occur when organizations try to use encryption to protect data at rest and data in transit and thus improve their security posture.

LinuxPlanet published an interview with Ibrahim Haddad this past summer on a topic near and dear to my heart.

Developers have done a great job in the last couple of decades of evolving their practice, so that development looks very different now than it did a couple of years ago. There’s a lot of competition for tools and methodologies, lots of publishing on these differences, and plenty of opportunity for new ideas to gain mindshare. This competition is really important to evolution: Unless there is opportunity and reward for better ideas and products, these better ideas don’t develop very quickly.

Sysadmins, on the other hand, have very little competition, in either ideas or products, and in the few areas where there is competition there generally isn’t much to differentiate the competitors.

Ibrahim’s interview mostly focuses on process, which is, I think, where a lot of the evolution has happened in development recently. The wizz-bang Agile stuff is all about changes in process, and has almost no impact on tools, other than having a preference for tools that enable the process. It doesn’t hurt that there are lots of consultancies set up specifically to sell Agile development methodologies.

I’d love to see the same kind of process competition in the sysadmin world, and I wish I knew how to kick-start it, but my only real area of expertise is tools, so I’m kind of forced to stick to trying to that.

I will say that I’m very happy that Puppet has modern competition in BCFG2, because it forces us to keep making our products better. I know that reporting in Puppet needs to improve, because BCFG2 has set the standard there, and I like to think that there are parts of Puppet that Narayan sees and knows he has to match in order to compete effectively.

Kathy Sierra has another great article up, wondering whether better tools make us dumber (e.g., does the use of calculators mean we don’t need to learn math). The article brought to mind a lot of the fear I see among sysadmins when I discuss tools like Puppet. Sysadmins aren’t so much concerned that better tools will make them dumber but rather that better tools will allow dumber people to replace them.

Kathy’s conclusion is that you still have to understand what you’re doing, but I think she missed the opportunity to talk about why we use better tools.

I’m lucky to be in a position where I am not forced to specialize on a single technology. I have always made a habit of keeping up with the job market, and it seems the trend is that the bigger the company you wind up at, the more likely you are to be staring at the same thing day in and day out.

There are people even less specialized than me - my buddy is a one-man IT shop. He’s in an environment that could probably use just about one more IT guy. He’s probably wishing for the luxury of slightly more specialization that might be afforded to him if there was someone else around to help out.

There are people more specialized than me, too. A family member of mine is just about the hardest of hard core WebSphere admins. Oh, he’s a UNIX admin, but he got a job at $HUGECORP and I get the impression that he hasn’t seen so much as a zone file ever since. It’s this level of specialization I’m fighting.

Last week, I spent the week doing database development (culminating in a bit of SQL goodness you can read about on my blog) This week, I’m replacing an aging print (CUPS/Samba) server with a virtual machine. After that, I’m afraid I have a little PHP code to write for Moodle, and then I’m back into the back end on a data warehousing project. Before February started, I was upgrading (read: PXE kickstarting) a 30-node teaching lab, helping debug some weird issues on a beowulf cluster, and retiring the first beowulf cluster I ever deployed.

Over this coming summer, I hope to migrate more services from our old NIS service to our new LDAP service, which I deployed using Fedora Directory Server about two months after it was released to the world. I’ve blogged here that I was happy about that. I still am :-)

…And I’m still a little bitter that I rarely get to touch our networking gear.

Oh yeah, one other way I’ve been able to fight off specialization is by consulting. I have clients that need things set up that I have no use for or have outgrown in my day-job environment. This experience has even provided a couple of useful experiences that I brought back with me to my day job, and certainly keeps me grounded in the realities of how other organizations go about making technology decisions, and how those decisions affect the business side of things.

What about you? Are you able to fight specialization in your work? Have you suffered or been saved by specializing/not specializing? Have you turned down bigger paychecks to avoid specialization? Are you sorry you did/didn’t take a highly specialized job? I’m interested in your views on this! Share! :-)

There has been quite a bit of fall-out over my post on the lack of evolution in system administration. Most of it’s about what I expected — “right on!”, I should have been more rigourous, there was too much hyperbole, I’m wrong, I’m insane — but it was still interesting watching it all happen. I’ll go into a bit more detail below the fold.

Following the now old :-) “tradition” of posting a security tip of the appropriate time interval (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #8: What Just Changed?

Let’s close our eyes for a second and dive deep into the bizarre and menacing world of a Windows event log. As I mentioned before, massive Windows server log collection got a jump start in recent years due to wide availability of agentless Windows log collection tools, such as Project LASSO. (yes, many people think that agents suck event when they are useful - weird, isn’t it?)

Windows event logs, the “Big Three” of System, Security and Application as well as other logs, share a lot of contradicting properties: way too much detail in some areas and missing critical info in others, consistent and thoughtful design here and sheer stupidity there, nice structured data sometimes and confusing mumbo-jumbo in other cases. And the universe of the event log is never static, the whole thing flows and morphs with each Windows release and at time with each update. New event IDs are being created, changed and loaded with new roles and new info.

In this tip, we will look at some fun Windows log entries and explain their meaning for your organization as well as cover what you should do if you encounter them. Given that the realm of Windows event log is so huge, we will start from looking at events that indicate changes of different kinds, mostly configuration and user account. So, what just changed?

I. “Computer Account Deleted” or “User Account Deleted”: obviously, service or user account was deleted. Who did it? When? Why? Answer all the questions above and then you can go back to sleep - or to your incident response plan :-)

II. “Computer Account Created” or “User Account Created”: same thing - depending upon when? why? who? this event means nothing or something pretty ugly.

III. “Computer Account Changed” or “User Account Changed“: similarly, changes to accounts are reflected in the events containing this text. Account changes do include privilege level changes that are often of particular interest.

At this stage, it might be appropriate to ask: why aren’t we going by Windows event ID to identify the above events of interest, but instead choose to use the above text blurbs? Well, up to Vista, Windows event IDs often aren’t :-) Meaning that they don’t identify the event sufficiently. Sometimes, they are overloaded and the same ID applies to very different things. Sometimes, the opposite happens - same event, different IDs (e.g. a lot of login/logout stuff)

IV. “Policy Change”: might mean almost anything on a Windows system. Thus, we can’t really tell you much; you need to read the event to see what actually changed (if anything!)

V. “The system time was changed” might not matter that much, but if you are looking to use your logs as forensic evidence (i.e. use them in court) you might want to track all the time changes since they will affect the log timestamps on the server where time changed.

VI. “The following schema object was modified” oooh, don’t you love Active Directory! This indicates that some of the AD objects changed - fortunately, the object name will be in the same event.

Enough for today! Windows logging makes most everyone’s head hurt (unless you are Eric or Randy, I guess :-))

So, to conclude, make sure that you collect Windows event logs and analyze them on an ongoing basis, preferably using your log management system.

As I mentioned before, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

I originally posted this on my blog, but thought the broader sysadmin community could benefit from the discussion:

Somebody on a mailing list asked a question about shell quoting. The quoting issue would not have been so difficult had it not been for the fact that it was a command that was to be run on the other end of an SSH connection, which adds a level of difficulty in interpreting what’s going on. Here’s the command he was trying to run:

ssh -t hostname ’sudo sh -c “echo \’test\’ ; echo \’test\’” ‘

(of course, nobody wants to run echo as root, it’s just an example representative of the problem at hand)

My understanding is that the person running this wants to see output from this command that looks like this:

‘test’
‘test’

Here’s how I believe the command would need to look to get that output:

ssh -t hostname “sudo sh -c \”echo \’test\’; echo \’test\’\”"

The key is starting with double quotes instead of single quotes, because single quotes fed to a bash shell is a toggle switch that turns shell expansion on and off. Until about 5 minutes ago, I had always had the rule “Don’t put single quotes inside single quotes” in my brain as a rule of thumb to avoid confusion. After doing some checking (in the bash man page), turns out it’s a bona fide RULE in bash. From the man page:

A single quote may not occur between single quotes, even when preceded by a backslash.

Whaddya know? I must’ve picked that up along the way and forgot about it.

If you think about the single quote as a toggle between ‘expansion on’ and ‘expansion off’, and NOT as a literal single quote, and then parse that first command again, you’ll see that this is probably not going to work.

I only found out about this because someone noted that the patent is being challenged, but it looks like OpsWare is even more egregiously closed than I thought. Here’s the summary of the patent:

The patent — number 7,124,289 — covers the cornerstone of Opsware’s technology, the ability to automate the management of servers and network devices across a data center through model-based control. This innovation allows models of granular configuration information about servers or networks to be stored as records in a database and, as changes are made to any aspect of a systems configuration model, to push those changes to any relevant servers or network devices. This technique supports automation for even extremely large and global datacenters, and it provides a fast, efficient and cost-saving approach to ensuring consistency and compliance.

As described, this patent covers pretty much all of the configuration management tools out there. Just another reason to give no respect to OpsWare, I guess.

An OpsWare employee often tells me I should give them more credit when I complain about how bad the commercial companies are. I’m not fond of the products from either OpsWare of BladeLogic, although I haven’t used either of them extensively and haven’t looked at them in a while, but last I heard OpsWare considers their user guide to be proprietary and confidential. This clearly indicates a closed, fearful mindset, and I refuse to give respect to a company with that attitude, no matter the product set. It’d be easier to give these companies credit if they had anything other than whitepaper puff pieces on their sites.

There’s a lot of good, interesting discussion happening on this blog as of late. The state of system administration tools, in general, is pretty poor. “Taking care of business” is still very dependent on the ingenuity and creativity of the individual administrator. Maybe being able to be creative is one of the things you like about administration, but wouldn’t it be nice to solve a problem more interesting than, say, installing emacs on 30 machines without working up a sweat?

And sure, more sysadmins could probably participate (or initiate) more projects, or provide more feedback. And more sysadmins could probably publish more of the bits of code they write than they do. And more of us could probably be more productive if we could find a way to interact with more admins, thereby realizing that we’re really *not* the only people who could ever make use of that 20-line piece of perl mastery. I agree on all of these points. However, I think much of this dances around a few large,
multi-colored elephants in the room. Flame if you must, but allow me to point them out for you….

Since fellow O’Reilly blogger Luke Kanies mentioned me in his excellent piece Why Isn’t System Administration Evolving? I thought I should clear up some misstatements.

My post on the fact that I don’t think system administration is evolving seems to have ticked some people off on the lopsa-discuss list.

Sure, I was a bit flippant in my post, but I think many of the critics are falling into the common trap of thinking that tools vary in functionality but not in fundamentals. Paul Graham does a great job discussing this in his article about beating the averages; it is about languages but it’s just as valid here. Just like languages, tool power varies considerably.

I don’t just mean tool functionality, that is, the specific features of the tool; I mean the overall power. I think of this variance in power as deriving from difference in models: Tools with low-level models are not very powerful.

For instance, take one of the big differences between cfengine and Puppet; cfengine focuses mostly on managing textfiles, while Puppet manages semantically more powerful constructs like users, services, and packages. Thus, Puppet has higher-level models than cfengine, and if you accept my premise that model power derives from how high-level it is, that generally makes Puppet a more powerful tool than cfengine.

Note that that doesn’t make it more functional — it could be less stable, there could be lots of interesting things it can’t do, whatever.

This is one of the things I think are really lacking in the sysadmin world: High-level models. Do you think in terms of /etc/passwd, or users? If you think about /etc/passwd, then LDAP is fundamentally unrelated, but if you think in terms of users, then they’re functionally equivalent.

It’s not just about how powerful the modeling of the tool is, though, it’s about the power of the models it creates.

Take dependencies and relationships between resources. Are there any sysadmin tools that effectively model these relationships? Of those tools, do any of them make the information exportable to other tools, so you’re encouraged to spend a lot of time developing these comprehensive models? No, not really. Sure, some of the monitoring apps allow you specify basic dependency information, but they’re extremely limited, and you can never export them so they’re useful elsewhere.

What really sets Puppet apart from Cfengine isn’t that it models higher-level resources instead of files, it’s that it allows you to model the relationships between those resources. Puppet’s lowest layer is responsible for the resource modeling (e.g., what it means to be a User on Solaris vs. OS X), but its language handles the high-level relationships between those resources, including how they’re grouped (e.g., they’re all part of the class of resources that makes apache work) and how they relate (e.g., apache should restart if its configuration file changes, and changes to the file should happen before the service is checked).

This is why better tools are so important: The models our tools use set an upper limit of the size of the problems we can comfortably work with. Unless we develop a whole slew of new tools with more powerful models, we’ll be forever stuck thinking about bits on disk.

That means it’s not just about configuration management, it’s about everything on your network. If my configuration management system can’t model the network, then I can’t set up IP addresses or firewall ports; if it can’t model the backup system, then I can’t provide guarantees of data availability or perform automated restores; if it can’t model change, then I have to do all migrations manually.

These models are critical, and they’re all in our heads instead of in our tools. This fact is a severe limitation on our field’s ability to move forward.

Now that I’ve gotten the introduction out of the way, I can get to the meat of why I’m blogging here in the first place. This is a long one, but I think you’ll find it’s worth it.

I’m not writing Puppet because I think I’m right or whatever; I’m writing it out of desperation, because no one else is even trying. Not only are people not trying to make better tools than we have available today, they’re not even using the crappy ones we do have available, which is just sad. Imagine if the computing world had just refused to write any code until C (or better yet, Ruby) showed up; where would we be now?

I’m less interested in why sysadmins don’t use the existing tools, though, and more interested in why they don’t publish their own. Most of the rest of the technical world seems to have figured out how to solve their problems using code and then how to turn that code into a self-sustaining project, either open-source or commercial. Sure, this stuff was complicated fifteen years ago, but it’s pretty straightforward now; and yet, instant messaging tools have larger development communities than sysadmin tools, and the majority of sysadmins spend their days toiling with a bunch of little one-off scripts that no one else will ever see or use and that the next sysadmin will gratefully /dev/null as soon as possible.

I’ve heard all of the standard excuses — we don’t have enough time, we can’t risk it, we spend all day doing computers and don’t want to do it at night, my company won’t let me, etc. Every software project that has ever evolved out of an internal project has exactly these same excuses, and yet they have somehow succeeded. Why have so few sysadmin tools evolved this way? Why are sysadmins so willing to believe their excuses?

Hullo. This is my first post on the O’Reilly sysadmin blog, and I’m about four months late. I told Mike Loukides I’d start blogging in November, and I’m just now getting around to it. It’s true that I run my own blog, but that’s mostly about the development of Puppet, and here I’ll be focusing more on the state of system administration and how we can maybe do better.

Incidentally, I’m the author of Puppet, a server automation framework. There are a few presentations online about Puppet, but the short summary is that I spent a few years developing and consulting, both on cfengine and on some of my own tools, and finally decided I could not take it any more, and figured I needed to either quit the field or create a tool I really wanted to use.

So, two years ago, Puppet was born, and that’s what I’ve been doing ever since. Puppet is an ambitious project for me, because I’m really hoping to do more than create a decent tool, I want to redefine acceptable best practice in system administration. Cfengine is currently the “state of the art”, but it’s barely changed in 10 years and it’s only got one developer who isn’t even working on it full time. I want to do more than create a tool that’s better than cfengine, I want to prove that sysadmins can use open communities to build great open source tools, and I want to use our community to build a whole new generation of tools, not just one monolithic “does everything” tool.

If you want even more gory detail, you can read the interview just published by Computerworld Australia.

In the meantime, I’ll do my best to write often. I don’t really know what I’ll be blogging about here; it’ll definitely be part rant on the state of system administration, but I’d also like to help keep people abreast of what I learn and do (for the record, I’m now a full-time developer working on Puppet, and I haven’t been an operational sysadmin (thankfully) in a few years), and I will post on the events I go to and my experience trying to evangelize a better way to do system administration.