There’s a lot of good, interesting discussion happening on this blog as of late. The state of system administration tools, in general, is pretty poor. “Taking care of business” is still very dependent on the ingenuity and creativity of the individual administrator. Maybe being able to be creative is one of the things you like about administration, but wouldn’t it be nice to solve a problem more interesting than, say, installing emacs on 30 machines without working up a sweat?
And sure, more sysadmins could probably participate (or initiate) more projects, or provide more feedback. And more sysadmins could probably publish more of the bits of code they write than they do. And more of us could probably be more productive if we could find a way to interact with more admins, thereby realizing that we’re really *not* the only people who could ever make use of that 20-line piece of perl mastery. I agree on all of these points. However, I think much of this dances around a few large,
multi-colored elephants in the room. Flame if you must, but allow me to point them out for you….
Community
First, the state of the sysadmin ‘community’ is in a somewhat unstable state for the moment. There’s a lot of hubbub lately around the “LOPSA or SAGE” question. Which one do I join? What’s the difference? Where do I fit? These questions need to be answered satisfactorily by someone who doesn’t really
care which organization you join. I’ll do my best to be a neutral third party and ask others to comment below:
I’m a member of USENIX, SAGE, IEEE, a local LUG, and LOPSA. I’m a co-founder of LOPSA-NJ, which is a New Jersey chapter of LOPSA. We have meetings, speakers, discussions, and the like. The group took about 20 minutes to form. We started a mailing list, put word out to a few of the other LUGs in the area, spoke to everyone we knew, and the first meeting was a smashing success. However, there was discussion at the meeting about this LOPSA vs. SAGE bit, to which I replied that the LOPSA vs. SAGE nonsense should be left to be fought over by those who have some reason to care about it, or who have nothing better to do.
Systems administrators should be more interested in getting things done than about who wins some political battle. I belong to SAGE because I derive some benefit from that. I belong to LOPSA for the SOLE REASON that they reached out and said they’d support us if we started a local chapter to gather systems administrators and form a community. Having a local presence, to me, and to many others I’ve spoken to, is the only difference between LOPSA and SAGE that matters to the average administrator.
Being able to get together with other admins and talk not about photo management in ubuntu, but about monitoring a globally dispersed collection of servers, regardless of what OS they’re running, is valuable to me. I also think that the lack of standards and best practices is better addressed through efforts that start with the community than those that start with a committee.
Conferences
LISA is the only conference I’ve ever seen that attempts to be devoted to system administration. Sure, there are service-specific conferences like ApacheCon and security-specific events like SANS, and they’re great, but overall, LISA is the only one that is truly and unapologetically admin-centric, and attempts to cover a broad range of admin-specific topics (ie, not just apache, and not just security).
I think this is a sad state of affairs, really. I went to OSCON in ‘06 as well as LISA, and I think OSCON could do much to exploit the overlap between development and administration. Even though I do write a little code, I was thoroughly disappointed in OSCON because, even in places where the administration end of things was germane to the discussion at hand, it was completely ignored; avoided even.
What’s sad about that is that I was really pretty unimpressed with LISA, and would happily take my business elsewhere, if only there were a place to take it. I view LISA as proof that there’s a demand for admin-centric conference content. I view LISA as free market research for O’Reilly or some other willing organization who can take the ‘AdminCon’ ball and run with it.
By the way, I’m happy to attend/speak at/help with the planning of an admin’s conference, or the admin component of an existing conference. I’ve done conference planning before: I helped php|architect magazine organize their first php|cruise when I was still Editor in Chief of that publication. I was unable to attend, but it was a packed house, and subsequent conferences have also done well, which, of course, speaks somewhat to the usefulness of the first. In fact, I’m so interested in seeing this happen that I’ll *volunteer* my time to help with organizing, planning, coordinating, etc., for the first year that someone attempts to address this outside of LISA. Email me (jonesy - linuxlaboratory - org)
Competencies
System administration isn’t evolving? Are you for real?
Let’s face facts, system administration isn’t what it was 20 years ago. Whether you want to believe it or not, administration of very large sites has gotten easier, not harder. Just ask anyone who has been doing administration for 20 years. They do not long for those days on the by and large.
One thing that inevitably happens as things get easier is it lowers the barrier to entry to the profession. Sure, there’s a TON of technologies I need to know about in order to do my job, but overall, maintaining
infrastructure services is a far more streamlined operation than it was in the days before luxuries like NTP and DHCP. Those two technologies alone save us all many hours of grief every week. In a nutshell, it’s probably safe to say that it’s not as ludicrous these days for someone without an engineering background to think about becoming a systems administrator.
System administration IS evolving. There used to be *no* configuration management tools. It used to be *completely* on the admin, and every single solution was specific to the site where it was deployed. There used to be no such thing as kickstart and jumpstart and FAI, and… the list goes on.
There used to be virtually no non-commercial solutions for monitoring that were not admin and site specific. RRDTool, MRTG, Nagios, Zabbix, Ganglia, Cacti, these people, and many others I don’t have room to name, are doing good work.
Splunk is also trying very hard to catalyze change in the way we all think about the most mundane part of administration: log analysis. Well, log analysis isn’t mundane (at least not to me), but writing code to do it is.
Google has also made many tools available that are causing administrators to think twice about how they approach deploying services, and with things like Google Apps for your domain, maybe some are thinking twice about whether they maintain particular services in-house at all.
Red Hat has open sourced Fedora Directory Server. The OpenID project is gaining traction and changing how people think about digital identity. And how could any administrator who manages large numbers of machines live without ClusterSSH? And what would beowulf administrators do without ROCKS or OSCAR?
I guess my point is made without going further (though that wouldn’t be hard to do).
The hard part about developing new tools specifically for administrators is that administrators are, many times, not programmers. I would classify myself more of a scripter than a programmer. I can whip out some quick perl or shell to do what I need. If something needs fixing (in whatever language I’m likely to run across on a day-to-day basis) I’m probably competent enough to figure out what’s going on, but that doesn’t make me a programmer. I’m not well versed in how to organize a large complex project, how to implement a proper MVC design from scratch, how to design complex and efficient algorithms to handle complex tasks. I *know* for a *fact* I’m not good at this because I once tried to start a project myself that eventually went by the wayside as another project came into existence which addressed most of the issues I was addressing, though in a much different fashion. The key there was that my project was being coded by an admin (me), and the other was being coded by a coder with direction of an admin.
As for me…
I give away all the code and information I can on my website. If you do, too, I invite you to link us to your stuff below in the comments. I’m also happy to post other information there that I didn’t write. If you have any, send it along (jonesy - linuxlaboratory - org).
I’ve contributed code to a few open source projects, and have reported bugs to even more. I try my best to pay attention to the questions I’m asked not only so I can answer the person asking, but so I can publish the questions and answers later for others to benefit from. Some of those answers actually
wound up in Linux Server Hacks, Volume Two which I was lucky enough to be invited to co-author.
My philosophy is that, if I know something, and someone else asks, I spill the beans. I tell them absolutely everything I possibly can that might be useful. My hope is that this will make the administration world a better place. Maybe something I gripe about will inspire a much more talented code slinger than myself to take action. Maybe I’ll be enlightened to another way of thinking by a more experienced administrator, and I’ll then be able to pass that information on to others. Maybe another administrator will follow my lead and from that day forward share everything he or she knows whenever possible.
In the end, I try to take the action that will make administration on the whole a better profession, and I like to believe that taking that attitude will make a difference.
My job has changed drastically in 10 years. The professions that have changed the most radically are computer related. Think back to the machines you ran 10 years ago.. Boot from floppy! Linux was brand new, NIS was state of the art, Samba was barely a glimmer of a working product, network and event management was a joke.
Now, you can install a new machine or lab of machines or a whole department of machines with System Installer Suite in like 7 minutes flat, enterprise event and network managers with really *good* features like event deduplication, correlation, root cause analysis, SLAs, and service monitors are common place.. Heck, you can actually monitor all your services and know what's going on without too much work. You can graph response times without having to dump it to a file and run gnuplot on it. You can actually predict when the capacity of a server is going to run out by consulting your graphs. You can actually track how much time you spend on things through a ticket system and easily share tickets and workflows with others. (Remember GNAT? eeek!). Do some of the old tools like lsof and strace still exist? Yup, but so does the screwdriver, the electric drill, the level, and the tape measure. Many, many other tools have died a righteous and deserved death of obsolescence or fallen into obsolescence. Remember glimpse? Remember when majordomo was the only game for mail lists? When you had to use adb to debug a kernel? When you had to analyze network protocol dumps by hand? remote support using vnc or rdp? forget it.. PXE boot of intel machines with virtual KVM or console redirect? no way. There was this thing called PC weasel that sort of worked..
The grass isn't always greener out there. Other industries have just as many if not more problems than we have. Look at engineering and the plethora of FPGA, ASIC hardware, materials analysis and other tools that they have that don't tie together well, if at all. Look at all the diagnostic tools that large scale industrial production has that don't tie together. Heck, look at what the business world uses. We know.. We support them! How many business people do you know that still insist on keeping what amounts to a database up to date by passing excel spreadsheets back and forth?
Sysadmin is at the leading edge of these things. We're not in the dark ages, we're leading the way in improving enterprise efficiency. We tell the business community how to do it better and faster without passing spreadsheets back and forth. We improve security. We improve the bottom line through improving efficiency. We are enablers! And, we're starting to gain respect..
10 years ago, how many of your family members knew what a sysadmin was? They didn't even know the term.. "I take care of computers" was about as close as you could get. Today you can even tell other people you socially meet "I'm a System Administrator", and most of the time they know what that means. Either they know somebody else who does it, or they work at a company that has one or more on staff. There are hundreds of thousands of people like us in the Unites States alone. Things have come a long way for this profession and they are only getting better. Be active, support a community, join a user's group. The best way to make things better is to get together with others that share a common experience and make things happen. Carpe diem.
disclosure: I'm a founding member of LOPSA(.org) and we are trying very hard to grow the community and the profession. But we can't do it just by founding an organization. It very much depends on individuals who are dedicated to making a difference. It's entirely volunteer based up to and including the board members.
The next LOPSA event is going to be in Philadelphia this Spring - I believe planning is just underway, it sounds like the Admin focused event you are looking for.
Hmm. 10 years ago NIS was just as hated then as it is now, I was investigating replacing NIS+ with LDAP, I had completely automated system installs using Jumpstart, we had Remedy installed for trouble ticketing, I was managing FibreChannel SANs using Veritas software, I was doing performance monitoring (using rstat, which fortunately has been replaced by tools like Ganglia and Cricket), I was debugging network problems with snoop, and process problems with lsof and truss.
If anything, I'd say that maybe the Linux world has finally caught up with the world of the commercial Unix, so I suppose it looks really different if you've been a Linux admin for the last decade, but it looks pretty darn similar to me.