Two days ago, I received an inquiry from a Fortune 500 company looking for a contractor to perform system administration work. I read the email and sat befuddled trying to figure out what this employer wanted an applicant to do. This solicitation read “LINUX Admin Wanted”. Perhaps you can figure it out. Here’s the job requirements.

Looking for UNIX candidates only. Skills: Sun and/or HP expierence; Solaris 10 experience; Clustering experience, Veritas or Sun Cluster; Domain experience, preferably on a larger sun system; Back up experience - HP/Solaris; EMC a plus; LVM Tools.

Here’s the job description:

Responsible for installing, configuring, and maintaining operating system workstations and servers, including web servers, in support of business processing requirements. Performs software installations and upgrades to operating systems and layered software packages. Schedules installations and upgrades and maintains them in accordance with established IT policies and procedures. Monitors and tunes the system to achieve optimum performance levels. Ensures workstation/server data integrity by evaluating, implementing, and managing appropriate software and hardware solutions. Ensures data/media recoverability by implementing a schedule of system backups and database archive operations. Supports media management through internal methods and procedures or through offsite storage and retrieval services. Develops and promotes standard operating procedures. Conducts routine hardware and software audits of servers to ensure compliance with established standards, policies, and configurations as defined by Customer.

Now, I must lack something because I couldn’t do this job. In fact, I cannot even guess how many people the employer would need to do this job. My guess: many people.

Aside from the the items above, the letter also had this information request:

Please complete the details below to enable us to submit your resume:

1 Primary Contact Number:

2 Secondary Contact Number:

3 Citizenship (If non-US, please indicate work status):

4 Please confirm if 12 months contract (extendable) duration is acceptable:

5 Please confirm if 44.1/hr (W2 hourly and all-inclusive rate) pay rate is acceptable:

6 Availability to interview (pls. indicate 2 days with 2 time slots each day):

7 Please confirm if you are willing to work at the following address: 2501 S. State Street, Lewisville, TX 75056 OR 100 Witmer Rd., Horsham, PA 19044-2319

Click here to get directions to the above location. This position does NOT offer TELECOMMUTING.

8 Earliest possible start date:

Anyone who wants this job can have it. In fact, if you want it, you can fill out another lengthy database form that will take you somewhere around two hours. I won’t fill out those ridiculous forms where recruiters accumulate data and spit out forms to send potential human resource customers.

Here’s some advice from someone who lives and breathes this business. Figure out in a sentence what you want a candidate to do and ask that. Stop sending out form letters and make a call. You stand a better chance of finding the person you want.

Also, stop putting in the key word “Linux” when the only thing you have listed says Linux experience a plus when you really want a Microsoft MCSE. When I get letters from recruiters like that, I also see some phrase like “if you know someone with these credentials, please refer them to us.”

I wouldn’t do that and I don’t know anyone else that would. The top IT people in the world - and they’re not in India - have suffered over the past four years from the incompetence of employers in the IT field. Don’t expect those people to re-enter indentured servitude again.

Recently, I had the pleasure of watching an article of mine show up on Slashdot, Digg.com, Newsforge and a myriad of other web sites at the same time. None of my colleagues had seen so much traffic on a Linux site before. Aside from the several million hits on our server, we had a quarter of a million unique visitors concentrated in a five-hour period.

When you see that kind of traffic, you don’t want the server to go down or you’ll miss new readers. In our situation, a reboot allowed the system to return to service for a few minutes, but then it locked up again. On a normal day, we used less than ten percent of our system resources, so we thought we had prepared for the hottest day of the year. Little did we know we would experience rolling blackouts.

I attempted to explain to our self-proclaimed system administrator that we needed a way to renice unnecessary processes automatically, restart those that failed and keep the server going at all costs. He just had no enterprise experience and insisted he knew the best way to manage the system. He used a RRD tool with a front-end. But when it came to handling processes, he insisted he would login with SSH and handled everything remotely.

This provides an example of hardheadedness and an inability to adapt and improvise. I know you do not know any one else like this, but I do. In fact, the majority of noise makers in the community refuse to use tools that execute in a web services environment.

You can characterize the modern business climate by friction, uncertainty, disorder, and rapid change. Each situation is a unique combination of shifting factors that cannot be controlled with precision or certainty. System administrators need to learn to adapt. Adaptability means shortening the time it takes to adjust to each new situation.

Two basic ways to adapt exist. First, if we have enough awareness to understand a situation in advance we can take preparatory action. Call this anticipation.

At other times we have to adapt to the situation on the spur of the moment without time for preparation. This is improvisation. To be fully adaptable, we must be able to do both.

We co-hosted our server at an ISP 250 miles from our base of operations. When the server went down, we had to call the ISP and have one of his service personnel run down to the server rack and power it back to a working situation.

From that point, our brilliant, know-it-all sys admin had to login into sshd. By the time the server powered back and started all the processes including sshd, the server locked up again. So what was next?

Insisting on using the command line, the sys admin had to walk a support technician through a series of steps to renice and shutdown PIDs without allowing the system to connect to the Internet.

Once he got the system dedicated to one purpose, he reconnected it to the Internet and though slow, the OS and web server stayed up.

Next, our sys admin had to stop several of the Content Management System processes to keep the server up and allow visitors to view a single article. Eventually, the server responded in a reasonable manner. But, consider all the trouble.

Sacreligion

I prefer using the command line myself. But some situations warrant something in addition. For example, when you manage dozens or even hundreds of servers, you need to poll and drill down into each server remotely and efficently.

One of the tools I like allows me to do the drilling. The developers call it Monit. Here’s the description from the web site:

Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You can use monit to monitor files, directories and devices for changes, such as timestamp changes, checksum changes or size changes. You can also monitor remote hosts; monit can ping a remote host and can check TCP/IP port connections and server protocols. Monit is controlled via an easy to use control file based on a free-format, token-oriented syntax. Monit logs to syslog or to its own log file and notifies you about error conditions and recovery status via customizable alert.

Monit allows you to view a server’s status through a web browser among other features. You can find the project at http://www.tildeslash.com/monit/. Additionally, Falko Timme has written a howto for installing and setting up Monit at http://howtoforge.com/server_monitoring_monit_munin.

Command line? Essential for any system administrator to know. But, consider adding adaptability to your skill set. It will pay off.

Related link: http://www.courtesan.com/sudo/sudo.html

A long time ago, I worked in an environment where everybody used sudo. Administrators, programmers, it didn’t matter. There was no password prompting, and any command was available. The only advantage this setup offered was that it logged every command sent to it. Of course, the logging was worthless if someone types in sudo bash, which was a common occurence.

I think sudo is a great tool, but there needs to be some thought behind its implementation. Here’s a couple of suggestions I’ve learned from using sudo in the past.

Sudos

Define who needs to use sudo

It’s my feeling that regular systems administrators should not have to use sudo. If they’re an administrator, and they are authorized to use the host in question, they should be trusted with the root password, a One Time Password phrase, or whatever authentication mechanism is used to grant root access.

Sudo’s best feature is the ability to grant restricted root access. It’s better suited for system operators that have clearly defined tasks, such as mail administration, or printer administration.

Create a restricted command path

Define what commands are going to be used with sudo, and who is going to use them. Put those commands in a restricted directory. If you have administrative roles that don’t overlap, such as tape administrators or dns administrators, create seperate directories, or specify each command individually in the sudoers file. Make sure sudo users cannot write new files to these directories.

Create hardened scripts

It’s very difficult to make a shell script 100% secure, especially if the script requires command line arguments. A script like the following can do some very bad things if it’s involked improperly.

$ cat remove_home_dir
#!/bin/sh

rm -rf /home/$1

exit 0

Make sure the script is tested against tainted command line arguments, no arguments whatsoever, and tempfile race conditions. If necessary, create a C or Perl wrapper around the script, and use the wrapper to parse the arguments.

Prevent overuse of sudo with OTP

If you only give your users a limited number of One Time Passwords, you can gauge how often they’re using sudo, and you might reinforce better behavior knowing they have a limited amount of opportunities to use certain commands.

Sudon’ts

Don’t assume sudo prevents mistakes

A friend of mine told me about an intern who ran the following…

$ sudo "mv /etc/passwd /etc/passwd.tmp"

Now, if this person ran this command as root, he could have easily corrected his mistake before it became a problem. But because sudo dropped root privlidges when it exited, he totally hosed the server. This goes back to restricting what commands should be used with sudo.

Don’t trust the environment or the user

I’m not suggesting deliberate misuse, but it’s very easy to create the impression that sudo can prevent the user from doing any harm. That won’t be too effective if it turns out sudo is blindly taking the user’s PATH variable which may inadvertently reference bad applications.

The env_reset option in the sudoers file strips many environmental variables from being passed to the sudo environment. It’s a good step to making sure that your command scripts are more secure.

Don’t overlook shell escapes

Commands like more, or vi provide a handy shell escape sequence that lets you run commands within the program. You can see where I’m leading with this, right? Sudo provides a mechanism to restrict EXEC calls, but you should check to see if it is supported in your OS.

Don’t overlook privilege escalation

Some operating systems have a privlege model providing an additional layer of security. Intead of creating security around certain commands, the OS provides prilidges for certain administrative roles, such as user X can administer the print queue, or user Y can reboot the host.

If your OS vendor gives you additional means of granting access with less effort than configuring sudo for the task, consider using it.

Su-nevers

Never forget the first line of defense

You can harden your root environment or your /etc/sudoers file all you want. But it won’t be very effective if your operator is accessing a server over the telnet protocol through an unsecured wireless Internet cafe on a laptop (that clearly identifies his employer), which he leaves unattended when he goes off to the bathroom.

Take the steps to make sure your host and user security is in place, then worry about your sudo configuration.

Sudoku

Come on. You know I had to do it.

Daily Sudoku

Related link: http://oedipus.rubyforge.org

The Oedipus Web Application Scanner project (that I am involved in some of the development) has just released it’s first beta release - version 1.8.1. Oedipus is a penetration testing focused tool, designed for penetration testers, and for technical security or web development folks to test their applications for web application security issues. It deviates from many of the commercial tools in that:

• Oedipus does not claim to be a one stop testing tool that will find every type of hole in your applications. It is, however, pretty good at finding the low hanging fruit so you can spend your time finding the really nasty problems manually
• Oedipus has some exploitation functionality built in, especially for SQL injection at this point, for generating working exploits for web application vulnerabilities. After all, the best way to show the business impact of an issue is to show it is exploitable
• It’s free, open source, and pretty easy to extend through the use of it’s plugin architecture

From the blurb - "Oedipus is an open source web application security analysis and testing suite written in Ruby by Pentration Testers for Penetration Testers. It is capable of parsing different types of log files off-line and identifying security vulnerabilities. Using the analyzed information, Oedipus can dynamically test web sites for application and web server vulnerabilities"

So, it is often reported that since the “bad guys” share technology information (such as exploits, bot access, malware, etc), the “good guys” should ramp up their sharing efforts as well. But companies’ unwillingness to share data that might, under the circumstances, be considered sensitive is legendary – and understandable.

Thus, while I was happy to see such projects as Splunk Base which lets users upload their logs that indicate problems (yes, security problems as well) and tag the logs with descriptive tags that enable other Base users to learn from their experience, described via tagged log samples. Just sharing logs is nowhere near as useful as sharing such experiences. Either way, this is a good initiative to watch.

Specifically, CNet says (http://news.com.com/Start-up+brings+glitch+wiki+to+IT+pros/2100-7346_3-6056530.html): “Instead, Splunk has designed its software and Splunk Base to allow system administrators to submit information themselves and then classify and search the collected information of their peers. ”

Well, it brings our the standard question: if you start a community for marketing reasons (this one clearly fits such definition), how do you make sure it actually takes off and starts a life as a real community of dedicated users (sometimes ramping up to “raving fans” :-)). I was reading this book by Guy Kawasaki (”Selling the Dream”) and it seems to have some answers… In any case, there is a difference between a real community and just a free platform for sharing which might develop into a community, might get monetized or just might tank. We will see what happens to this one.

Security remains an issue as well. Passwords are not too uncommon in Unix and Apache logs (if users mistype them for a username). Other things to watch for include allowed email addresses, IP addresses of critical servers, access control rule information, types of security software used and maybe a few dozen other possible thingies… An intelligent sanitization algorithm seems very important!

My experience with Honeynet Project data tells me that sanitization is not as easy as some think. So, given you have a serious issue – that you might or might not want others to know about, and that might or might not contain sensitive data, do you want to post that data to an open forum hoping that a) someone would help you and/or b) your experience will help someone else? Just post the comments here.

Another fun thing is the “added intelligence” factor. It has to be better (make it “much better”) than simply dumping the logs on the public HTML page and having good ole Google search them…