I love free software - free as in freedom. I’ll take a free beer too. I didn’t know about those things back in 1998 when I finally got Slackware to boot on my DEC Prioris and suddenly had a fast version of , ah, UNIX. Naw, this was free .

I immediately attempted to work that server into a project in an all Microsoft shop at Cap Gemini US. Somewhere along the line one of the tech support guys grabbed it and locked it in a closet where others attempted to scavenger it. I got it back and took it home where it eventually became my workstation.

One of the cool things about Linux? Remote administration. I could log into the server from afar and do stuff. I didn’t have to go down to the data center, use a KVM switch and look at a NT GUI.

I remember remote administration, industrial strength servers running on Intel processors that scaled and lots of developers working on the core system. Those were the days when geeks were men!

Well, I just got another dose of Microsoft in the form of Fedora Linux. I started installing their free directory server and what a mess. Actually, it’s a kluge. It’s broke like Red Hat 5.0 was broke. But now, you have to use a KVM switch and look at a desktop even after you figure out how to make it work.

By the way, kluge has another spelling and here’s the definition:

(I found a definition of kluge on Wikipedia that I like better than any I have seen before)

A kludge (or kluge) is a clumsy or inelegant solution to a problem. In engineering, a kludge is a workaround using unrelated parts cobbled together. People demonstrating the force of the term often say that it takes a skilled craftsman intimate with the task, the material at hand, and the operating environment to construct a workaround clunky enough to be called a kludge.

That’s the definition of Fedora Directory Server. It reminds me of the broken NT servers on which I got certified - that’s in passing tests not crazy. Broke and in need of workarounds.

Last week, we attempted to get the console to work but to no avail. What was wrong? Here’s what the documentation said and what we did to work around the problem:

Try to start up the admin console (this is a graphic user interface, so you’ll need to be running X):

# cd /opt/fedora-ds
# ./startconsole -u admin -a http://server1.centralsoft.org:12853/

If you get this sky-is-falling output:
GC Warning: Out of Memory!  Returning NIL!
GC Warning: Out of Memory!  Returning NIL!
GC Warning: Out of Memory!  Returning NIL!
Exception in thread "main" java.lang.OutOfMemoryError
   --No stacktrace available--

then you have a Java package problem.

As this was written, the Java packaged with Fedora (gcj) did not quite work with Directory Server, even though it’s version 1.4.2. Check http://directory.fedora.redhat.com/wiki/Install_Guide to find any updates or fixes. If the problem has not yet been resolved, change one line in the start-console script. Remove the -ms8m and -mx64m options (bold in this example to help you find them) from /opt/fedora-ds/startconsole.

Of course, that didn’t work.

This week we get a new message from the folks at Fedora:

Java runtime. The JRE is required in order to use the Console. Either the Sun or the IBM JRE version 1.4.2 or later is required. Unfortunately, the console does not (yet) build and run with the open source GNU gcj/Classpath java implementation, but we are working on it. We thought that gcc/gcj 4.1 included with Fedora Core 5 would work, but it still has many problems, so your best bet is to use Sun or IBM JRE.

So, I built Java the hard way. I wound up at another URL and it said:

Fedora Core 4 users are advised not to use the Java RPM provided by Sun. It contains Provides that conflict with names used in packages provided as part of Fedora Core 4. Because of this, Sun Java might disappear from an installed system during package upgrade operations. Fedora Core 4 users should use either the RPM from jpackage.org or manually install the Sun Java tarball into /opt. Sun Java 1.5+ is recommended for stability purposes.

I won’t put you through the pain, you can go to http://fedoranews.org/mediawiki/index.php/JPackage_Java_for_FC4 and see it for yourself. I built it even though the contributor wrote the instructions for FC4. I found broken links. So, I went looking for the packages and found them on other sites.It all worked.

But here’s a rub. Openoffice.org doesn’t work. But a workaround exists:

The JPackage java RPMs support switching between java implementations using the "alternatives" system.

[localhost ~]$ sudo /usr/sbin/alternatives --config java

There are 3 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
   1           /usr/share/java/libgcj-java-placeholder.sh
   2           /usr/lib/jvm/jre-1.4.2-gcj/bin/java
*+ 3           /usr/lib/jvm/jre-1.5.0-sun/bin/java

Enter to keep the current selection[+], or type selection number: 2
[localhost ~]$ java -version
java version "1.4.2"
gij (GNU libgcj) version 4.0.0 20050519 (Red Hat 4.0.0-8)

So, you can switch to number 3 to run Fedora Directory server and back to number 2 to run Openoffice.org. Slick.

Final Thoughts

I like the command line. I’m not a big GUI guy unless no alternative exists. So with the console in a state of disrepair, I tried searching FDS from the command line. Then I went to another GUI tool, LDAP Administrator from Softerra. I got a picture of Fedora Directory Server’s DIT. Oooooh is it ugly. I like to think I’m a pretty good LDAP admin, modeler, etc. but I don’t wanna work on FDS unless they get that console working.

And if they do: Well, get a KVM switch, hope that it’s good one and you can keep your keyboard and mouse running when you change to another console and back. Cause you’re going to need to run X and you might as well get used to those admin tools in System->Admin->?

Admittedly, this is again a repost from my other blogs (see here and here), but I am sure it would be useful to my O’Reilly readers.

Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #2: Watch those Pesky Log Rotation Routines

What is the biggest threat to your Linux logs, which reside happily at their home in /var/log? Is it crackers? Blackhats? Evil BOFH sysadmins? Dumb users? Malware? No, its the log rotation routine!

For example, many Linux distros such as RedHat, Fedora, etc use logrotate tool to get rid of “unwanted” logs. Other distros use other log file rotation tools that accomplish the same. However, many ship with default settings that are not optimized for log analysis and long term retention.

Upon deploying a Linux box one needs to look at /etc/logrotate.com file as well as /etc/logrotate.d/ directory to make sure that you are happy with retention settings (and you shouldn’t be!)

One can safely increase the retention of most log files from a default of about 4 weeks (it differs by file type and distro) to at least 12 weeks (about 90 days, a common practice) or even more. While at it, one should also enable old log file compression since it is off by default (for whatever weird reason…) to save some disk space.

Note that if you are using a central log management system, this might not apply to you. But then again, in this case you are farther ahead of most system owners on the road to operational excellence.

Also, here is a link to my previous tip of the appropriate-time-interval (#1) :-)

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Writing perl scripts that are better suited for scheduled tasks.

Admittedly, this is a repost from my other blogs (see here and here), but I am sure it would be useful to my O’Reilly readers.

Upon seeing folks giving security tips of the day on their blogs (like here, here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #1: Crank it Up!

Crank what up? Logging, of course! You’d be happy you did, I promise. The gist of this tip is that if you increase the level of logging on your system (even without looking at the resulting logs periodically…), you’d fare much better in case of an incident. And, by incident here I don’t mean that those evil hackers will 0wn you with a custom-coded 0day, but something much more common: a computer crash, spyware problem or system malfunction.

On Linux/Unix it will likely involve editing the /etc/syslog.conf and on Windows it will involve changing the Audit Policy.

Of course, you’d be commiting a mistake, if you only turn it on. But that’s a separate story.

I recently got a call from a friend in Amman, Jordan who said the very large web site he manages needs a Linux system administrator. But guess what, they can’t find a single Linux sysadmin in the entire country. With plenty of Linux desktop users around, I’m starting to wonder if anyone wants to learn how to use chkconfig.

The next problem that I’m starting to sense deals with a lack of standards. A client company in Dallas-Ft. Worth (my hometown) demonstrates a global condition. A Linux system administrator goes to work and builds a nice application for this media company. He sets up the system and then like many Linux sysadmins decides he can’t stand seeing Windows XP on the desktops and in film editing bays. So he leaves.

The server needs maintenance and someone to show the users how to back up the considerable data on the server. So, we get a system admin over to the client to help out. He finds a kluge of workarounds, experiments with software raids, deprecated libraries and so on. We can’t figure out how the previous Linux guy did what he did. Of course, our predecessor left no documentation.

You got it: We mounted the file system, compressed the data files and moved them to a Win32 server. Then we install a new Linux operating system, build the data management application again and transport the data back to the server. This time, the system has documentation and a record of what we did.

Plenty of companies experience similar outcomes. Linux sounds like a good idea (and it is) but Linux slobs leave their IT departments and a mess no one can figure out. It’s not giving us a good rap, friends.

We even found a system where the sysadmin rewrote the entire init system. Try coming along after that.

Hey, I love Linux and will always advocate its use. But, I hate slobs and the arrogance in the community.

Now, I expect lots of trash in the comment section from people in denial. All I can say is to stay in denial, remember to voice your opinion that Microsoft will fail and Linux already rules. How about this: “in your dreams”.

Ok. Here’s the deal. As the term “Web 2.0” makes its way into more and more glossy ads printed prominently in the largely irrelevant technical drivel outlets (er, magazines), more PHBs are going to want it. In many cases, this is probably a justified desire, given all that Web 2.0 encompasses. Some of those who have a clearer view of the technological landscape have commented recently that operations could well be *the* differentiating factor, the “value add”, that determines success or failure in Web-2.0-land.

I’ve seen very little, even from the usually outspoken developer community, to dispute these comments. Of course, I’m an operations person, and I think that most of the comments are spot on. They are also cautionary in their tone, which is also spot on, because operations is sort of a forgotten landscape in the technological ecosystem. It’s worthy of concern.

The only questions that seem to be going unasked, then, are “What do the
operations folks need, and what plans exist to meet these needs?”

Well?

Commentary has gotten us to the point of consensus. It’s time to get moving on
a plan of attack now. I haven’t a clue exactly where to start. For my part,
I’ve tried to clean up a lot of my code, and post any snippets that can be of
use to others on my website - http://www.linuxlaboratory.org. Unfortunately,
what’s there is hardly even a drop in the bucket.

(NOTE: if you have code you want to put there for operations folks, drop me a line)

In addition to working in operations, I’m also someone who loves discovering
and exploring new technologies (and old ones, too), and I love to write and
help others understand how things work, the context that different
technologies operate in, the patterns that exist in different technologies and
operating paradigms, and the like.

This, I’m told, is something that a vast majority of people working in
technology despise. Nobody likes to document things. Maybe part of it has to
do with r0ml’s ‘bozo’ theory: documenting things means somebody will read it,
and perhaps you’ll be found to be a bozo for one reason or another. Maybe part
of it is sheer laziness: nobody’s found a way to automate documentation for a
service deployment that I know of. Whatever reasons exist for operations
people tending to stay away from vi for anything other than config files or
code, I’d like to help fill the gap in whatever ways I can.

So here’s how this works:

I’ll ask you, the operations people at large, some questions. You will either
leave replies as comments to this post, or you can email them to
operations2.0@gmail.com

I’ll monitor the responses and email, and I’ll use any and all resources at my
disposal to come up with some directions that I and others can take to address
the needs of the operations community. If it means development work, we can
bring them into the conversation. If it means more documentation is necessary,
I’ll try to do my part there, and share the information I get with other
authors and experts whose documentation I can help develop if need be. My
hunch is that there’s plenty of things needed by operations people that a
whole team of smart people who have access to resources themselves haven’t
even thought of. That’s what makes this whole experiment interesting.

So, without further nonsense:

What is it that you need? Do you need tools? What kind of
tools? What tasks need automating for which there is no good tool? Why do you
think the current tools suck? What exactly sucks about them?

What things lack the requisite integration? What services are lacking in
features? What features do you want to see? How would these features help you
in your work?

What services don’t even exist yet in a form that you can use? Why can’t you
use what’s available now? Why does it suck?

What are you currently writing code to do yourself? What platforms are you
working with? What is the “holy grail” for your environment? Why aren’t you
there yet?

What things have you left behind because they’ve failed to scale or perform?
What have you moved to? What decisions have you made that you’ve been forced
to revisit in light of new requirements? What’s the driving factor of the new
requirements? Is there a theme or patterns in the requirements you’re seeing
now as opposed to two or three years ago?

What tools have you been happy with? How can they be even better?

Basically, in a nutshell, “What’s causing you pain, why is it painful, how can
it be alleviated, and why do you think the right solution doesn’t exist
already?”

Be as specific as you can in your replies. Name names. Bring on the links. If
you think that application x is only useful to you if it can integrate with
service y, and it doesn’t, then say so, and also say anything else you can
think of that would make either app x or service y more useful.

If you work with software vendor foo and you need tool bar, but you can only
get it if you sign your life away, then SAY SO! Chances are, some of us who
work with nothing but open source solutions aren’t even aware that these
commercial proprietary tools exist, and if we were aware, we’d say “hey,
that’s cool, I need a free tool that does that”. If anyone has proven that
they have the power to make it so, it’s the open source community.

I look forward to your responses.

I wanted to put a note here for myself, and for others who are looking for a way to get information about a particular executable running in a zone on their machine. While it is (to the best of my knowledge) not possible to do this from within the local zone itself, you can run dtrace from the global zone and specify the zone name and executable by using a logical AND in the predicate, like this:

dtrace -n ’syscall:::entry /zonename == “webserver” && execname == “httpd”/{ printf(”%S”, curpsinfo->pr_psargs); trace(pid) }’

This is a command I used to get some quick information about all of the httpd processes running on a web server that exists as a zone on a solaris 10 machine. Here’s some output:

0 6485 write:entry /var/local/httpd/bin/httpd -DSSL\0 12248
0 6779 llseek:entry /var/local/httpd/bin/httpd -DSSL\0 12248
0 6489 close:entry /var/local/httpd/bin/httpd -DSSL\0 12248
0 6789 pollsys:entry /var/local/httpd/bin/httpd -DSSL\0 12248

The command is run from the global zone. This isn’t exactly what I wanted — I wanted all of the arguments passed to all of the system calls, but I got the arguments passed to the httpd process itself at start time instead. I’m still chugging through the documentation, so if I find more cool stuff, I’ll put it here. Meanwhile, if you know how to do what I’m trying to do, or how to do some other cool stuff with DTrace, please share!!

Here are a few links about DTrace that I’m using to get more information. It’s plenty to get you started, but not a lot of info on how to do anything remotely advanced. Even coverage of ‘advanced’ features is pretty dumbed down compared to the crazy stuff I’m seeing in some of the example scripts. Nonetheless, this stuff *is* useful:

http://www.snpnet.com/sun_DTrace/dtrace.html
http://users.tpg.com.au/adsln4yb/dtrace.html
http://www.sun.com/bigadmin/content/dtrace/
http://blogs.sun.com/bmc