As promised, here is another “Top 11 Reasons” which is about log analysis. Don’t just read your logs (definitely don’t just collect them); analyze them. Why? Here are the reasons:

1. Seen an obscure log message lately? Me too - in fact, everybody have. How do you know what it means (and logs usually do mean something) without analysis? At the very least, you might need to bring additional context to know what some logs mean (example: IP address -> hostname -> server owner)
2. Logs often measure in gigabytes and soon will in terabytes; log volume grows all the time - it definitely passed the  limit of what a human can read a long time ago, it then made simple filtering ‘what logs to read’ impossible as well: automated log analysis is the only choice.
3. Do you peruse your logs in real time? This is simply absurd! However, automated real-time analysis is entirely possible (and some logs do crave for your attention ASAP - e.g. major system failures, confirmed intrusions, etc)
4. Can you read multiple logs at the same time? Yes, kind of, if you print them out on multiple pages to correlate (yes, I’ve seen this done :-)). Is this efficient? God, no! Correlation across logs of different types is one of the most useful approaches to log analysis.
5. A lot of insight hides in “sparse” logs, logs where a single record barely matters, but a large aggregate does (e.g. from one “connection allowed” firewall log to a scan pattern). Thus, the only way to extract that insight from a pool of data is through  algorithms that “condense” that collection of logs into usable knowledge (some say, visualization is the way to go)
6. Ever did a manual log baselining? This is where you read the logs for a while and learn which ones are normal for your environment. Wonna do it again? Thought so :-)  Log baseline learning is a useful and simple log analysis technique, but humans can only do it for so much before burning out.
7. OK, let’s pick the important logs to review. Which ones are those? The right answer is “we don’t know, until we see them.” Thus, to even figure out which logs to read, you need automated analysis.
8. Log analysis for compliance? Why, yes! Compliance is NOT only about log storage (e.g. see PCI DSS). How to highlight compliance-relevant messages? How to see which messages will lead to a violation? How do you satisfy those “daily log review” requirements (again, see PCI DSS)? Through automated analysis, of course!
9. Logs  allow you to profile your users, your data and your resources/assets. Really? Yes, really: such profiling can then tell you if those users behave in an unusual manner (in fact, the oldest log analysis systems worked like that). Such techniques may help reach the holy grail of log analysis: have the system automatically tell you what matters for you!
10. Ever tried to hire a log analysis expert? Those are few and far between. What if your junior analysts can suddenly analyze logs just as well? One log analysis system creator told me that his log data mining system enabled exactly that. Thus, saving a lot of money to his organization.
11. Finally, can you predict future with your logs? I hope so! Research on predictive analytics is ongoing, but you can only do it with automated analysis tools, not with just your head alone (no matter how big :-)) …

 Past top 11 reasons:

• Top 11 Reasons to Collect and Preserve Computer Logs
• Top 11 Reasons to Look at Your Logs 
• Top 11 Reasons to Secure and Protect Your Logs

Introduction

If things go as hoped documenting how a Fortune 1000 size enterprise implemented an Open Source strategy will provide guidance to others. The term, Open Source, started as a way to approve licenses for software whose code became published and available for others to modify for their needs. Almost tens years after the Open Source Initiative, we have discovered that the concept has far surpassed its original meaning and intent. Today, people realize that Open Source has eclipsed the vision of the OSI founders. The methodologies behind Open Source give organizations the power to reduce development time and produce a superior product. You can make comparisons to other paradigm shifts through history and the Open Source movement will come out favorably.

What topics can you expect?

1. Dissecting the methodology used to produce most Open Source software
2. The existing culture in traditional development shops
3. Finding champions to further acceptance
4. Education - getting through the alligator pit
5. Setting up an internal OSS development environment and selecting a pilot project
6. Disclosing the bundle if OSS tools and technologies
7. Finding resources to administer the development environment
8. Introducing existing OSS for immediate use: Firefox, Openoffice.org, GIMP and so forth
9. Delivering the pilot
10. Writing the findings report.

Here we go. You’ll look at drafts and ultimately a finished book. You can also help by working on our wiki.

I’m looking forward to this project.

I’ve been wanting to create those for a loooooong time and finally - here they are (you can guess I’ve been on a long flight :-)). Some are admittedly tongue-in-cheek, but useful nonetheless. So, enjoy Anton’s “Top 11 Reasons to Collect and Preserve Computer Logs”, presented in no particular order:

1. Before anything else, do you deal with credit cards? Patient info? Are you a government org under FISMA? A financial org? You have to keep’em - stop reading further.
2. What if there is a law or a regulation that requires you to retain logs - and you don’t know about it yet? Does the world “compliance” ring a bell?
3. An auditor comes and asks for logs. Do you want to respond “Eh, what do you mean?”?
4. A system starts crashing and keeps doing so. Where is the answer? Oops, it was in the logs - you just didn’t retain them …
5. Somebody posts a piece of your future quarterly report online. Did John Smith did it? How? If not him, who did? Let’s see who touched this document, got logs?
6. A malware is rampant on your network. Where it came from? Who spreads it? Just check the logs - but only if you have them saved.
7. Your boss comes and says ‘I emailed you this and you ignored it!!’ - ‘No, you didn’t!!!’ Who is right? Only email logs can tell!
8. Network is slow; somebody is hogging the bandwidth. Let’s catch the bastard! Is your firewall logging? Keep the info at least until you can investigate.
9. Somebody added a table to your database. Maybe he did something else too - no change control forms were filed. Got database log management? How else would you know?
10. Disk space is cheap; tape is cheaper still. Save a log! Got SAN or NAS? Save a few of them!
11. If you plan to throw away a log record, think - are you 100% sure you won’t need it, ever? Exactly! :-) Keep it.

Have more? Feel free to suggest your own reasons below!

Coming soon: “Top 11 Reasons to Look at Your Logs”

This fun article covers some of the mistakes that often occur when organizations try to use encryption to protect data at rest and data in transit and thus improve their security posture.

LinuxPlanet published an interview with Ibrahim Haddad this past summer on a topic near and dear to my heart.

Developers have done a great job in the last couple of decades of evolving their practice, so that development looks very different now than it did a couple of years ago. There’s a lot of competition for tools and methodologies, lots of publishing on these differences, and plenty of opportunity for new ideas to gain mindshare. This competition is really important to evolution: Unless there is opportunity and reward for better ideas and products, these better ideas don’t develop very quickly.

Sysadmins, on the other hand, have very little competition, in either ideas or products, and in the few areas where there is competition there generally isn’t much to differentiate the competitors.

Ibrahim’s interview mostly focuses on process, which is, I think, where a lot of the evolution has happened in development recently. The wizz-bang Agile stuff is all about changes in process, and has almost no impact on tools, other than having a preference for tools that enable the process. It doesn’t hurt that there are lots of consultancies set up specifically to sell Agile development methodologies.

I’d love to see the same kind of process competition in the sysadmin world, and I wish I knew how to kick-start it, but my only real area of expertise is tools, so I’m kind of forced to stick to trying to that.

I will say that I’m very happy that Puppet has modern competition in BCFG2, because it forces us to keep making our products better. I know that reporting in Puppet needs to improve, because BCFG2 has set the standard there, and I like to think that there are parts of Puppet that Narayan sees and knows he has to match in order to compete effectively.

Kathy Sierra has another great article up, wondering whether better tools make us dumber (e.g., does the use of calculators mean we don’t need to learn math). The article brought to mind a lot of the fear I see among sysadmins when I discuss tools like Puppet. Sysadmins aren’t so much concerned that better tools will make them dumber but rather that better tools will allow dumber people to replace them.

Kathy’s conclusion is that you still have to understand what you’re doing, but I think she missed the opportunity to talk about why we use better tools.

Would it surprise you to discover that Linux administrative and support employees have created barriers to entry for others with similar talents? What if I told you that a difficult job climate has emerged because of your Linux buddies? Would you believe it?

Get a grip because that’s something with which you may have to deal if you attempt to change jobs or enter the market. Recruiters tell me that “Linux guys” take job offers from predominantly Microsoft shops, go through training and within an average of three months leave their employers hanging. That means having Linux credentials could work against you. Technologists with Linux on their resumes might find something akin to age or gender discrimination when they start applying for work.

That’s a shame, because plenty of excellent technologists suffer from the antics of a few. I personally don’t care what platform on which someone employs me to work. I’ll optimize Active Directory and safeguard an Exchange Server when asked. If that’s what the company uses, then that’s what I’ll give them.

I also know plenty of people with the same feelings. In fact, one of the more ardent Linux trained system administrators I know works in an all Microsoft shop. He also gets a fair share of assignments doing technical reviews for Linux books in spite of his day job.

I also know the leader of one of the larger Linux User Groups in the US who spends his days working in a Microsoft shop. He doesn’t go off in a rage if someone won’t listen to one of his proposed solutions using Linux instead of Microsoft. I like him because he’s a diplomat and doesn’t endanger others.

I know you don’t want to hear this but the warnings from publications like “Information Week” about this issue fall closer to home than you may want to admit. Aside from creating barriers to entry for job candidates, IT managers have started shunning Linux and Open Source solutions because of the people issues. So, Linux and open source adoption may suffer as well.

While the mention of Linux is starting to make the hair rise on the necks of many recruiters and IT managers, Open Solaris fanatics have similar problems. In its attempt to retake their market from Red Hat, Sun has spawned a new breed of crazy people too. Hiring managers in turn have started losing their objectivity toward Sun. I’m starting to see migrations from Sun to Microsoft I never thought possible.

What’s the Problem?

The lack of a pragmatic approach to the job market and clear thinking lies at the core of this issue. Windows fanatics appear to get away with their pranks because Microsoft dominates the market. That makes them invisible to managers already enamored with Microsoft’s usability. The same managers will turn a blind eye to the technical advantages of Linux, for example, because of an unwitting perception of Linux fanatics who color the water for everyone.

Define a Pragmatic Approach

In ordinary use, pragmatism refers to behavior, which sets aside an ideal to achieve some specific current or urgent need. If you ever studied Maslow’s scale of needs, then you understand the approach. Advocating free software, while an admirable endeavor, doesn’t usually pay the bills, feed the kids or get you to work in a new automobile. In the common sense survival hierarchy, fanatic ideals fall way short of the basics required of people meeting their primary responsibilities.

Throughout history, vocal minorities have brought about societal change. But what about the rest of the people? While the masses may benefit from the sacrifices of a few brave men and/or women in the long run, those same masses are often quite busily engaged in seeing that their children have something to eat and a roof other their heads.

Fanatics in the IT Industry

For some reason, it doesn’t seem logical to me that an operating system should generate religious fervor. History shows us otherwise. It shows us that people using different computer systems for some reason engage in irrational behavior toward each other as if a holy war actually existed.

Regardless of the history it still makes no sense to me. When someone I’m paid to support needs me to fix their email account what difference does it make if they use Outlook or Evolution? In business process engineering terms, where’s the value add from a preference for Linux, Mac OS X, Solaris or Windows? The client wants his or her email to work; it’s a closed issue.

Fanatics are characterized by excessive enthusiasm for and intense devotion to a cause or idea and are often motivated irrationally. And while quite eloquent at times, fanatics in the IT industry are unnecessary when you think about the other issues on this planet such as starvation, HIV, war mongering, etc. at this time.

In other words, while you may have the means to wreak havoc on the commercial market, your reforms may not benefit mankind as a whole. In fact, the masses probably could care less about your arguments and positions about free versus proprietary software regardless of which side of the fence you’re on.

Potential for a Failed Experiment

While I do not personally believe Linux will go away, I said the same thing about Netware once. When IBM controlled 90% of the PC hardware market, no one believed they would ever lose that business completely. But they have. If Linux fanatics continue to muddy the waters for recruiters and hiring managers, the grand experiment could fail like others before it.

The Linux project demonstrated to the world that a global collaboration of computer engineers and technologists could create a highly functional operating system for free. The concept of collaboration and community has since spread into other industries and has given a lift to countries otherwise bereft of computer resources.

So regardless of the success or failure of the Linux operating system, we have gained from its rich heritage. That in itself could wind up as the legacy of Linux. Who really knows? Are Local Area networks the historical legacy of Netware or a computer in every home the legacy of the IBM PC Company? The possibility exists.

Time for Ordinary Pragmatism in the Job Market

I began this little essay asking a question. Would it surprise you to discover that Linux administrative and support employees had started creating barriers to entry for others with similar talents? It surprises me because I had not realized the degree to which Linux fanatics had infected the market for jobs. But, they have infected the market and you should feel concern if you chose a Linux career path.

As for me, I consider myself a technologist. I’m dazzled by Linux. Does that mean I’ll demand that people only use Linux and shun other operating systems or development environments? No. The only people I’m shunning these days simply don’t know how to keep their mouths shut and their obsessive opinions to themselves.

As a Linux advocate or should we say a bigot, I recently correlated enough of a survey to recognize that the Linux community has not arrived fully. Of course, if you want to discuss the issue of arriving, you also have to define the destination. I have a reference for meeting the goal and that came from Linus Torvald’s speech in March 1999 in San Jose when he uttered the words “world domination”.

Then, I also have my own itinerary. At one time, I felt angry toward and betrayed by Microsoft. So, during the Justice Department’s anti-trust suit, I had hope that the Redmond gang would meet with a break up. Someone dashed my hopes after George Bush’s assumption of office in 2001. That’s a little off topic.

I created an itinerary where Microsoft fell under the weight of Linux and free software. I envisioned the major vendors working together to hurt the giant of Redmond. And why not, I reasoned that free software mostly dumped on the users of the world by the National Science Foundation would see massive adoption.

But, let’s forget my itinerary and Linus’ world domination statement and look into the world of enterprises. Hey, guess what, Linux and free software haven’t made it. Not much demand for Linux people exists. Speaking with a bevy of recruiting firms convinced me. Here’s a few bullet points:

• Rarely do recruiters see job orders for Linux administrators. Those that do see a few live in the top tier.
• Demand for Linux admins seems concentrated in the ISP/Hosting market where Linux has about a 60% of the business.
• Advertisements for Linux skills on job boards and classified listings tend to say “Linux experience a plus”.

You didn’t read anything so far saying that Linux is losing, failing and not gaining any ground, etc. Linux has captured some technology markets. Unfortunately for the rah-rah crowd, Linux has not made much progress at capturing desktop market share. Where Linux has an advantage has not produced technology to benefit the desktop. In other words, the advances in Linux benefit servers and embedded devices and not the desktop.

I once proposed that someone create a DVD player with a daughter board. I thought that one could create an embedded device to accompany a DVD Rom using Cyberlink’s products to allow Linux desktops to play DVDs and other proprietary formats. I even thought about a PCI card that had the Cyberlink products on it. That’s not a stretch.

I also suggested that one of the major Linux vendors start a distribution and license Cyberlink’s audio and video products. That distributor could sell a stand alone version of the software as an add-on in different formats. People could buy that components alone. That would put Linux on even footing with other desktop operating systems.

So far, we have no takers.

I have concluded that Linux cannot expect to remain 100% free software and gain market share on the desktop. People will want features not available as free. That begins and ends with the current rave in music and video. We can give that fight up to the money people.

Last night, my DVD player failed. I own a RCA box that attaches to my Hitachi Television. In the middle of the story, the DVD player just started making ugly pictures and skipped tracks. Having no other choice, I booted up the computer my wife uses at home with the same OS she uses at her job. I finished watching the DVD on an Intel computer with an Apple Studio Display running XP.

I would have liked to have used my Linux desktop. But alas, I live in the US and we don’t get to use the Win32 codecs or the DeCSS DVD-decryption tools. We’re video and audio challenged.

I wonder. If Linux could perform the functions available on the Windows desktop, then would we start seeing jobs for Linux skilled technicians? I think so. Instead of just desk side support for Windows, companies would need to support people for Linux too. That would cause some growth.

In the mean time, I have given up my disdain for Microsoft. That’s right, I don’t feel strongly about them one way or the other. They’re just there and I will deal with them when I do.

As far as Linux and the demand for Linux system administrators, the US job market sees it in the server area but not on the desktop. At least, not yet.

I could consider a variety of titles for this short article. For example, we could call it “Stupid, the Internet isn’t the sum of all information technology”; or “Have you really stopped thinking about your Local Area Network?”; or “Have you ever heard of mainframes?”. I expect you get the gist of the topic.

My latest adventure with local networks came when I had to do a refresher on setting up and configuring OpenLDAP. I hadn’t done one since the publication of “LDAP System Administration” in 2003. Thank goodness for Gerald Cater. He did a splendid job of reminding me about installing and configuring directories.

Right now, I have the task of setting up an OpenLDAP directory. Since I have used Debian as the underlying OS, everything I need comes right off the default /etc/apt/sources.list. I don’t have to compile everything from source and I have the Debian applications database to help me maintain my “directory” of installed packages. I’ll leave off the discussion of the pros, cons and other ways of keeping track of what one has installed on his or her server.

I began working with LDAP directories around 1998 when the FAA contacted me about setting up several DEC’s Alta Vista directories servers around the US/Globe. The problem at the time dealt with distributing directory trees. We didn’t have LDAPv3 or the corresponding RFCs. We encountered some serious challenges at the time trying to bind different directory objects to one another on different servers.

Reacquainting myself with basic LDAP brought back a flood of memories. I even correlated relearning LDAP with such things as attempts to regain proficiency in some foreign languages I studied over the years. Truth be known, much of what we gather in the course of our work lives in our short-term memory. We study, get it down and move to the next subject. Once I installed and configured LDAP, I went on to something else. It reminds me of being in Cancun one time and while trying to remember the word for butter at breakfast, I told the waiter to bring me a butterfly.

Which brings me back to the central subject of this article. System administrators have to babysit servers and users. While we might wind up babysitting servers on the Internet, we can’t leave the many areas of our skill set in short-term memory. We should all remember that the bulk of our tasks usually involve users. We wouldn’t practice this profession without the internal clients.

In researching job ads for Linux administrators I noticed that NFS, LDAP, SAMBA, application and database servers, with an emphasis on automation and monitoring dominated the qualifications for many administrators with DNS, LDAP, FTP, SMTP, Postfix/Sendmail, etc. accompanying the requirements. While other requirements existed in those ads, most of them had to do with Soft skills like functioning as a liaison with other internal support groups such as Change Control, Application Development, Engineering, Databases Administrators, Web Services, Storage, Intel (Intelligence), Operations and Command Centers.

So, consider this a reminder. The Internet fascinates many of us. But, if we want to make a living in this business, we should remind ourselves of the cold hard reality of network troubleshooting, escalated service desk support, technical support, on-call consulting advice for the hardware and operating system environments, etc.

Here is a fun piece that I wrote recently, based on some stuff I read in the security media. I was planning to publish it elsehwere, but this place is as good as others :-)

Will security ever get done?
by Anton Chuvakin

Here is a fun thing to think about: with security, will we ever really be “done”? Before you, my esteemed security colleague, emphatically scream “NO!” let us consider this – admittedly philosophical – problem in-depth. There is also a related question that we will try to answer en-route to the above more general pursuit: will security become so boring that only boring people will do it (something akin to physical security guards)? In addition, we will touch another “messy question,” the one of “security consolidation”, that generated some attention lately (see, for example, this, this or this where some pundits and pundit wonnabes spout about it…)

Before we hear some pro and con arguments, let’s stop and think for a second: what security are we talking about? Network security? Software security? IT security? Data security? Or, information security in general? I would prefer to have this question answered for the broader information security realm.

So, why some folks think that “security problem” will be solved in the future?

• OS and application vendors will improve the security of their wares and gear so that security problems will not gather as much attention as now
• Network infrastructure vendors will “embed” security in their offerings and thus address a wide range of current “top shelf” security problems, such as worms, overall reducing the importance of security
• Similarly, large security companies will combine all sorts of defenses into largely automated “security bundles” and will “protect everybody” with them
• As new technologies develop, people will learn from the mistakes that plague us now and will start doing things right from scratch (e.g. IPv6 vs IPv4 situation)
• In particular, new software projects will “build security in” and thus will not provide such a huge attack surface as do the current “crapware” products
• IT users, both home and the enterprise kind, will be finally educated and thus will avoid the most costly security mistakes, such as running untrusted code (OK, this one is just a tad too naïve to be mentioned here (, if not for the sake of completeness)

Did I miss any? Feel free to comment or email me and I will update the list.

Why others violently disagree?

• New technologies that use the Internet and whatever other future networks will come out, some say at an increasing pace, and thus result in a dramatic increase in a number of “things to steal, break and abuse”
• Overall increased connectivity will also enable new attacks and open new exposures, thus needed novel creative solutions
• In general, new threats will always be there because there is no shortage of people who are both smart, creative and evil
• Increased reliance on IT systems will strengthen the resolve of cyber-criminals and all sorts of other bad guys to “go cyber” instead of committing “normal” crimes (“…since that is where the money is”)
• New uses of old technologies – networked fridge anyone? – will also open holes and exposure in the areas where none mattered before (SCADA security is one fine example)
• Economics always favors fast product delivery and thus lowers the quality of released current and future software; even though it might be devoid of obvious and easily found flaws, it will still be exploitable
• Increased regulatory pressure will sometime create the need for either new uses of security technologies or even motivate people to create entirely new security technologies (scalable log retention for compliance comes to mind)

Did I miss any here? Feel free to comment or email me and I will update this list as well.

In addition, some folks aggressively attack the pro arguments instead of coming with their own cons. Specifically they claim:

• OS and other infrastructure vendors will always lag behind, since, by the very nature of being large established companies, they cannot respond to the “fast lane” rate of threat change
• IT users will not learn and in fact will become worse, since the overall population is getting dumber (note that I am not sure I agree with this one…)
• Software developers will also not learn from the mistakes and, in fact, will repeat them, since economics seems to favor bad software quality

Let’s step back and try to come up with – no, not the compromise, that’d be silly :-) – the conclusion. Here is what I think the answer is.

Certainly, there will be consolidation in the security market and defenses will get embedded in both operating systems and network gear, eliminating some of the standalone network and system defense solutions. It is also likely that some types of bugs will be eliminated, if not by the good will of developers, but by the changes in the commonly used programming languages.

But, on the other hand, the explosive combination of the march of ever-more-critical new connectivity technologies with the presence of dedicated evildoers will, in my opinion, guarantee that information security will remain relevant, vital and fun for years to come! Security technology innovation will not dry out any time soon

Dr Anton Chuvakin, GCIA, GCIH, GCFA is a recognized security expert and book author. A frequent conference speaker, he also participates in various security industry initiatives and standard organizations. He is an author of a book “Security Warrior” and a contributor to “Know Your Enemy II”, “Information Security Management Handbook” and the upcoming “Hacker’s Challenge 3″. He also published numerous papers on a broad range of security subjects. In his spare time he maintains his security portal http://www.info-secure.org and two blogs.

Falko Timme will be thirty this year. He’s possibly one of the most popular mentors for people wanting to become Linux system administrators on the Internet. His step-by-step tutorials have gained a large readership.

He was born in Celle, Germany. He studied industrial engineering in Dresden and Braunschweig and speaks and writes perfect English, even though it’s not his first language.

I have interviewed a lot of people over the years. I don’t remember an unpleasant one. But this interview with Falko stands out in my mind as the best. I hope you enjoy spending time with him. He’s a remarkable person.

On Digg.com

TA: Falko, your tutorials show up on just about every Linux system
administration search I do these days. When did you start writing
them?

Falko: I started the “Perfect Setup” tutorials in 2003 as part of the 42go
ISP-Manager documentation. At projektfarm GmbH in Lüneburg, we had developed the Linux web server control panel called “42go ISP-Manager”, and though we had an installation manual for it, we got lots of support requests from people that had no idea how to install the base system for 42go.

At that
point I decided “Ok, let’s write a tutorial that covers every single step of
the system installation and configuration so that people can simply copy and
paste the commands from the tutorial so that 42go runs on these systems out
of the box.”

First, we put those tutorials (we started with Debian Woody) on
projektfarm.com and also on my personal web site falkotimme.com. We
noticed that we got a lots of page views from our tutorials. We
even had visits from people that weren’t interested in 42go at all, so we thought that there’s a need for the kind of tutorial we wrote.

We created HowtoForge in 2005 as a source for high-quality tutorials, and it’s open for everyone. Anyone can contribute. I don’t test every tutorial that gets submitted, but I test my own tutorials to verify that they don’t contain errors or that I don’t omit
small but important details.

TA: When did you start working with Linux and how did you get started?

Falko: I started in 1998. I wanted to host my small search engine and
web sites for friends so I bought a Cobalt RaQ2. That was my first Linux
server, and I barely had Linux knowledge then.

It had a nice web frontend
that you could use for basic things, but soon I realized that if I want to
do more sophisticated things with this server, I had to dive in into the
Linux world and get my feet wet on the command line. Many sleepless nights
followed…

TA: I notice you answer questions on a lot of forums. Do you ever sleep?
Where can people go to ask you questions?

Falko: Don’t worry, Tom. I get enough sleep - and I don’t dream of little penguins…

I’m active on the HowtoForge forum, so people can ask me questions there.

TA: Tell us a little about projektfarm.de -what is it exactly and when did
you get involved?

Falko: projektfarm is a company in Lüneburg, Germany that focuses on software for
Linux web servers like ISPConfig. We offer ISPConfig which is free, the 42go ISP-Manager, and the 42go SPAM-Filter. We also offer professional Linux services for companies and individuals.

When a company or individual has problems needing support with Linux
or if they need help with Linux system administration, then they can contact us.
Our job involves providing help and we actually enjoy doing it.

TA: OK. And you got involved with this company how?

Falko: I met the founder of projektfarm GmbH, Till Brehm, in 2001, and we soon realized that we both had a Cobalt RaQ and weren’t satisfied with the
control panel. Neither of us saw any alternatives at the time. Some projects were on the market but they were immature.

So we said, ” Let’s write our own.” That’s a typical Linux person’s response. So we gave birth to
the 42go ISP-Manager. I was fully involved with projektfarm at that time.

TA: I had several Cobalt boxes myself. I thought they were insecure. But now, I have ISPConfig running on my server at home and it’s tight and hard. That must have been an objective in your development.

Falko: That’s correct. Till wrote to you and explained that.

TA: He wrote and said “My server was fully patched and got hacked twice because Cobalt had not released all patches available from Red Hat. These experiences lead to the decision that the base linux system of an ISPConfig server should be kept up to date with the packages from the underlying Linux
distributon without breaking ISPConfig”.

Falko: That allows you can keep your server up to date with the most current patches for SSH, BIND, proftpd, etc.

TA: I want to change the subject a little. We hear a lot about Linux in Germany. And with your country as the second largest economy in the world, just how is the climate in Germany for Linux?

Falko:It’s very friendly, especially on the server market. If you want to
rent a dedicated server here, you’ll find many people who offer Linux. Other providers offer other operating systems, but in the ISP hosting market other operating systems are vanishing.

Also many communities have switched their computers to Linux. I’m sure you know about Munich. Even the servers of the German Parliament run under Linux. SuSE was a German company until Novell bought them.

Linux is widely accepted on servers here, and if you search for Linux topics
on Google, I’m sure you’ll find many German web sites, forums, mailing lists and so on that deal with Linux.

TA: And of course, Howtoforge is one of those sites. OK. Let’s get back to your business. You’re on the ISPConfig team. Tell us about the project and your role.

Falko: ISPConfig emerged from projektfarm’s 42go ISP-Manager. We didn’t sell as many copies as we thought we would. The price was moderate, EUR 189,- per copy.

We wanted to sell more and so we analyzed our business. We believe we didn’t sell as many as we should have for two reasons:

First. we are good programmers, but we are no marketing gurus. Secondly,
the name, 42go ISP-Manager, didn’t connect with people. It just didn’t stick as a brand.

So, we decided to fork a branch of the 42go ISP-Manager. We chose a new name that people could remember. We chose ISPConfig. We also
decided to make it free and put it under a BSD license which, generally
speaking, says that you can take it and do whatever you want, which means real
freedom for the user.

ISPConfig has all the features of 42go and a lot of new features that you don’t find in 42go. It isn’t a limited version of 42go. I think it’s likely that we stop 42go and merge it with ISPConfig and just continue the ISPConfig development.

We made this software free because we thought it would get the
attention it deserved. We changed our business model to sell
support and services instead of software.

TA: That makes so much sense. People have to like what you’ve done.

Falko: With a large user base, you will find users who also want professional support and want to pay for it. Also, many companies and government units require support contracts. So, we see a market of people who will want and need to pay for support.

This business model works better than the previous one. And as an open source project we offer free support on the ISPConfig forum. We have a growing
and active community there.

TA: And the project itself?

Falko: Right now ISPConfig has a development team of 15 people, most of them concentrate on translations. ISPConfig is available in English, German, Spanish, French, Italian, Dutch, Polish, and Swedish. The core team members are Till and me. Till’s focus is mainly the web frontend, mine is more
on the backend that automates configuration. The borders
between these two areas float.

TA: Are you active on any other projects? Or is this enough?

Falko: I am involved in other projects. I just released MyDNSConfig (), a web-based
control panel for the MyDNS name server. This is now a standalone
application. And it will be part of ISPConfig’s next iteration.

TA: On your web site you tell people that they can write you about Linux
support issues, then you say you would accept a small contribution.
You’re personally offering your service for free. How can you do that?

Falko: Basically, I’m trying to help people in the HowtoForge forum. This is a free, growing and very active forum with a very friendly atmosphere where
everyone is welcome - newbies as well as Linux experts.

We manage to find solutions for lots of problems there, but sometimes it’s
hard to guess what causes a problem. If they like,
they can then contact me and ask for my support. I will even log in to their
systems and fix the problem. But this kind of support can’t be
free.

TA:Your tutorials are all on HowtoForge.com. They are also on your web
site. Tell us more about HowtoForge.

Falko: When we had our first tutorials on projektfarm.com and falkotimme.com, we
realized that we got lots of page views because of the tutorials. The
feedback was also very positive. So Till and I decided to create HowtoForge
in April 2005. We put all our existing
tutorials on HowtoForge, and we post our new tutorials exclusively on
there.

HowtoForge is open to everyone and any one can contribute Linux
tutorials. We want to make it a source of high-quality Linux tutorials. I
really think there’s a need for this because very often when I read Linux
tutorials and try to follow them, I do not succeed.

Typical documentation does not describe the exact steps you have to take. I mean I see things like “… then patch the kernel”. But how do I patch the kernel? What are the exact steps?

In other situations people forget to mention some small but important details. I think it’s the usual case. Even I have problems following existing tutorials. So, I wonder how a
newbie or even an experienced system administrator is supposed to do follow the existing documentation and become happy with Linux?

On HowtoForge we publish tutorials that describe every single step.
As a reader you can simply copy and paste the commands to your shell, and
you’re done. This gets the people started, and if they are newbies they can
at least play around with a working system in order to understand how it’s
working instead of pulling out their hair.

If there are questions or
problems, people can go to the HowtoForge forum.

TA: Your tutorials are in English.

Falko: We decided to publish only tutorials written in English on HowtoForge
because English is the language that most people dealing with computers
understand.

Often, when I search on Google for a solution to some Linux
problem, I find pages in Spanish or Portuguese or Polish, etc. and I think
“Damn, they seem to have the solution, but I don’t understand a single
word…” We could have decided to write the tutorials in German, but that
would have been unfair for most people.

TA: Do you have any books planned for the future? If so, what are you
going to write?

Falko: I haven’t planned any books yet. Maybe a compilation of my best tutorials,
but I don’t know yet.

TA: Well, you can write one with me any time you want. And I hope you will let us read more about you in the future. Thank you.

By centralizing a ticket tracking system on the Internet you can create a virtual 24 by 7 Linux support business. You’ll want to use a combination of VoIP and a global clearing provider for taking payments. A system like this costs pennies to start and has the ability to scale rapidly.

The Need for a Friendlier Service Model

In the open source software market, you have to provide excellent service and provide easy access to your support personnel. Business people who believe they can built a viable company on unique software alone will fail. Unfortunately, not many successful service models exist and even consultants in this area have a difficult time managing a service-oriented function.

The best models I have seen use automation to cut costs. But those systems tend to annoy clients. Promising to provide a living, breathing person on the end of a phone call sounds great in a TV Commercial but rarely solves the problem. Have you run into a friendly service provider lately?

To give you an example, I had a lousy experience with a VoIP provider. I tried AT&T CallVantage for several months last year in combination with Charter Communications high speed cable. I discovered that both companies outsourced so many of their services that I could never get the VoIP system to work properly. I had to move back to copper to get reliable service and to bring the costs in line.

My Case Study

Three years ago, I noticed a significant trend of declining sales in the IBM business partner segment of a business I managed. I looked at every indicator I could find and nothing stood out as a problem. While tracking the cause of the sales decline, I noticed that my entire customer service department had go to lunch at the same time and left the phones unmanned for over a hour and a half.

I inquired and found out that in spite of policies to the contrary, the service personnel had followed this practice for several weeks. So, with their desks vacated, I logged on to one of their workstations and discovered the team was 85 days behind answering service requests. I could hardly believe it.

I went back to my programming department and discussed the problem with our head engineer. I asked him to research ticket tracking systems and come up with a high quality solution. Within a day, he came back to me and showed me a system called Request Tracker (RT) by Best Practical Solutions. I had worked with several enterprise level trouble ticket systems at major enterprises such as Gateway and Ericsson, but I had never seen a product of RT’s quality.

My head engineer informed me that the product was free and open source. I felt dazzled. How could someone give this product away I asked myself.

I called a meeting of the development team and posed the problem. We decided to write every client in the old queue. I explained in an email that we had started auditing the service department and found significant delays. I wanted to know if the customer had received help from our company and had they solved their problems.

We used RT to do the mailing and it automatically assigned each customer incident a tracking number. The feedback created some intense situations and many customers felt betrayed. But using RT and splitting up the work load so that each engineer covered a three hour segment, we caught up with the service requests in less than ten days.

Within a week, our sales started to climb back to the levels of the previous quarter. After we caught up, we agreed to continue following the protocol of having engineers devote time to service requests. Our goal was to respond immediately with a ticket number which RT did automatically and to communicate with the customer within 30 minutes.

If I only Knew

Two years before I discovered the issues with my support center, we had built one of the first pay-per-incident Linux call centers. We built the business from scratch, stayed independent and finally became an outsourcer for a major Linux distribution. Initially, we received referrals from Red Hat and Caldera.

I had little problem finding Linux people to man the phones. A call center named Stream had lost a major contract and laid off many highly trained call center employees. As we grew, facilities and infrastructure kept us from scaling.

Red Hat and Caldera saw our business growing and began call centers of their own. Corel had abandoned their Linux business and we had difficulties working with Mandrakesoft. So, we abandoned the call center business and devoted our energies tobecoming a Linux ISV. We wrote Linux software which worked with Outlook and MS Exchange.

After seeing how well we managed our service department with RT, I realized that we could have used it to keep the costs down and to provide better service than we had previously. I wrote another business plan for a Linux support division with RT at the hub. The startup costs were insignificant and I found lots of people familiar with Linux in the Information technology field looking for work from California to Germany.

The model called for setting up a self-service web site with a significant database of FAQs. We also wrote howtos and built both backports and packages that the original Linux distributions refused to offer. The pilot focused on Sun’s JDS Linux distribution.

The success of the web site convinced me to set up a pay-per-incident call center. Ready to launch, Sun decided to abandon its Linux desktop. Prior to that event we had some interesting opportunities.

One of the top five IT Consulting firms asked us to sell them a bundle of pre-paid incidents. We also had similar discussions with resellers and integrators. Because of our low cost structure, we could sell incidents and the channel people could resell them at a significant profit.

A Low Cost Start-up with Significant Potential

Two developments in IT provided a new model for a service oriented business. The first involved the use of VoIP using Skype and Asterisk . The second involved qualified Linux support people available globally.

In discussing the business plan with friends and acquaintances around the globe, I discovered a significant appetite among Linux people who wanted to contract and cover request queues. For example, I found people who excelled at Debian, Red Hat, SUSE, etc. Some excelled at DNS, databases, mail and other specialties. Some were good generalists.

Some of the people who contacted me wanted extra work and some wanted to simply work at home. The key to creating a low cost global communications system involved setting up Asterisk servers in locales where the contractors worked and then centralizing and exchange using one of the free VoIP services like Skype.

Using caching techniques, the request queues fill up and people manning RT can grab a new request when the next engineer becomes available. If an engineer grabs a request but can’t service it, then he or she can pass it to an engineer who can. I’ve seen this work particularly well using RT.

My first call center produced opportunities over and above initial level one calls. In a majority of circumstances, the level one call turned into multiple incidents and/or projects. In fact, two case studies I wrote for Macmillan were the result of rollouts of Linux in two enterprises. We got those contacts because the businesses started using our call center, then our level two and level three technicians. They asked for us to answer tenders and we won the business.

So many people around the globe use Linux and desired to work in a Linux company that we had no scarcity of people wanting to join the business. I plotted out the locales where we could have placed Asterisk servers that could feed into an exchange that our telecommunication costs looked small. In addition many of the tickets closed could feed a knowledge base available from Practical Solutions that attached to RT.

Is this a Feasible Business?

When Sun backed out of the Linux desktop business, I felt that we wasted a year developing our pilot. We also lost the funding opportunities available by selling incidents in advance. Having walked over burning coals in my previous encounter with VCs, I had no desire to work with them again.

I doubt many people would want to work 90 hours a week for three years and have some person tell you to surrender your stock, sign a five year contract at the end of which you could earn back nine percent of your previous holdings. In addition, those people went to the other members of the firm asking them if they would move to another city while failing to inform you.

While I chose to delay a move to into another venture, I still see the value of Request Tracker as the hub of a service business coupled with VoIP technology. This model allows you to centralize your support in a web portal while using a distributed work force. Any software provider wanting to offer support to end-users, channel partners or resellers can use this combination to provide high quality service. Just make sure you man RT and get to your customers quickly.

What Else Do You Need?

By centralizing a ticket tracking system on the Internet and publicizing the business opportunity, Linux people should be able to organize a 24 by 7 support business. You’ll want to operate it like a project and share the revenues. You should also be able to work with Asterisk to build a global VoIP exchange while clearing payments. I hope you will look into this as Linux is growing and users need support and are willing to pay you for it.

When someone writes an article for a publication like Lxer, they need to use an expository style. Expository articles expose or reveal a subject through facts, reason and argument. When you use an expository style you deal with an event, concept, or idea using facts and examples and not opinions.

When you write an expository piece it does not have to be dry and boring. Your observations and experience often form the basis of your finding in the first place. Using your experience humanizes your writing and makes for an interesting read.

The Format

An article contains a title or heading, lead, conclusion, a body of paragraphs and headings and an ending. Most articles run between 400 and 800 words. Some articles contain 1500 words or about four pages of writing on a word processor.

Consider the heading of your article a meta-phrase. When you write a heading try to summarize your point in three words. That may be impossible but if you can do it, you will catch a readers attention. Sometimes, you will want to use a working title until you finish your article. Afterwards, you will find writing the title much easier.

When you write a lead, remember that it summarizes the topic while saying to the reader that some conflict exists. You tease out something about the drama so the reader can see value in reading the article. Your have competition for the reader’s time so let them know right away what they will get if they read your piece. Try to stay with three sentences in your lead.

Write your conclusion at the top of your article body after the lead. For example, you might start your article with a paragraph like this:

After interviewing the participants Forrestor used in the survey, we found discrepancies in their conclusions. When asked, 78 percent of the people surveyed said they used Linux in mission critical applications instead of 34 percent. In addition, the original surveyor biased the outcome with his or her introductory remarks.

You might consider starting an article with a conclusion odd. It does serve an important function. It tells the reader that facts, reasoning and investigation have yielded a result that affects them.

Next you begin your body of paragraphs and headings. Start with the work you did. Tell the reader that you investigated something. For example, you might simply write:

During the past six weeks, we investigated an Enron partnership still running in the Bahamas. The tip came from a former attorney formally connected with the partnership. We found the original partnership filing documents in Austin and gained access to the partnership bank accounts. We tracked down the payments and verified them with the Secretary of State.

I don’t expect you to have found such a tip, but the above example gives you an idea of how to describe your work. You could also say that a friend of yours told you about a new database. You downloaded it and installed it on an instance of Debian Sarge 3.1 r2. You ran a series of tests, created tables from MySQL and benchmarked performance of both databases.

Now you have informed the reader what you did. You can proceed to write the body of paragraphs and headings. This will explain your finds and allow you to make a case.

Expository paragraphs usually have three sentences. On occasion you can have more than three sentence. You would only add additional sentences if you needed to support your paragraph’s topic further.

The previous paragraph demonstrates my point. We call the first sentence the topic. The other two sentences support the topic sentence.

Notice that the first sentence declared the topic that expository paragraphs usually have three sentences. Then we wrote something about the topic sentence. The support sentences gave information about when the assertion in the topic sentence might differ.

You have already given your conclusion and stated the work you did. That should provide ample subject matter to continue writing your article. Start with the most interesting and/or important information first.

I like to add drama or tension into an article. You can accomplish this by providing information in small chucks. You want to provide a way to get to “who done it”.

When you follow the who-done-it approach you simply eliminate one possibility at a time. You start with the plethora of possibilities. You then say why one possible or another doesn’t answer the question you raise.

This differs from a simple news item. In a news item you simply announce an event. You can do so in a few sentences and usually two to three paragraphs.

If news leads to a bigger story then you have the basis of an article. An article allows you to explain something that no one has solved before. Your article should solve or nearly solve that something.

I investigated Microsoft’s political activities for nearly three years after I encountered their lobbying effort in Texas. I helped introduce an open source bill. Then I saw the dirty tricks for the first time.

I found ample information about Microsoft’s attempt to influence the case against them in Federal Court. It didn’t seem to make sense. I continued to collect the information I found hoping it work make sense.

I saw a small news article about a former Preston Gates lobbyist who had something to do with Tom DeLay’s potential ethics hearing in Congress. I searched for information on that that lobbyist and discovered that Preston Gates had paid an American Express bill for part of one of DeLay’s trips trips to Scotland. That allowed me to put the information I had gathered into a cohesive argument and gave me a scoop on Jack Abramoff.

Once I had a conclusion, I wrote an article and presented my facts. The facts lead to a way of reasoning and an argument. When comments surface that said Microsoft acted like any other corporation I had a counter argument.

Other corporations make contributions and lobby Congress. At the time the events occurred, Microsoft faced a breakup of the company. They had a different incentive for their intense lobbying effort.

When you structure the body of your article look at the kinds of thinking that exists in your readership. Will you article help them interpret the facts? If so, then start writing.

When you have eliminated the possibilities of who-done-it, you can begin the ending. You will find the ending much like the conclusion at the start of your article. You want to write an ending using less formal language and a clever statement.

Writing for a publication like Lxer requires an expository style. If you understand the guts of that style it can offer you a fun experience. Now, please go off and write something for me.

Last fall I wrote an article entitled “Critical Shortage of Linux Talent Slowing Adoption”. The article used a parody, a spoof about human resource management. I wrote:

Most human resource people believe Linux is an air conditioner company. They get confused between the term Linux and Lennox. So, HR recruiters define their job profiles like this:

Linux programmer needed by enterprise. Skills required:

REFRIGERANT METERING DEVICE CALIBRATION
LEAK TESTING
LIQUID & SUCTION LINE SERVICE VALVES KNOWLEDGE
START-UP
CHARGING FOR TXV SYSTEMS

Five to ten years of relevant training and master plumbers’ license required. Will accept equivalent for H1B applicants. Microsoft Certifications a plus.

The article title has become an urban myth and from the comments I have read about it, most people took the title to heart and never read the article. So, let’s set the record staright.