Related link: http://lxer.com/module/newswire/view/38971/index.html

One of the top stories of the year at LXer warned mightily of Microsoft’s capabilities to change the political landscape to its advantage. I spent three years researching the material in that article and spoke to so many government people that I had to keep maticulous records.

My first glimpse of the political machinery behind Microsoft came when I helped sponsor a bill in Texas to open IT procurement to open source and free software. My sponsoring senator was John Corona. At the time, all we wanted to do was get some budget money freed up to use in areas Carol Strayhorn, the state comptroller, was cutting from social services.

Part of my research involved tracking budget cuts and finding spoilage in the state budget. But then, I did one better. I went down to the county hospital and saw a doctor and got some medicine. It took two days waiting in line to see the doctor and two days waiting in line to get the medicine. People died waiting and that’s in America.Thousands of people milled around sick in those waiting areas.

I met people with cancer who had no choice but to wait because they had no other way to get their medicine. The sites and sounds will always haunt me.

But back on the floor of the Senate in Austin, representatives from all over the country; people that had nothing to do with Texas, testified in their two thousand dollar suits about the evils of open source software. It was a finely tuned and well rehearsed progression of Microsoft shills that spoke with furry against letting the state Department of Information Resources use Apache.

So, the State of Texas cut social services and continued funding overdue and over-budget IT projects using Microsoft technology. No matter where I go and no matter to whom I show the benefits of using free and open source software in the government, Microsoft is there.

Remember Peter Quinn, the Secretary of the Commonwealth and Senator Marc Pacheco

Anyone doubting the power of Microsoft, should consider what we wrote at the end of June 2005. We’ve brought it back to demonstrate its relevance and rewrote pieces to tie in the events of the last few weeks. But the documents we uncovered are still in place. The people within Microsoft’s grasp politically are still listed. This isn’t a story you scan. This is one you read.

When you finish reading it, ask yourself if any company should be allowed to conduct business the way Microsoft has.

And remember what Franklin D. Roosevelt wrote preceding the Standard Oil Monopoly litigation:

The liberty of a democracy is not safe if the people tolerate the growth of private power to a point where it becomes stronger than their democratic State itself. That in its essence, is Fascism - ownership of government by an individual, by a group or by any controlling private power.

Referenced Article

Related link: http://lxer.com/module/newswire/view/50721/index.html

In a song called “It’s alright ma, I’m only bleeding”, a
young Bob Dylan says “He not busy being born is busy dying”. That became a mantra in years past for being into personal growth. We see less of those movements today. In fact, critics of the human potential movement have positioned it as weird.

So if you have an interest in personal growth and don’t want to wear the brand of a weirdo, you can start learning Linux system administration. I don’t mean learn about it, I mean do it.

How So?

Put together a small server, get a dedicated Internet pipe into your home with static IP addresses, register a domain name and build yourself a server on the Internet. Push yourself out there and start learning what Linux can do away from the desktop.

With some people, that’s the equivalent of jumping out of an air plane or learning to do ropes on a ten thousand foot mountain. Just remember that fear is often just a set of body sensations designed to keep you stuck. You can be afraid and also accomplish many things at the same time.

Where Does One Start?

The paragraph above gives you a summary of the steps, but you still have to put an operating system on the box you choose. I started with an used Intel box with a CPU two generations older than current models. I put in a new hard drive, added memory and went with a no frills version of Linux.

The people over at HowtoForge have a number of step-by-step guides to building a Linux server. One recent tutorial you might find interesting is The Perfect Setup: Mandriva 2006 Free Edition.

The description reads: This is a detailed description about the steps to be taken to setup a Mandriva 2006 Free Edition based server that offers all services needed by ISPs and hosters (web server (SSL-capable), mail server (with SMTP-AUTH and TLS!), DNS server, FTP server, MySQL server, POP3/IMAP, Quota, Firewall, etc.).

If you don’t like Mandriva, you can chose ISP-Server Setup - Ubuntu 5.10 “Breezy Badger” or The Perfect Setup - Fedora Core 4. They also have some tutorials for SuSE.

The HowtoForge tutorials will just help you set up a distribution as a server that’s ready for deployment on the Internet. You can ease into some of the things you need to learn by adding ISPConfig
to your server as described in the HowtoForge tutorials. ISPConfig is a free web based management tool similar to one used by Cobalt. It’s not an extensive web based tool and I recommend using it as a walker to get started. Don’t stop with ISPConfig and make it your administration GUI.

Time to Add to Your Skill Set

The next step after getting your system setup is to learn Domain Name Services. I often refer to it as the parent process for so many others. DNS will test your fortitude because the terminology will not seem intuitive to you. So, find a tutorial such as this one and start dragging yourself through the mental fog until DNS makes sense. Then write your named.conf and zone files and get your web server ready to post some web pages to your site.

Once you have embraced DNS, you’ll want to learn Internet mail services and Apache. This guide provides a very useful approach allowing you to use postfix and fetch mail with Gmail. The Apache foundation has a number of tutorials to get you stated and take you deep into web services.

If that’s not enough, you’ll find a plethora of information on the Internet to help you with these. Learn to manage these by hand without the help of web based administration tools. Otherwise, you’ll never get a sense of the nuances of Linux system administration.

What Should You Do After That?

If you get through the first tasks outlined above, you’ll have achieved incredible personal growth. You’ll also have a lot of self confidence you may never dreamed possible. So, at that point, pat yourself on the back and see how much more money you can make.

Knowing how to set up a server with the services mentioned above and then mastering DNS, Mail and Apache should provide the foundation for moving in the direction of your choice. I recommend going after database administration. As a system administrator, developers will expect you to set up a database server, create a client and provision a database instance for them to use. You have a lot of choices, but I
suggest learning postgreSQL or MySQL first.

You’ll also want to learn how to do network administration, which will overlap with what you already learned. But, in this area of systems administration, you’ll learn to manage users, groups, file systems, hardware, routers, firewalls, samba, etc.

Should I Say More?

Since this was my way of challenging you to step out, I won’t leave you hanging. In fact, you’ll see much more information about system administration in the days and weeks ahead. You can do this and I promise to keep prodding you.

First things first. Take a look at the HowtoForge tutorials and get your system set up on the Internet. I don’t care which system you choose. Just make a commitment and do it.

Related link: http://www.sans.org/sans2006/

In another minor bit of self-promotion, I wanted to bring to your attention this fun presentation that I will be giving at SANS 2006 in Orlando, FL. The title is “Baselining Logs and Audit Trails for Security.”

Many people, when asked about log analysis, say “you’ve got to create a baseline first”, but few clarify what it really means. I try to address that information void by presenting the results of my research.

Here is an outline: “This presentation will focus on creating the methodology for learning the log baselines and then matching the current state of the environment against the baselines. It sounds simple, but an effective methodology for it still hasn’t been created. The talk will cover what the good (and bad) possible baselines are, how to create them and how to use them for security.”

Note, that some info might overlap with my previous presentation on log mining in 2004 (See “Log Mining: Knowledge Discovery in Logs”

The time is Wednesday, March 1, 7:00pm-9:00pm

Related link: http://www.networkcomputing.com/shared/article/printFullArticle.jhtml?articleID=…

Its the end of the year, and I am doing my end-of-the-year sweep for security predictions. People predict a lot of fun and sometimes outright silly stuff that will supposedly happen in security next year. I will post my full review in January, but this is one fun piece I liked: “Survivor’s
Guide to 2006: Priority No. 1: Data Protection” in NWC Magazine

The sheer number of breaches and other events which resulted in data theft or data destruction this year is staggering. However, I suspect that it has also something to do with increased attention and heightened expectations that a loss will be reported.

Other fun predictions I’ve seen will also be posted soon. In fact, feel free to check my del.icio.us feed for all the prediction items I tagged:
Views on Security in 2006

Related link: http://paulgraham.com/procrastination.html

I eagerly await each article that Paul Graham.com posts to his web site. They are brilliant, educational, fun to read. Plus I have to admit that since I wasn’t in Silicon Valley for the dot com bubble, I like to live vicariously through his recollections and stories about those years. Sit down and read every essay in one sitting and you’ll feel like you just got an MBA with a specialization in startup-ology. Pretty cool stuff.

In this month’s article he makes many excellent points about procrastination. The point of his article is to say that if you send all your time working on little stuff and errands you’ll never fix the big problems. This is correct. However, his advice about how to skip the little stuff is to just “let delight pull you” to the difficult problems. That’s a beautiful and poetic platitude, but useless the average system administrator or technical grunt that is flooded with zillions of tiny tasks that can’t just be “put off” because if we do, our employers will be very, very, unhappy.

While I think his advice is good for people that are brilliant scientists and inventors of web 9.0, I think system administrators are of a slightly different mindset, and a very different problem.

The biggest time management problem for system administrators is interruptions. Your boss values your ability to get projects done but the people you serve value your “availability” (how easy it is to interrupt you so that you’ll focus on them). These two things are in conflict with each other. There are many ways to fix this. My favorite is the “mutual interruption shield” technique. You catch all interruptions for a co-workers in the AM, letting him or her do project work. Then you switch in the afternoon, i.e. you focus on projects with the support of your co-worker catching all interruptions. System administrators often need some kind of management buy-in to get this kind of structure set up. If you can’t get management buy-in, at least structure your seating arrangement so that people that want to interrupt you have to walk by your co-workers before they get to you. Even if they have to pass by a single sacrificial co-worker to get to you the reduction in interruptions can be significant. (Note: buying them t-shirts with a big target on them might be going too far.) TM4SA includes additional techniques, each requiring a varying levels of management buy-in.

The next biggest TM problem that system administrators have is the inability to prioritize. If you have a written/PDAed todo list, you have a chance to start prioritize. It is incredibly difficult to prioritize if you don’t have your little tasks, big tasks, and your your life-time goals written down. I spend four chapters in TM4SA on “The Cycle,” a simple system for tracking “todo’s”, “events” and “goals”. (”Four chapters is simple? Ha!” Ok, ok, it’s a lot more simple than the number of chapters would indicate. To be honest, I was told ORA would license one UserFriendly comic strip for each chapter, so I kept the chapter short and made lots of them.) The side-effect of recording your todo’s and goals is that you actually can start to prioritize. This is why the chapter after The Cycle is on (ta da!) prioritization!

Paul writes,

When I talk to people who’ve managed to make themselves work on big things, I find that all blow off errands, and all feel guilty about it. I don’t think they should feel guilty. There’s more to do than anyone could.

I appreciate what he’s saying, and I agree that people shouldn’t feel guilty, but I go about solving the problem 180 degrees differently. I find that I can’t focus on the big things if I have too many little things on my todo list. The little things bother me whether real or perceived reasons, or just because they’re cramping my brain leaving little space for my tiny brain to work on the big things.

In college I got most of my big projects done between midnight and 4am. This is not because I’m a “night person” (I’m not). It’s because I would spend all evening whittle away all the small dumb errands that I had (laundry, cleaning my email box, putting up posters, and so on) untiil they were eliminated and I had no other excuse but to work on the big project that I should have been doing in the first place. Also, since so little new email would arrive after midnight (my smarter friends were asleep, or studying, or doing other non-email-generating tasks) I wasn’t tempted by that constant distraction. With my todo list eliminated, the only thing left to do is the big project that should have been my top priority.

Now that I use The Cycle things are very different.

The Cycle encourages you to make 365 todo lists per year. If you have a PDA, software like DateBook5 makes it easy. If you have a paper organizer, each month you load the next set of 30 Page Per Day refill sheets.

Here’s how I start each day. I look at today’s todo list and say, “Sweet Jesus! That’s more than I could do in a week!” So I mark each item with an A, B, or C priority. Then I move the “C” priorities to future todo lists. That new software package I need to experiment with? Move it to next Wednesday’s todo list. That script to automate such-and-such? No harm will come if it doesn’t happen until Monday, so I put it on Monday’s todo list. Calling that salesperson about blah-blah-blah? Move that to tomorrow’s list. If there is still more work than can be accomplished today, I do the same for the “B” priorities too. Look Mom, I’m prioritizing!

I mentioned that interruptions take a lot of my time, so when I calculate “how much work I can get done today” I include 2 hours of “interrupt time”. That’s about average for me in my current job. If it turns out that I have less than 2 hours of interruptions today, I have spare time at the end of my day. W00t. If an interruption wipes my entire day, at least I know I can look at today’s list and quickly triage the things that have to be done no matter what. Now I’m prioritizing and I’m dealing better with interruptions. Joy.

If you recall I mentioned that in college I couldn’t focus on a big project if I had any little tasks to do. Here’s where The Cycle saves me. I just move all those little items to tomorrow’s todo list. Ha! Takes a minute to do, and gets them “off my plate.” In college I would have spent all night actually doing those tasks but this is much better. I get to the important big project faster, and yet I don’t lose those small items “by mistake.” While they aren’t “done”, they are “managed”, which is much more important than actually doing them. Once they are “managed” off today’s todo list, they are no longer making me crazy or otherwise making it emotionally difficult to work on “the big project”. No guilt.

That bears repeating: the important tool here is that you are shifting from “I want to get everything done” to “I want to make sure all my tasks are managed”. “Managed” might mean picking another date to do something, deciding to never do it (and notifying those that will be affected), delegating it, or pushing it back to your boss (politely, of course).

Now we can add a task to be managed called, “work on the big things”, schedule 2, or 4, or 8 hours of it a day, schedule it, or do it guilt-free when the inspiration hits us.

Don’t procrastinate: tell me what you think!

Related link: http://lxer.com/module/newswire/view/50521/index.html

As the Internet’s evolution moves in the direction of XML we are all faced with a scary proposition: XML wrappers. That’s been the subject of great debate in places like Massachusetts and in the back waters of the OASIS OpenDocument Format discussion forums.

None of us should focus on open XML wrappers for document formats. Instead, we should be concerned with what goes inside of those wrappers, because if Microsoft is allowed to go forward with its standard, they win and the world loses.

What’s this about?

Microsoft cleverly proposed an Open Document standard that
would exclude everyone by setting a requirement that Word and
PowerPoint formats preserve legacy support for Microsoft’s prior aborted attempts at XML. If you would like to see Microsoft’s filing,you can download it from this web site.

In Section 10.3.1 of their proposal to ECMA entitled “Alternative Format Import part” allows a WordML file to directly embed content from a legacy file format such as RTF, MHTML, or earlier WordML formats. A conforming application would be required to read and understand these
legacy formats.

If Microsoft’s schemas are licensed royalty free only to conforming applications, and conformance require support for fragments in older abandoned formats like RTF, WorldML and VML, then that would make it impossible for anyone to use these formats other than Microsoft. That would allow Microsoft to recapture its sole possession of the productivity market and the standard for Internet content.

Following is a passage taken from the writing of Gary Edwards, the OpenOffice.org representative who helped start the OASIS OpenDocument Format technical committee and who formed the OpenDocument Foundation, Inc. Gary explains the difference between Open XML and MS XML. Read carefully.

“As we move the focus of discussion from the traditions of software and platform bound binary desktop productivity file formats, to that of Open Internet ready XML, the differences between ODF and MSXML become decidedly stark and clear.

“ODF is a ‘wrapper’ of Open XML technologies, and specifically states so in the charter. MSXML on the other hand, is a wrapper of proprietary technologies. Even if the ECMA rubber stamp effort were to become open and unencumbered with multiple vendors and users participating in the standards process, there’s still the problem of all the proprietary dependencies wrapped in MSXML.

“For instance, where ODF implements W3C XForms, MSXML uses a WinForms - InfoPath derivative. Where ODF implements W3C SVG, MSXML is geared to the up and coming proprietary “sparkle”. Where ODF uses standard HTML, MSXML embraces the bastardized MSHTML. The list goes on and on, with one point becoming increasingly clear: Microsoft continues to embrace and extend open standards with proprietary enhancements designed to break both compatibility and interoperability with everything outside their OS Stack.

“Microsoft insists that the world can live with two different desktop productivity XML file format standards. The problem is that this isn’t about desktop productivity. It’s about the Open Internet. And who among us wants to relive the nightmare of shamelessly self-serving Microsoft inspired incompatibilities?”,2.0. There is no “Live Web”. There is no “next generation of collaborative computing”.

“XML isn’t going to provide us with a better PC based desktop productivity environment. It’s going to provide us with a means of meshing the desktop productivity environment with the Open Internet. XML ready desktop information engines become first class Open Internet ready participants. That’s the ‘why’ behind this urgent and rather dramatic rush to dump our traditions of binary formats and race to XML.”

The world has changed overnight and few people understand it. But the people at Microsoft understand it and they know how to take advantage of a technical ignorance in the population.

If they take their time and play it cool, they wind up with all the marbles. As Gates stated about Linux, on October 1, 2004, at an appearance at the Computer History Museum in northern California, when someone asked about a possible threat from Linux, Gates replied: “Microsoft has had competitors in the past. It’s a good thing we have museums to document this stuff.”

Also, I’ve been playing with various blog tools and the more I do it, the more I think that O’Reilly should upgrade the blogging system…

Technorati Profile

Related link: http://news.com.com/Computer+forensics+tools+maker+hacked/2100-7349_3-6001756.ht…

Not much to say about this one; its all in the article and in the subject of this post.

I’m sending out Christmas cards, filling in the envelopes by hand; and I ask myself how my own geekiness changes the simple process of writing on an envelope. That is, if I wasn’t a geek, I wouldn’t give any additional thoughts on the subject. But, since I am, I always want to find ways to make it better (or if not better, then cooler).

First of all, am I the only guy out there that goes the extra mile to find the zip+4 code for US addresses? Unless I’m sending the cards in bulk, I don’t think there is an improvement in delivery time.

I’m also a little bit of a barcode geek, so I’m kind of bummed out that while OpenOffice 2.0 will let me print out an envelope with address fields, it doesn’t have any built in tools to throw in a PostNet barcode. Those barcodes could probably speed up the delivery of my cards by 50%. Heck, the format is so simple, I could probably just draw the lines myself.

Of course, my non-geek friends will probably see through my arguments, and say something along the lines of, “Why did you wait until now to try and send out those cards?”

Related link: http://lxer.com/module/newswire/stories/viewstory.php?rid=50221

I consider the Fourth Edition one of the best Linux books
available. O’Reilly managed to keep it down to 672 pages. Aside from
updating those pages, the new edition has another 300 pages of excellent material. That’s how much increased capability Linux has acquired over a two and a half year period.

When I heard the rumors that Adam Oram had started pulling the team together for the latest edition, I wondered about that. I had just read a review of the Fourth edition lauding its relevance to Linux users.

Then Andy gave me a call and asked if I would have an interest in filling in some specialized areas. That’s when I realized O’Reilly planned to update Running Linux again. I felt honored to be a part of it.

A little History

O’Reilly & Associates began publishing Linux books early. Andy joined the company back in 1992. One of his first projects involved publishing The Linux Network Administrator’s Guide which Olaf Kirche began writing for the Linux Documentation Project in 1992.

Looking back, one has to admit that printing a book that anyone could read or download from the Internet seems somewhat bold. But, the good people at O’Reilly & Associates published The Linux Network Administrator’s Guide back in 1993 and released the Third edition this year. I have always considered such moves remarkable considering how other publishers
operate.

Running Linux came out in May 1995 and got another update in August 1996. I thought the Third edition published in August 1999 provided some major insights. I had just started a pay-per-incident Linux call center and finding Linux books at that time was not that easy. Most of the books people published seemed like compilations of
materials already on the Internet.

The Fourth Edition seemed like a fitting culmination of the
previous work. It continues to sell and people continue to use it. So,preparing a Fifth edition seems like another O’Reilly commitment to excellence and Linux to me. I think they could lay back and let the book continue to just sell.

Available in Early January

Writing a book for O’Reilly requires more than a monetary
commitment. The editors, production staff, reviewers and contributors work hard. Sometimes I feel like a kid in a factory watching people build complex machinery when I observe Andy Oram putting a book together. Writing an O’Reilly book requires a team effort and lots and
lots of work, especially by the editor.

Knowing the magnitude of effort required in writing any O’Reilly book allows me to know that their commitment to Running Linux means people will get a great book. If you want a glimpse, you can see Chapter 6: Electronic Mail Clients. You can also see the catalog page on the O’Reilly web site.

Give it a Look
If you have a passing interesting Linux or if you’re a seasoned veteran, you like Running Linux Fifth Edition . It’s not only a good book to read, it’s an excellent reference. Linux
will continue to grow at exponential rates. If you work for a company that wants you to have Linux experience, I suggest you pick Running Linux as your guide. You’ll find it worth the wait.

Originally publihing on LXer.com with republication permission granted.

Related link: http://www.theregister.co.uk/2005/12/15/unix_gcg_study/

A new study on the major players in the Unix server market has declared IBM the clear customer favorite and brought to light some serious issues with Sun Microsystems’ product line. Most alarmingly for Sun, the company appears to have lost its cachet as the dominant Unix player and done so while alienating customers. Sun finished last in almost every one of the Gabriel Consulting Group (GCG) survey’s categories, spanning technology performance, customer satisfaction and software tools.

Related link: http://lxer.com/module/newswire/view/50001/index.html

Not much has changed since Microsoft started selling preload software. Today, we cannot know if people would still buy Microsoft products because the Government protects the monopoly. What percentage of the market would Microsoft have in a fair market?

The only way we can answer that last question is to stop manufacturers from preloading Windows. Until then, we do not have a free market. Microsoft has no way to prove itself otherwise.

While we can see the power of preloading an operating system today, we didn’t think much about it in 1992. Today, 95% of the people who own computers use what comes on their PCs.

Preload agreements seemed like a clever marketing tool. In the least they gave someone an edge. At the other end of the spectrum under the Sherman and Clayton acts they should be potentially felonious.

Back when these agreements started, preloads gave OEM’s a good price on DOS. But OEMs had to agree to buy Windows, too. They would have to pay for DOS and Windows for every computer they shipped, whether DOS and Windows were on it or not. We don’t know what agreements exist today because disclosure goes beyond the scope of the US Government’s enforcement of their agreement with Microsoft. Everything is done in secret without public disclosure.

What do Linux folks want?

*We ask the US Administrative branch to cease and desist from lobbying on behalf of Microsoft.

* We ask the US Congress to evaluate the administration’s role with regard to Microsoft.

* We ask that Microsoft be prevented from continuing or entering into preloading agreements.

* We ask for a comprehensive review of Microsoft’s political activities.

* Finally, we ask that all OEM manufcaturers disclose their policies with regard to selling Microsoft products. We feel that such disclosures should be included in companies’ SEC filings. We also believe that separate schedules of payments to and from Microsoft and OEMs be disclosed immediately.

That might even the playing field.

Digg Story

In the spirit of the new O’Reilly Emerging Telephony site, I thought I’d share one of my best (or worst) VoIP horror stories.

Before my current employ, I worked at a telecommunications company that wanted to make headway into the VoIP market. The company invested in new network gear, consultants, training, and infrastructure. Their goal was to be the best VoIP company in the whole state.

We filled the market with buzzwords like converged, and QoS. We held parties to show off our infrstructure, in the hopes of landing a nice big corporation, or maybe even a government contract. When the product rolled out, they had about ten nice sized customers. So far, the product rollout looked successful.

But all of that would change on one cold, normally uneventful February day. I was sitting at my desk working as always, when the network connection to my PC went down. I double checked the link light of my network card, but the problem was really confirmed when three or four co-workers asked aloud if there was a problem with the network.

The network support group tried to call the main office to see if they were having network problems. And since I said tried, you can guess where this tale is heading. No dialtone. No nothing. The Cisco 7940s sitting on everyone’s desk suddenly became $800 paperweights.

We reached the other building by calling the cell phone of a co-worker with another cell phone. They were aware of the problem, and a network engineer was coming over to look into the situation. It was very likely the core router took a nosedive.

It took about another minute for those words to sink in. If the core router is down, would that mean that all of the other customers using our VoIP service are down? But, if they’re down, why don’t they call?

Because they can’t.

If you’ve ever worked for a big ISP, you know that the phones will usually jam up whenever there’s some kind of major outage. A large queue of holding calls is a pretty good indicator that there’s a big network problem.

This outage was different. This outage was filled with nothing but an errie silence. The network support team had a major outage, but there was no way to guage how the customers were coping. We never even knew if they were aware of a problem.

The customer database was unreachable, so we couldn’t proactively call. We had some contact numbers written on paper, so we reverted to cell phones. That worked, except for the customers that had unwittingly used their VoIP number as an emergency point of contact.

We were able to confirm the worst. Everything was down. Networks, phones. The new network infrastructure that was built from the ground up had died.

It was rumored that the head engineer was on site and working to fix the problem. He carried with him the tell-take laptop with a light blue RJ45 serial cable trailing behind him. It was also rumored that the Chief Operating Officer was right next to him, wanting desperately to be kept in the loop. They would both be staring at a 15 inch LCD screen with cell phones pressed up against their ears.

About four hours later, the problem was fixed. I’ll admit I don’t remember what the actual cause of the problem was, or the resolution. It could have been an IOS bug, a routing table that had gotten out of hand, or just a bad configuration upload. At the time, we were just happy to get the phones working.

The aftermath wasn’t pretty. Some customers were completely unaware until we could finally call them. Miraculously, none of the customers left us. New promises and reassurances were made, along with the expectation that things would get better down the road.

After the incident was over, things around the company changed. New procedures went into place, policies would be updated, and the upper management would be keeping a watchful eye over everything that happens in the future.

I’m not writing this to disparage VoIP, VoIP product lines, vendors, or providers. I am only writing this because this was my most detailed memory of my brief encounter with a VoIP environment. I still think it’s a great technology, but there needs to be a higher level of maturity and stability before I’m ready to adopt it for personal use.

Worried about having to pay BMI or ASCAP fees if I used music in a podcast or video that I was thinking of making. Then I found the solution: UniqueTracks.com

UniqueTracks.com provides pay-once, use-many royalty-free music. Their website is very searchable. If you need music for a rainy scene, or a happy time, just plug those words into their search-engine and it will find a couple selections for you to preview. Purchasing can be done with most major credit cards. The download starts once payment is confirmed.

The previews are in Flash, QuickTime, and RealAudio. My browser (Safari on a Mac) seems to work best with Flash. I’m glad they provide many choices.

If you prefer CDs, they have discs that give you a variety of themes all in one place. For example, Themescapes has tracks that are “Uplifting, Motivational, Festive, Contemplative, Mysterious, Good Times, Sensual, Action, Inspiration.” Cool, eh?

You can see which music I selected here.

Comments are music to my ears. What do you have to say?

Related link: http://lopsa.org/book_club_200512

LOPSA has named Time Management for System Administrators as their first ever “book of the month.” I’m honored.

[To get this info our without further delay, this might not be as proofread as usual.]

The Usenix LISA 2005 Conference is happening this week. This is the big Open Source system administration conference. I’m trying to summarize each day’s activities. Wednesday began with me receiving an award.

While the conference has had activities since Sunday, Wednesday is when the “technical session” begins and is thus the traditional “start of the conference.”

Opening Plenary

Chair’s Welcome: The keynote began with a presentation by conference chair David Blank-Edelman. It was hilarious and informative. He explained how the conference comes together and thanked the many people that make it happen.

Awards: He then introduced Doug Hughes who presented Christine Hogan (now Lear) and myself with the SAGE Outstanding Achievement Award for our book The Practice of System and Network Administration (see here). The Chuck Yerkles award for community involvement to Brandon Allbery for his helpful involvement on community forums.

Keynote: The keynote was by Qi Lu, VP of Engineering at Yahoo! Inc. His 1-hour presentation explained Yahoo!’s move to search that includes public (the web), private (your information), and shared (your friend’s) information. For example, with Yahoo! Toolbar, when you visit a page that you like you can bookmark it, which records it into your Yahoo!-stored bookmark. You can add your own keywords and such, and mark it as viewable by your friends. Your friends do that too. Now you can do a search that takes into account the recommendations and tagging of your info and your friend’s info. You have to see it to understand it, it looked really cool. Then he delighted the technical geeks in the room by explaining many of the algorithms involved. Doing that kind of thing sounds easy but once you realize that it all has to be done in real-time it becomes very difficult. He explained how their distributed computing platform works, how they use a Bloom Filters to do some of their trickiest work.

Invited Talk: Computing on Amateur Satellites

Bdale Garbee spoke about the history of amateur satellites, which surprisingly have been around since 1961! The computing on these things is very interesting because it has to be simple, inexpensive, and not require service. Oh, and since you can use a fan to cool it (think about it… you are in a vacuum) you have to get rid of heat other ways.

He didn’t make comparisons to the work we do back on Earth. That’s ok. At LISA, most IT talks like this don’t. Some ITs teach, others entertain, others just spark creativity. This is in that last category. Relating it to our careers is the homework.

The most interesting part was the last 10 minutes when she showed that a new project will send an amateur satellite to rotate around mars. That just blows me away. I predict that if they are successful, the political ramifications will be huge and may lead to democratizing the space exploration process.

Invited Talk: What big sites can learn from small site

I gave this talk. It was based on my experience at my last two jobs where I got to work at a lot of different networks. One of which was in crisis mode when I arrived. I explained that when you are at a big site having every bit of infrastructure is important. However, at a small site you need to prioritize in an even more highly-focused manner. Working in a crisis situation crystalized for me what the key priorities are: (1) get stability, (2) get basic infrastructure in place, (3) clean up and deploy second-priority infrastructure items, (4) growth. Anything you do to get stability will be replaced when you are getting infrastructure in place, thus triage is important.

Invited Talk: What small sites can learn from small sites

My friend Strata R. Chalup gave this presentation.

She had an excellent quote which I think summarizes how I feel often:

“Suffering increases in proportion to knowledge of a better way.” -Jim Hickstein

Wow. At every conference there is always one thing that I learn that is so valuable that it makes me blurt out, “ah! That just paid for the conference.” For LISA2005, it was that quote.

She talked about so many good topics that I can’t remember them all. Hopefully I can get a copy of her slides later. She had a really good idea about priorities in trouble-tracking systems. Nobody likes to list their request as “low priority”, so instead create categories that have a number and a name particular situations: “01: end-of-quarter financial report impacted”, “50: new user creation”, “99: on hold / tracking”. This requires extra work to investigate what the company’s priorities are, but it is worth it.

Meet The Authors

I hung out with Randal Schwartz, Brent Chapman and other authors and autographed books. Ooooh… feed mah egooo. :)

Dinner

I had dinner with my future co-workers at Google at P.F. Chang’s. This conference facility is walking distance from the Fashion Valley Mall, which has a lot of food options. We had a blast.

BoF: Oslo University

Norway’s Oslo University had a BoF (Birds of a Feather) session to promote their “Masters in System Administration” program. They use my first book as a text book, so I promised to stop by and say hello. However, I didn’t get back in time.

I ended up going to a party sponsored by LOPSA. The theme of “Cheers” (”where everyone knows your name”) comes to mind.

More updates in a few days.

Related link: http://www.bsdcertification.org

Wow, I can’t believe it’s been over three months since the last time I had a few moments to blog. Where did the last quarter of 2005 go?

As some of you may already know, a fair chunk of my time is being spent as the acting chair of BSD Certification Group Inc., a registered non-profit creating the standard for assessing the skills of BSD system administrators. We’ve accomplished a lot in 2005, but still have a very busy 2006 ahead of us.

As our first year draws to a close, I’d like to take a light-hearted and slightly tongue in cheek look at what it is like to be involved in such a massive undertaking. If you’re involved in an Open Source project or sell/advocate Open Source solutions, you’ll recognize where I’m coming from.

#1 Where’d everybody go?

Have you ever tried to pin down a few dozen people to discuss an important topic or vote on a decision? Oops, forgot to mention these are all volunteers with at least one day job, scattered throughout the globe and several timezones. Don’t bother–it’s impossible. You can create mailing lists and instant messaging channels, arrange conference calls and agree to meet at conferences. Without fail someone will be called to pull a triple-shift at work due to a flood in the server closet, another will end up in the hospital, another will have a family emergency to attend to, someone else will miss a plane. In short, life has a full bag of tricks to draw from and there appears to be some Universal Law preventing more than a few people at a time from concentrating on the same thing simultaneously.

However if you can resist the urge to repeatedly bang your head against your monitor while chanting “why me”, you’ll find that stuff still gets done. It sometimes takes an inordinate amount of time, but it still gets done. There appears to be another Universal Law at work here: the smaller the job or decision, the longer it takes to happen. The truly big stuff almost happens by itself.

#2 Ignorance is not bliss.

Admit it, it sometimes sucks to be into Open Source, especially when your favourite project just isn’t on other people’s radar. Even more so if you’d like to make some of your living from said Open Source project. Sometimes I wonder if I’m stuck in a Twilight Zone episode or I’ve been transported back to High School and find myself outside of the “in” crowd.

Admittedly, I have heard some good one-liners over the years. “If it’s so good, how come it’s free?” or “if it’s so good, how come I’ve never heard of it?” still jockey for top position. And this one is always a treat from someone who has just spent 2 hours listening to musak in order to talk to a representative who knew less about the product than he did: “but it offers paid support”.

#3 Where’s the beef?

If I’ve learned anything over the past few years it is that geeks make terrible marketers. Sad, yes, but true. Geeks thrive on technical details. However, in the world of marketing, as in the world of cocktail parties, exuding the technical details of a product will only win you glazed stares and bored “uhuhs” while the target scans for the nearest exit.

In the marketplace, sex sells. Glitz sells. And the occasional surprise, such as a pair of little old ladies peering into a hamburger bun, sells. Free doesn’t. The downside to selling Open Source is figuring out how to market something that could have been had for free. The upside is that Open Source is a huge, mostly untapped market. If you’re a marketer and know how to use sex, glitz, hamburger buns, or anything else to sell BSD Certification, drop me a line.

#4 Show me the money.

Did I mention that geeks are terrible marketers? I suspect they are even worse at raising money. I’ve also heard rumours that it is hard to get people to give cash when they are used to getting something for free. I hope this is not the case, as I need to raise a fair bit of money.
My experience with Open Source has been that people are very generous with their time and their skills. I’ll let you know how they are with their pocketbooks.

#5 When in Rome…

I sometimes wonder how much of this adage applies to Open Source. In my mind, Open Source can be used to redefine existing models. The example I’m pondering as the year draws to an end is this: IT certifications are currently delivered using proprietary solutions on the Microsoft platform. Not surprisingly, many of the delivery agencies I’ve spoken to don’t see the sense in spending money on creating an alternative solution based on Open Source. What did surprise me is the number of people in the Open Source community who don’t see the sense in spending money to create a new model for Open Source certifications.

Am I being too anal? My editor didn’t think so when I wanted to use vi instead of Microsoft Office when I wrote BSD Hacks. Or is certification one battle that isn’t quite ready for Open Source? I’d be interested in hearing your opinion on the subject.

[In the interest of getting this up on my blog, it isn’t going to be as heavily proofread as usual. Please excuse the typos.]

The Usenix LISA 2005 Conference is happening this week. This is the big Open Source system administration conference. I’m trying to summarize each day’s activities. Tuesday I taught a full-day tutorial called, “Help! Everyone hates our IT department!” and went to dinner with people that had interesting things to say about some new technologies.

Day:

Today I taught my full-day tutorial “Help! Everyone hates our IT department!”.

Under a different title, I’ve taught this workshop 3+ times before in both the US and Europe. The workshop goes over the basics of making sure that when a user arrives in your organization they have a “soft landing” and you make a good first impression. You only have one chance to make a good first impression. Then there are techniques for making sure they have a pleasant stay. How do we do that? Have a helpdesk, communicate effectively, and use monitoring tools so that you find problems before they are reported. Lastly the class talked about training strategies for IT staff. As you can guess, most of the class is oriented towards managers.

Good first impression: Discussed techniques for making sure someone’s first day is productive: that they have a PC, a phone, their accounts are created, etc. Subtopics included the politics of getting permission (funding) to have preloaded machines, get notification that a new person is coming (so you can prepare), and so on. Students brought up an interesting issue: they are in environments where new people “aren’t in the database” for days until after they are hired. Solutions? Bribe spies that are “in the know” about who is being hired in each department (for example, the department secretaries).

Having a good stay: About 2 hours were spent on advice for managing a helpdesk. Theory about why the 2-tier system works, a model for how a helpdesk can work, and the policies that (if they are written down) make everything flow a lot easier. For example, if you have a written “scope” document, then your staff is empowered to say “no” (politely, of course).

Visibility: The problem with system administration is that if you do a good job, people don’t see you, and they think you don’t do anything. Thus, to do a great job you have to create your own positive visibility. A number of techniques for advertising your goodness were presented.

Communication: How we communicate with users determines how they see us. Tips on mass-emails (keep it brief, put the important info in the Subject: line), “Town Hall” Meetings, status web pages, and so on.

Monitoring: Discussed the value of monitoring: real-time monitoring means that you get alerts when there is a problem. If you can fix an outage at midnight before people come into the office at 9am, then they don’t see an outage, right? Historical monitoring lets you prevent problems in the long-term. For example, if you collect info about network utilization, you can predict when you’ll need to upgrade your internet connection. Otherwise, you’ll wait until its overloaded, people complain, and then the upgrade will eventually happen.

Training: Strategic training gives people vision, tactical training gives you specific technical skills. Also discussed mentoring, books, and other educational forums.

Commentary: I have to say that having taught this material 4+ times I kept getting a feeling that what I was saying was “old hat”. I kept thinking to myself, “wow, I’m boring these people. Who doesn’t know this already?” However, at each break people kept coming up to me saying how much they were learning and pointing out specific examples of what they liked about the tutorial. That made me realize that no matter how long I teach this tutorial there will always be people with these kind of issues; these issues are timeless.

In all, I enjoyed teaching the tutorial.

Evening:

During the evening I had dinner with a bunch of people that had attended the Advanced Technology Workshop (ATW). This is an informal day of facilitated discussions about what’s new and hot in system administration. I didn’t learn a lot, but I had a good meal. I’ll get the notes emailed to me soon and/or will provide a link.

Night:

At night there are Birds-of-a-Feather (BoF) sessions where people of various communities get together. I wasn’t able to make any because (1) I went to the Usenix Open Board meeting (where nothing happened except people were very positive about the SAGE/LOPSA split, and I think both sides want to see the other have a good future; they’re going to be providing mostly-non-overlapping services which tells me they will be successful), (2) I really needed to spend time in the hottub. Primarily because my back was killing me and the hot water would do wonders. Secondarily because a little known fact about LISA is that some of the best industry gossip happens at the hot tub.

I got to sleep around 1am. It was a very full day.

Related link: http://lxer.com/module/newswire/view/49475/index.html

Korea’s anti-trust regulator Wednesday (Ed: Dec 7th) imposed a fine of roughly 33 billion won ($31.9 million) on Microsoft for its violation of the nation’s fair trade rule.

Microsoft will be required to offer two versions of Windows within six months - one stripped of thus far incorporated programs (Ed: Messenger and Media Player) and the other that includes most software available here.

The case focused on accusations that Microsoft violated fair trade rules by unlawfully incorporating its audio-visual program, Media Player, and instant messaging service, MSN Messenger, into its dominant Windows platform.

What does it mean for the rest of the world?

If Microsoft refuses to deliver the versions the government of Korea orders, then someone will need to provide the codecs that allow Media to be played on Linux. That could change the game considerably. Korea, China and Japan have created their own distribution of Linux. TurboLinux offers Cyberlinks PowerDVD player on its Japanese distribution.

Perhaps, Cyberlink will be forced to offer their provide to Linux users worldwide. We will have to see. I never count Microsoft out, even when the top offcial makes a ruling. They have a way of going over everybody’s head.

Today at the Opening Plenary of the USENIX LISA2005 conference, Christine and I were presented the 2005 Outstanding Achievement Award for our book, The Practice of System and Network Administration. I accepted the award on Christine’s behalf, as she is in Switzerland and couldn’t attend the conference. Here is the transcript of our acceptance speech

Related link: http://www.vigilar.com/event/netforensics2/

Here is a new webinar on security metrics that I will be doing next week. Why promote it here? Because security metrics is a very hot subject that a lot of folks are interested in. So, here comes the blurb:

“Your security operations team has the critical task of monitoring, measuring, and continuously improving key compliance and risk indicators. But what indicators should you be monitoring, how do you establish a baseline, and how will you know if your efforts are effective? Traditional IT performance measures, such as ROI, are difficult or impossible to use in an information security context. Using the security data that exists in your organization, you need to be able to the measure security specific performance using security benchmarks and measurement techniques.

In this webinar you will learn how to develop security operations metrics, and measure and continuously optimize operational performance by leveraging existing security information.”


See ya there…

The Usenix LISA 2005 Conference is happening this week. This is the big Open Source system administration conference. I’m trying to summarize each day’s activities. Monday I attended the day-long “Workshop on Configuration Management” (things like cfengine, etc.)

Day:

I’m a bit too tired to write a big summary. Suffice to say, it was mostly theoretical, with people talking about everything from doing basic updates, to ideas for expert systems that would be able to process requests like, “Add a web server to the least loaded server.”

However, I was sitting next to someone from ControlTier who showed me a demo of their software. It is a very powerful commercial CM system that has recently gone open source. The company sells consulting services. A number of companies use the product. I liked the user interface and how things were structured. If you are looking for a powerful CM system, you might want to investigate ControlTier. (Tell them Tom sent ya.)

I presented a few slides about the impediments to adopting CM that I’ve seen, and what alternatives people adopt when they don’t use CM. I felt this was useful information to people building CM systems. It sparked a lot of discussion. :-)

Night:

I’m about to go out with 12 people for steak. Lots of AFS and Pittsburgh folks. I’m sure it will be fun.

The Usenix LISA 2005 Conference is happening this week. This is the big Open Source system administration conference. I’m trying to summarize each day’s activities. Sunday I attended the day-long “Workshop on Managing System Administrators”

Day:

“Workshops” at LISA are day-long sessions that let like-minded people talk about a particular topic for an entire day. Some of them are mini-conferences onto themselves, with scheduled speakers, discussion, etc.

Cat Okita and I co-facilitated the 4th Workshop on Managing System Administrators. Unlike other workshops, this one is more like a support group. In fact, it’s confidential. That way we can all talk freely about problem employees and situations.

What I can say, however, is that this workshop had about 16 people, mostly managing 2 to 12 system administrators. Everyone attending was either an ex-system administrators or currently trying to figure out how to be both a manager and a system administrator simultaneously.

The workshop began with everyone taking 5 minutes to explain their situation and list their issues. The issues were accumulated on a noteboard. Issues included things like how to hire people, managing “difficult” employees, how to be technical and a manager at the same time, finding time for “vision” work, finding time to mentor, integrating new subteams into the group, career management (including “I’ve grown bored. What should I do?”).

The issues were grouped into categories. We then multivoted to determine which topics were most interesting to people. Then we literally spent the rest of the day tackling these issues, offering to share our experience to learn from each other.

To break up the day there are two breaks (one in the morning, one in the afternoon) where cookies and soda is provided. The lunch that was provided was a very good Mexican buffet.

At the end of the workshop we went around the room and had each person
name one thing they learned today that they’re going to do when they return.
Without breaking confidentiality, here are some of the things that were said:

• Make a new attempt at working with [name of other division] now that I understand their culture a bit better
• Arrive an hour before my officemates so I can work without interruption
• Start MBWA (Manage By Walking Around)
• Use a flat Wiki structure
• Use the suggestions about how to get senior people to document procedures

Night:

At night about 30 of us went out to dinner at an all-you-can-eat Asian buffet. I was able to network with people that work for Google, which I will be an employee of on Jan 16th, so it was really useful to talk with them ahead of starting there.

Today I realize that I’ve seen three very different ways to allocate people to projects. While each is very different, each project was profoundly helped by the use of the particular method. I thought I’d record them here so I can remember them better.

I can’t take credit for these ideas. While I’ve anonymized these anecdotes, I can assure you that they are examples of people much smart than me.

Anecdote 1. Take a project, give everyone a different piece, and try to get it all done.

This is pretty traditional. It makes sense in the modern division-of-labor society we seem to live in. Everyone worked in parallel and got their piece done.

Anecdote 2. Take a project, break it into milestones, and have everyone work on a milestone as a group until it is done. Then move to the next milestone.

Once I was involved in a major software project. The next release was going to have a lot of new features. After some analysis, someone realized a better way to do the development would be to break the features into three groups based on the part of the system they affected. They would do three releases, one for each group of features. Marketing could just ship the last release or ship intervening releases if it made them happy. The important part is that the entire team was focused on getting each milestone done correctly rather than being scattered about. Everyone could help everyone because they were all focused on the same part of the system. QA could test each milestone better because they knew to do the heavy testing on the part of the system being modified. QA also liked this plan because they could test milestone 1 while milestone 2 was being worked on. Ah parallelism. Also, instead of having to do a lot of testing at the end, they would be doing heavy testing of group 3, and simple retesting of the previously tested parts. This reduced the chance that they would be blamed for delaying the ship date. It opened opportunities for more parallelism in other areas too.

Management bought into the idea and the project was very successful. Interestingly, however, if upper management hadn’t bought into the plan it wouldn’t have mattered because the developers could have done it under the radar… the only difference is that the customers would only have seen one release.

Anecdote 3. Take your best people and put them on the most difficult problem. When they are done, move them to the next biggest problem. Keep moving them until your biggest problems are quite minor.

A person was hired to manage an IT team that spanned 3 areas. We’ll call them Area A, B and C. The systems and networks in each area were a disaster. Users weren’t getting their requests done in a timely manner and outages were frequent. After a month he saw quite clearly that one area was more messed up than all the others. While his predecessor had her office in Area C, it was Area A that needed the most work. Thus, he broke with tradition and moved his office to be in Area A’s building. He moved two of his top team members to this building (one was already there, struggling with the chaos) and focused on fixing the problems there. After a while things were stabilized and much improved. Then he moved his office to Area B and took those two top team members with him. They repeated their success in Area B. Finally they were able to move to Area C. By focusing concentrated effort to the places where the biggest impact could be made (rather than doing the easiest segment first), he was able to create the biggest positive change quickly. By bringing the best people on his team to the task, he reduced the risk of failure. By using the same people for all three segments, he was able to have a team that learned and improved as they moved on, getting better each time rather than repeating the same mistakes over and over.

When it comes to allocating people for projects I’m sure there are many other ways to do it. I’d be interested in hearing your thoughts.

Comments? Suggestions? Recipes?

Related link: http://chuvakin.blogspot.com

I just launched another blog at http://chuvakin.blogspot.com.
I will keep the O’Reilly blog for information security related issues and will use the new one for personal comments on … well, the rest of the stuff.

BTW, after I used the Blogger interface, I feel like I am in the freaking Stone Age when typing this post :-)

Related link: http://www.newsfactor.com/news/Podcaster-Falls-Prey-to-RSS-Hijacking/story.xhtml…

Erik Marcus’ podcast of Vegan.com was hijacked. External hits to his mp3 files went from thousands of hits to almost none. It is believed that once a tidy sum of extortion money is paid to a podcast directory site, the problem will be fixed, and Erik’s traffic will return.

From reading the article it’s not 100% clear if the false feed was registered on podcast search directories, or if the RSS URL was always the same and suddenly redirected to a different location. Either way, doing searches for the podcast through different podcast directories may yield incorrect results. If you go staight to the Vegan.com site, you should get the proper RSS file.

While the web is still unorganized and unstructured, podcasts are very organized. That’s because there is a small number of initial vectors to find podcasts. Most users are going to go through iTunes, PodShow, Odeo, or whatever directory is configured through their podcasting software.

Unfortunately, that puts the podcast directories in a unique power situation. They have dedicated traffic from the clients and full control of what content is sent back to the user. Once podcasting grows, and more podcast directories become available, the ability to hijack podcast traffic could decrease.

I spent a few minutes trying to think about solutions to get around this kind of problem. Signed RSS files? Cross indexed podcast directories? I started to get hungry, and my throughts drifted towards pancakes and bacon cooked on my double-burner cast iron griddle. I realised that this isn’t a problem with technology, this is a problem with people and policies.

If you run a podcast directory, you need to make an honest effort to make sure it has reliable information. Don’t just accept anonymous submissions to feeds without checking the URLs submitted. If a little bit of due dilligence was made, problems like this could be avoided. Pointing to incorrect feeds harms your own reputation as a reliable source of information.

If you write podcast playing software, don’t rely on one source of information. Make it easier for users to import RSS feeds from multiple directories, or from the main site associated with a podcast.

And if you run a podcast, keep the feed URLs under your control. Be more protective of your property, and don’t rely on a limited number of sites as your sole source of traffic.

Friday is my birthday. Instead of asking for a gift, I’m giving one. For the last month I’ve been working to produce the following 45-minute video that highlights many of the techniques in the new O’Reilly book “Time Management for System Administrators”.

http://tinyurl.com/cklq6

or

this very long link

I was able to find the perfect royalty-free music (podcasters take note!) at UniqueTracks. I’d like to thank the good folks at Pixient for their help with video production and editing. The video was recorded at the November meeting of the $GROUPNAME users group, on Rutgers University campus in New Jersey.

Note: The link is alternating between working and not working. Anyone at Google able to give me a permalink?

Post a comment. You know you want to. All the cool kids are doing it.