Splunk, LTSP, and others all contribute to a healthy evolution.

LinuxWorld Boston seemed relatively small this year. I was glad to see fewer flashing lights and to hear fewer barkers’ exhortations for crowds to shout out (and when the crowds did shout out, there were fewer of them doing it), but I missed some of the energy of previous years.

Nevertheless, the feel I got on the show floor, and the occasional sessions I got to, suggests that there’s good news in the GNU/Linux world.

Good news

What’s happening in Linux is a new stage of development made possible by a maturation in the stance of large customer sites. Nobody has to petition the largest companies to take a look at Linux; the commitment is there.

What this means is that I hear free software developers and their potential major users holding frank and productive discussions. The terms and concepts used by the two sides are still hard to bridge. (how, for instance, can you draw a line between a change in the caching policy used for network buffers in the kernel and the throughput desired by enterprises backing up their data every day?) But they seem to be trying hard to work together.

A lot of government agencies around the world are similarly committed to free software. For instance, sixty people attended the forum on government at LinuxWorld, organized by Leon Shiman of the X.org Foundation.

So the big news today, I think, is the shared sense of purpose and the conviction that freedom is making progress. The focus is not on the relatively minor infighting that still goes on. Different vendors jockey for market share, but they collaborate around technical fundamentals such as improving the desktop experience.

The experience of Fabric7 Systems is relevant here. (I planned on saving individual companies for later, but what Fabric7 told me just fits in here too well to postpone.) XuFabric7 aims right at the high end, offering Linux or Windows on I/O and processor configurations that other vendors reserve for proprietary Unix operating systems.

Fabric7 is convinced that the commodity Intel and AMD chips can pay off on large configurations, and at LinuxWorld they were pushing Linux as a solution for this space. They also believe that Linux is going to become dominant because more and more students will come out of school familiar with Linux and antsy to load it on anything they see that sports a push-button and an LED. Fabric7 president Sharad Mehrotra claims they’ve sped up I/O on their system by offering a “virtual I/O system” in the switch.

In the days of the 2.4 Linux kernel, Fabric7 had to apply a lot of patches to get a kernel they were happy with on their high-end hardware. Now, with 2.6, they need no patches. That’s a data point in the evolution of Linux to meet large enterprise needs.

Yes, there are still uncomfortable issues we have to address. Large sections of the economy still exist, unfortunately, where managers don’t know what free software is or are afraid of it. The beneficial effects I talked about earlier mean we have a critical mass of enterprises on board, but too many are still missing out.

Technically, I still feel nervous with every package system I’ve dealt with, because if one thing goes wrong I am left with a corrupted database and have to fall back on installing software by hand. Given the frequency that I want to update my system with new software or revisions, this is a real hindrance to my use of Linux.

And we’re still arguing over which license to use. Luckily, a lot of work is being done on compatibility among licenses. But along with the usual complaints about the General Public License, many people criticize the GPL version 3 because it explicitly takes on battles with Digital Rights Management and patents. While I have my doubts whether a software license should embody specific political choices, I stopped by the Free Software Foundation booth to renew my membership. They’re responsible for creating and maintaining a huge range of indispensable software, and I want their viewpoint heard.

Before I delve into details about the show, I should note that LinuxWorld deserves praise, despite this year’s small turn-out, because the show continues to evolve and try new things. Its Mobile Linux pavilion, along with the Mobile and Embedded track, successfully pulled off an attempt to bring into the show this critical area of new development. I saw a lot of people participate in the new introductory sessions, offered under the name OpenSolutions World; the room occupied by TUX Magazine, the publication about desktop use, was well populated. Opening keynoter Nicholas Negroponte filled his hall and, with his rousing manifesto for “One Laptop Per Child,” proved that the show could look higher than who offers the fastest cluster benchmark or lists the most check-off items on Exchange compatibility.

Self-healing communities: Splunk solves problems collectively

My favorite new initiative at LinuxWorld was the snazzy peer-to-peer problem-solving system created by Splunk. It’s just the kind of creative use of distributed input from the community that I go for. Whether it proves successful in meeting system administrators’ needs is a question we’ll have to wait a while to answer.

Splunk Base is, to put it crudely, an enormous collection of log file entries gathered from different systems. System administrators use the database by running Splunk’s (proprietary) software on their systems to collect all kinds of information, as the system churns away doing its everyday Web serving and mail delivery and so forth.

You can contribute to the collective database by submitting a snapshot of some anomaly on your system. Splunk strips off information such as IP addresses that may reveal information you don’t revealed, and thereby produces a generalized description of the problem that can potentially match other people’s problems. Lots of information can be stored about each incident: log file information, protocol headers, and comments from you.

You can query the database by searching on a wide range of criteria, such as software packages and error codes. There’s not much you can discover from a single entry, but cleverly, the database’s search functions tie together related entries by virtue of their shared characteristics. For instance, a spam filter logs information independently from a mail transfer agent, but because they both record the unique Message-id on a message, you can find the spam filter entry that worked on the same message as the MTA.

Splunk provides both an open database that anybody can join, and a commercial service. Another way that a community can build around each database is for users to create wikis around particular incidents and submit information discovered while researching the problem; they can even creating new wikis around related or underlying topics.

As I said, it’s too early to tell whether Splunk’s metadata, categorization, and search are powerful enough to solve system administration problems better than a vanilla search on Google or some other engine. But I just think Splunk is the neatest combination I’ve seen of metadata, search technology, leveraging community, and cutting-edge Ajax Web interfaces.

Plug-and-play over a network

The next thing that got me all fired up was to drop by the Linux Terminal Server Project booth in the .org Pavilion and watch the designers plug a USB memory stick into a system; the contents of the stick showed up within a second and a half on the screen. What’s remarkable about this is that the desktop software was not running on the system they plugged the memory stick into. This being an LTSP environment, the desktop software was running on the server over an Ethernet cable.

What LTSP has done–with little fanfare–is to take plug-and-play to a new level by allowing it to work over a local area network. To get this operational, developer Jim McQuillan had to create a distributed version of D-BUS (he calls it lbus) so that a hotplug event can be reported by the local Linux system to the Linux server. He also had to create a new distributed file system (which he calls ltspfs) to send over the filesystem information; he found both Samba and NFS inadequate for the task.

I am just waiting for the world to move massively to LTSP, the ideal solution for any environment trying to provide computing for a set of local users with limited technical background. The hard thing about Linux and free software is getting all the software packages installed and working together. LTSP lets you consolidate this problem on a single server, and free everybody but the central administrator from doing system administration.

More big news

I don’t actually have that much news to say about big systems, but there’s a sense that the space is wide open for somebody to offer a very robust and scalable solution. The buzz about clusters, which has gone on for years, has been joined in the past year by buzz about virtualization.

VMWare has opened several of their components, including most recently their virtual machine disk format, which they hope will become a standard. The virtual machine disk format stores all the data about a virtual operating system, so that the underlying host can handle all its needs such as sharing CPU time with other systems, doling out memory, and so on.

As a proprietary firm, VMWare is certainly being chased by free software projects such as Xen, not to mention proprietary competitors. But I see in VMWare a company that continues to race against its competitors, to think creatively, and to work closely with the Linux community so it can be accepted by them both commercially and as collaborators. They boast one million downloads of their free player. And they now have a free server as well, which is fully functional in terms of virtualization, but lacks the features they figures major sites need such as migration and failover.

As a new layer on a computer system, virtualization increases possibilities without restricting people’s use of older technologies. I visited a company I’ve talked to regularly, BakBone Software, and asked whether virtualization forces them to change their systems for storage, backup, and replication. I was told they work fine with virtual systems and don’t need to alter their product; well-layered systems are transparent ones.

Red Hat, according to executive vice president of engineering Paul Cormier, is also working hard on virtualization. Three years ago I noted that Red Hat was “moving up the stack,” which translated into more support for Java- and Web-related projects. They are still doing that, but moving down the stack as well. They are providing a Fedora-based operating system for Negroponte’s One Laptop Per Child program. And I noticed that their booth drew quite decent-sized crowds despite the low attendance at the show overall.

Why put Linux on an embedded or mobile device?

I talked to a couple companies in the embedded space; I wanted to see what Linux had to offer besides a way to bypass license fees. (Of course, plenty of Linux vendors charge license fees, and some of their systems are licensed by embedded developers, but the embedded developers are creating systems on a scale where they often find it worthwhile going through the pain of building a license-free Linux.)

Mark VandenBrink, a lead architect at Motorola, gave me a report that confirmed their is more to Linux than a cheap ride. Motorola began considering Linux as a mobile operating system in 2001. The real cost savings of Linux, so far as mobile phones go, don’t lie in the area of licenses. The savings lie in finding large swathes of necessary software–drivers, applications, and a rich community of developers for the kernel and other software–already in place. Motorola would regularly go out seeking some piece of software and find it already available as free software.

Another advantage of Linux is that Motorola can assemble a working system quickly at any time, rather than waiting for a commercial vendor to announce a release. In the fast-moving telephone market, where competitors are always trying to one-up each other and consumers clamor for the latest and greatest, such flexibility is a big advantage.

Motorola does its interfaces and applications in Java, because it’s a proven technology that they’ve used since 1999. I discussed a very different interface, Qtopia, with Eirik Chambe-Eng, who is president of the company that develops it, Trolltech.

UPDATE 14 April 2006: Chambe-Eng tells me Motorola uses Qtopia as wel as Java. He writes,”Their interfaces and most of their applications for their Linux-based phones are made with Qtopia Core from Trolltech. The A1200 and ROKR e2 are just 2 of many examples.”

Qtopia is a C++ framework based on the same Qt libraries that underlie KDE. While Qt on Linux is based on the X Window System, Qtopia has its own engine going right down to the hardware. Qtopia’s evolving display system can take advantage of a lot of accelerated features in advanced graphics hardware.

And while Trolltech made the Qt libraries available under an open source license several years ago, Qtopia remains proprietary, but its source code is available for viewing.

Chambe-Eng suggested that one new element of Qtopia, its Secure Execution Environment, could make Linux more appealing to mobile phone developers. Many mobile phone services are very strict about letting applications be downloaded to their phones, for both security and regulatory reasons. Traditionally, the phones have enforced restrictions by providing binary-only interfaces. Qtopia’s SXE offers an alternative form of security that may allow Linux to give assurances that it can meet regulatory requirements.

SourceForge Community Choice Awards

At LinuxWorld, SourceForge unveiled the carefully kept-secret results of their Community Choice Awards, which nominated projects for 13 categories (plus an all-around winner) and let SourceForge visitors vote. All the winners were no doubt highly deserving, but several of us detected a pattern: winners were always projects with names easily recognizable to the free software community. This is to be expected when votes are solicited from the general public. I wonder what we’d find if we could focus the voting on the narrower community of users who depend intimately on a particular type of software for their day-to-day work, and may be able to judge each project on its details.

It’s also unlikely that such a process can reflect the projects’ relative future importance: that is, which project will turn out to be the shining white knight as technology evolves and new opportunities open up.

But none of this is meant to be a heavy critique: I think the awards were a really nice thing to do and a well-placed boost to the world of free software: a boost that calls attention to the wide range of problems that can be solved with free software, and celebrates the amazing creativity of free software producers.

Most interesting from a social standpoint is the winner in the final, “Overall Most Popular” category: Azureus-BitTorrent Client. We can see here the growing perceptual split between doers on the ground and those in control of public opinion. The former see a file transfer program as an eminently useful tool, while the latter sometimes seem to see it as nothing less than a attack on the whole Post-Industrial Information Revolution.