This blog will talk about two organizations I met with and their members–the Linux Mobile Foundation and the Open Solutions Alliance–plus other interesting people I met at LinuxWorld Expo.
General LinuxWorld comments
I should make it clear that this weblog focuses on issues that are of interest to me personally, and hopefully to readers of the O’Reilly Network, which might not be the issues dominating the minds of the LinuxWorld organizers and vendors.The show itself, while offering a range of topics in tutorials and sessions, focused on large data center issues such as storage and virtualization. It was held in conjunction with another large-scale trade show, Next Generation Data Center, and the joint theme could be summarized by a central statement made by the first keynoter: “Data centers are cool and exciting.” As one might expect, the bottom-line message in this keynote (by Werner Vogels, VP and CTO of Amazon.com) is that after you’ve considered all the complexities of running a data center (overprovisioning, disaster and downtime prevention, load balancing, etc.) you’d do better to rent space from Amazon. And it’s true that S3 has gotten fine reviews. I mentioned it in a blog several months ago, comparing the model to CleverSafe (which was also at LinuxWorld). But on to the show.
Mobile Movement
I’m starting with the Linux Mobile Foundation because I figure it’s truly significant for major vendors such as Motorola, NEC, DoCoMo, Samsung, Vodafone, and McAfee to throw their fate in common on a jointly developed reference platform. A consortium of handset manufacturers and service providers, they’re also looking for chip manufacturers (which could integrate the platform at the chip level) and application developers to join. Linux Devices indicates that the platform is not open source, but I’ll be curious to see how dear the vendors treat the restrictions if they discover benefits from sharing.The LiMO representatives I met with stressed that development of open source applications by a free software community was important to the success of the platform, and thus of the products that LiMo members were putting their money into. This suggests to me that pressure from programmers to loosen the terms of the license can have a strong effect. Motorola has released one SDK, and Japanese manufacturers NEC and Panasonic have released another.
The reference platform is created from working software components used by various members in actual products, so it starts out at a high quality level, although the foundation will take a year to integrate and test it, and those deploying it obviously have to add their own testing.
The LiMO platform is currently aimed at mid-range phones and high-end Smartphones. Phase 2 is hoped to encompass consumer devices; in effect, someone will get used to using a single device both to control consumer equipment in the home and to access voice and Internet content on the road.
The range of components is pretty comprehensive. For instance, protocols include cellular voice access, WiFi, Bluetooth, and GPS. When moving in and out of a WiFi region, the phones should be able to switch seamlessly between Wifi and cellular.
The user interface is left up to each service provider, but there is a common user interface framework based on GTK+. LiMo representatives said that GTK+ was appealing not only because it was open source but because of its “rich technology.”
I also talked to representatives from Motorola, who pointed out the continued growth of Linux-based mobile devices. They shipped 9 million 0f them since their first offering in China in 2003. Their Linux-based MOTOMING handset became the best-selling SmartPhone in China, capturing 1% share of the entire handset market in that country alone.
At the show, Motorola announced their next generation mobile Linux platform, MOTOMAGX, which is designed to support three development environments: Java (the most popular), native Linux, and WebUI (based on Web Core). They are offering potential developers a series of MOTODEV summits in the U.S., England, and China on the new platform. Current phones utilizing Motorola’s MOTOMAGX platform (such as MOTORAZR2 V8 and MOTOROKR Z6) contain some proprietary software, but Motorola intends to move to the open architecture defined by LiMo for future MOTOMAGX releases.
The LiMo announcement has no immediate effect on at least three vendors in the mobile Linux space. I talked to a representative from Trolltech who cited the features of Qtopia (their proprietary toolkit for mobile phones, based on Qt). Most recently, Qtopia integration with the WebKit browser engine allows any application to draw on the same advanced features as the Konqeror and Safari browsers. Flash animation will also be integrated into Qt over the next three to six months. Increased modularization in Qtopia 4.3 allows platform developers to choose the features they want (the JVM, messaging, etc.) and swap in components from other vendors.
Representatives at the booths of both ACCESS and Unicon Systems (who are both promoting developer kits) also said they were confident in the value of their platforms and were not planning to change them in response to LiMo.
Open Solutions Alliance and Application Vendors
Open source projects, ironically, are often harder to integrate than proprietary ones. Dominic Sartorio of SpikeSource believes this is because many large proprietary companies, such as Microsoft and Oracle, have whole suites of applications under their ownership, and their managers can dictate how they interoperate. Meanwhile, typical open source application communities focus on solving the needs served by their projects, while not fully considering interoperability issues.Sartorio is president of the Open Solutions Alliance, formed six months ago from seven application vendors in the open source space. Their goal is to make all their products easy to integrate, an effort that can have a wider effect on free software development as well. They have provided a reference implementation combining their products, and hope to provide a good example to other open source projects.
OSA uses the members’ projects internally whenever possible. Thus, they manage the company using Team Elements from Centric CRM. Single sign-on is implemented through SpikeSource’s product. This provides one critical form of integration–sharing the user account among all components–and is achieved through a language-independent interface. Many new languages were added to this single sign-on interface through a hackathon at the O’Reilly Open Source convention. Another critical form of integration is sharing data, accomplished through a product created by another OSA member–Talend–using standard JDBC and web services.
Centric itself has 11,000 registered members, who help with major efforts such as ports to new databases, as well as routine bugs, translations, etc. As with many vendors of open source products, Centric tends to hire anyone who they find contributes a lot of good software.
Another cofounder of OSA is Openbravo, a comprehensive ERP platform. This was the #2 download on SourceForge when I met with Josep Mitja, its COO and a board member of OSA, at LinuxWorld. Enjoying 250,000 downloads by an estimated 50,000 people, Openbravo is already looking beyond direct support (which is its income source) and has built a set of service integrators to serve its customers. Training has therefore become a major company activity.
Openbravo offers web-based interface and runs over Tomcat with either an Oracle or a PostgreSQL database. It is thus a cross-platform solution. But 30% of questions on the forum from people who download Openbravo for free concern installation. For corporate customers paying for support, therefore, Openbravo uses rPath’s appliance-building tools to provide a turn-key product. The rPath building system creates a software image that can be loaded onto bare metal, into virtual machine such as VMware, and so on.
Integrating a company’s data into the Openbravo system, along with writing functions in Java to process activities for which defaults don’t exist, takes an estimated 400 hours. They aim for the SME market.
Building a user community is critical to the company’s success. All source code is open; there is no difference between the commercial and free versions. Participation on the wiki is strong. Mitja says, “Everything we do has to support either dissemination or monetization. Dissemination is the first requirement for success in open source: if it doesn’t get out to the users, it can’t thrive. So if an activity supports dissemination, we don’t force it to support money-making as well.”
I stopped by the booth of SugarCRM, one of the earliest open source companies in the CRM space, and also one of the first to prove that open source could thrive at the application layer. Chris Harrick, Director of Product Marketing, says that the company overcame observers’ initial bias that open source was only for infrastructure, and their skepticism about its ability to produce successful commercial applications.
Like the other application companies, he considers building a community around the software a crucial goal. “Our developers live on the forums,” he says. The user base has responded with bug fixes, extensions, and contributions to SugarCRM’s developer web site, SugarForge.org.
Linux Professional Institute: A Global Force
I spent a good deal of time with staff and board members of the Linux Professional Institute (some of whom helped write O’Reilly’s LPI Linux Certification in a Nutshell) and heard some interesting assessments of Linux’s spread as well as certification.The certification market was oversold in the mid-90s and experienced a dip, but is now rebounding. Globalization accentuates the importance of certification. For instance, lots of Chinese companies would like to prove that they offer competent IT staff, and need to show this through a global standard. Many endorsements of LPI show that’s recognized as just such a global standard. This Toronto-based, vendor-independent, non-profit organization has put tremendous effort into developing international affiliates and translating its exams into many languages.
A Latin American affiliate reported progress there. By now, even Microsoft supports LPI in order to promote interoperability, because the exams test for interoperability in areas such as Samba and LDAP. But the affiliate manager said that what he called the “war against Microsoft” has held up adoption of Linux and open source in Latin America. Companies intrinsically know that their adoption of open source will have to be gradual and that it will have to co-exist with Microsoft products, so they worry about the free software community’s ability to enable that co-existence. A lack of applications has also held up adoption, and–the area LPI can help with–the lack of qualified professionals.
Closer to home, U.S. veterans can now use their educational benefits to pay for taking the LPI.
Over 46,000 people are certified on various LPI levels. This has not been achieved by handing out certifications willy-nilly, because over 150,000 exams have been taken. LPI has developed a special Ubuntu exam, and have been asked by some other distributions to do the same for them.
Notes from an interview with Ian Murdock
I was lucky to get an interview with Murdock, where I could talk about both his opinions on Linux distributions and his current work for Sun Microsystems. Ian describes himself as “always loving Sun.” As a student in the early 90s, he “wanted a Sun workstation, like everybody did.” A Sun staffer said they hired Ian because Sun had contributed a lot to free software but were inadequately recognized for that, and needed to build relations with the community.The biggest community effort now, with the most potential payoff, is Project Indiana, which makes it easier to obtain a binary of OpenSolaris. When Solaris was opened, its source was put up publicly, but Sun still built binaries internally for its customers. It was hard for anybody outside Sun except technically sophisticated developers to build binaries. Thus, Sun is reorganizing the architecture of Solaris to make builds easier and working with Solaris community members to smooth out the binary build process. Getting OpenSolaris will then be as easy as loading the popular Linux distributions that offer binaries.
Comparing the distribution of Linux and Solaris, Ian pointed out that Linux vendors used to differentiate themselves partly on the basis of the kernel (making it hard for ISVs to support the different distributions) but that they’re much more similar now at the kernel level. The variety of distributions is great for choice, but the lack of compatibility that remains in file locations and other matters still complicates application development and system administration. Sun, through Project Indiana, hopes to achieve the same flexibility without having the fragmentation. The differentiation between distributions will presumably lie in the higher-level packages offered, not the kernel.
In particular, offering an ABI for device drivers makes it easy for outsiders to build Solaris drivers. The device companies don’t have to offer the source, and the ABI is stable so that drivers should continue to work through many future kernel upgrades. The Linux community has politicized the issue of binary drivers and strongly discourages them. Ian thinks the issue will ultimately be decided by the marketplace. If open source is fundamentally an advantage, companies that offer the source to their drivers will do better and will win out.
The availability of device drivers for Solaris has improved a lot since OpenSolaris was released, although it hasn’t caught up to the number of devices supported by Linux.
So according to Murdock, “Solaris is not trying to be Linux.” It is trying to build community and offer diversity in Sun’s own way. Sun’s advantage in the marketplace is its presence at all levels up the stack from the SPARC chip to middleware and applications.
Backups Behind Your Back
I stopped by Righteous Software (R1Soft) and talked to CEO David Wartell, because I was intrigued by a uniquely flexible backup product they offer. RiSoft can run incremental backups once every fifteen minutes (or at another interval configured by the administrator) without interfering substantially with system operation.The key to the system is a device driver that sits below the Virtual File System (VFS) and above the block device (SCSI, ATA, LVM, etc.). The device driver keeps track of which blocks were written, and the filesystem metadata. After the first backup, the driver marks which blocks are dirty and backs up only the relevant data and metadata. The driver therefore consists of two parts: one recording the data written, and the other communicating with network software to perform the remote backup.
This is more efficient than most backup solutions, which require a userspace component and therefore have to go through the kernel on the way up from the filesystem to userspace and down from userspace to the network device. The operation is comparable to change tracking, but ends up with a remote copy instead of one that shares the local media. Data can be restored down to the granularity of a single file.
Snapshots are supported natively by NTFS’s Volume Shadow Copy feature, and by LVM. For other filesystems supported by R1Soft–the popular EXT2, EXT3, and ReiserFS–R1Soft adds shapshotting. Each snapshot requires a very brief lock of the filesystem. Other than that, device driver overhead is negligible. It requires about 3 Mb memory for each 100 Gb of hard drive.
The driver is binary-only, but can be rebuilt on the R1Soft web site so that it can work with localized versions of the kernel. The administrator doesn’t have to know the details of how the kernel was built for his or her system, but just has to enter the pathname to the kernel at the R1Soft web site, and then download the driver as a loadable module. R1Soft will track open source developments and is willing to consider open-sourcing the driver in the future.
R1Soft offers a similar product for MySQL backups. Instead of filesystem metadata, database metadata is stored. Thus, data from MySQL can be restored down to the granularity of a single table for MyISAM and InnoDB storage engines. Wartell also says that R1Soft’s backup system is the only way (other than LVM snapshots) to take a hot backup of MyISAM without locking tables, and has several advantages over Innobase’s commercial utility for taking hot backups of InnoDB.
What’s the Unbreakable in Unbreakable Linux?
I asked Monica Kumar, Senior Director for Linux and Open Source Product Marketing at Oracle, what the company meant by that term. She answered, “Unbreakable Linux is not a distribution,” a point that I think is lost in many criticisms of Oracle’s marketing. Instead, she writes, “Unbreakable Linux is all about seamless deployment of Linux and enterprise-class supportability of Linux.” In short, it’s a promise of support rather than a piece of software: “our commitment and laser-sharp focus in helping end users deploy Linux for mission critical applications.”Oracle has made significant contributions to Linux, including asynchronous I/O (AIO) and powerful distributed filesystems, their newest announcement being called Btrfs. Asked to describe the demands made by large files on databases and file systems, as well as the contributions made by Oracle’s filesystems, Kumar listed:
- Efficient snapshots
These will make it easier to perform live backups and other administrative tasks.
- Efficient incremental backups
Btrfs can quickly find the metadata and data that has changed since a given revision, enabling very fast incremental backups.
- Online filesystem check
An error in one part of the filesystem will not prevent the database from coming online and doing its own internal datafile verification.
- Metadata mirroring
Crucial filesystem data structures will be mirrored, and if one copy of the metadata fails a checksum, Btrfs will be able to use the mirror instead. This should reduce filesystem downtime in general.
- Extent based storage
A number of Linux filesystems use extent based storage, because it requires much less metadata and CPU time to address large files.
- Device mapper integration
Btrfs will be able to work together with the device mapper layer to optimize I/O on top of large storage arrays.
Finally, I asked whether Oracle’s acquisition strategy (SleepyCat, Innobase, and an offer for MySQL) as well as the free-for-download Oracle Database XE, showed they were leaning toward releasing the database as free software and concentrating on revenue from higher-level tools and applications. She rejected this suggestion, pointing to Gartner’s report that Oracle on the Linux platform grew 72% in one year, much faster than overall RDBMS market growth (14.2%) and faster even than general RDBMS growth on Linux (67%). Out of 2.35 billion dollars spent on RDBMS on Linux, Oracle gets 1.95 billion.
The virtualization space: an interview with VMware
Operating systems are interfaces between applications and hardware, and therefore have two big jobs to do:Scheduling and resource management for applications running on top
Managing the drivers, memory, and other physical aspects of the system underneath
The Linux kernel, from version 2.6.20 on, now has a two-layer virtualization scheme. VMware helped to develop this, along with Xen developers, IBM, Red Hat, and others. The guest operating system rests on a layer called Paravirt-ops, which provides a common source-level interface and in turn directs requests to hypervisor-specific backends. The VMware backend for Paravirt-ops is called VMI, which makes calls into the VMware hypervisor. No significant performance hit has been detected due to this two-layer scheme.
The kernel developers are also taking more account of virtualization in designing kernel features, particularly now that Linux includes its own virtualization layer (KVM). For instance, they are allowing the operating system to generate fewer timer interrupts, and to take the time of the next timer interrupt into account when determining when to wake up a system that is being put to sleep.
One of the interesting new VMware feature is record/replay, which manages to take a trace of all important instructions and store it in a fairly small space. This has proven useful for debugging hard-to-track bugs (such as application bugs due to race conditions) and many other interesting ideas are circulating about how to use it.
Brief Notes
The world of MVC web frameworks got a serious new contender: Zend Framework, released under a BSD license. Although other frameworks for PHP have been developed, Zend Technologies expects this one to consolidate community efforts and give people confidence the framework will continue to grow and perform well. Feedback from the early adopters, some of them very large companies running-high traffic sites, indicates that Zend Framework performs and scales very well, much better than expected for a 1.0 release.
Next month, the Eclipse Foundation will release PDT, an open-source PHP interface to Eclipse. Zend believes so strongly in the superiority of PDT that it is making it the underpinning of a new commercial, Eclipse-based version of its own Zend Studio IDE.
Perhaps in a couple years, just as every word processing file contains author metadata and every desktop application is associated with properties, we’ll get used to seeing licensing information attached by a computer system to each document. If so, this will probably come about through the efforts of a number of talented young students who presented tools they had developed at a Creative Commons salon I attended. Authors or uploaders can tag documents, graphical interfaces can attach icons, and embedded links can take you directly to a licensing site, just as O’Reilly’s Safari book service offers access to standard copyrighted material through the Copyright Clearance Center’s Rightslink. However, no hint of monetization came through at the salon. The enthusiastic experimentation there seemed to bear out the article I wrote two years ago,
- The Commons Doesn’t Have a Business Plan.
Open source security tool vendor Untangle announced the results of testing several open and proprietary anti virus products. As just a sample of the results, Norton came out substantially better than McAfee, and as for Watchguard–well, go look at the web page I just cited. Untangle engineers told me they’ve run these tests annually for three years, and get consistent results each time. Untangle, of course, is built entirely on open source software.
Does leadership style determine the social interactions and psychological atmosphere of a free software project? It would not be far-fetched to say so, and many project founders have been praised or blamed for drawing in contributors or driving them away. But one member of a free software project suggested to me that everything down to the interactions on technical mailing lists can be traced back to the personalities of a few key leaders. Readers are left to search for evidence to confirm or repudiate the hypothesis–but one can then go on to ask whether the leaders who are inspired to start projects are shaped by the environments in which they begin.
Black Duck has announced support for the GPL version 3 license in its software compliance offerings, while Palamida is tracking project conversions to version 3.
Thanks so much for the mention in your LinuxWorld overview. We had a lot of interesting traffic to our booth, much of which was driven by the announcement of our successful scan of the OSA's authentication mechanism in their interoperability prototype. As Palamida is the only vendor of its type to provide code base scans that help mitigate IP issues as well as security vulnerabilities, it was important to the OSA to deliver a fully vetted security component for the show and we were there to ensure that it happened. As early members of the OSA, Palamida has been working closely with the open source community to continue education at the C-level on the challenges related to open source code in the enterprise. We're all for it, we just want to be sure you're not caught unawares at go to market time! Our GPL3 site continues to get outstanding page visits and developer contributions. It's a testament to just how closely people are watching the GPL3 conversion controversy. It's an exciting time to be a part of the OSS community!
Melisa Bleasdale, Palamida