Did you ever wonder why one of the most-used pieces of open source software was named Apache? Aside from the obvious nod to the Native American tribe, it is also an acronym for "A PAtCHy server" as it was based on some existing code and a series of "patch files." (Editor's Note: The link to apache.org was inadvertently left off when this article was originally published). This patchiness is an important characteristic of open source software.
Open source software is built to be resilient and to evolve organically. The first release of a new open source component may not cover all the failure scenarios, but its source code transparency and the wide use of these components help to improve the quality.
Traditional software vendors have adopted various testing methodologies and techniques that are broadly categorized as white-box, black-box, and in some cases gray-box testing. Typical tests include unit, regression, load, integration, boundary, negative, functional, standards, and security tests. For a good overview of the testing that one major commercial software company puts its products through, look at how Microsoft tests its software.
Open source software testing faces some additional challenges because of factors like source code transparency, velocity mismatches between component releases, and risks due to security and IP rights. The projects and source code in and of themselves are superior to many of the commercial products, but customers want them to be "tested, certified, and supported" for production-quality usage. Why? Because many enterprises run into some of these classic challenges when using open source:
There needs to be an "underwriters lab" for open source components:
For enterprise customers to
For open source developers to
 CodeZoo is a site for any developer who wants to avoid writing code. We believe the best code is the code you don't have to write -- the pieces already done for you, as well or better as you would do them yourself. |
Testing different versions of components on different hardware for different types of functional usage brings up the challenge of combinatorial explosion. For example, a typical SpikeSource release integration involves optimizing 272 parameters distributed across 189 configuration files across 63 components yielding 7 preconfigured stacks for developing applications in Java, PHP, Python, Perl, C, and C++.
We need a new approach in testing to address these challenges. Our recommendation is to apply the very nature and strength of the "architecture of participation" to open source testing. To quote Tim O'Reilly, "we need a system that allows for a real free market of ideas, in which anyone can put forward a proposed solution to a problem; it becomes adopted, if at all, by acclamation and the organic spread of its usefulness."
Hence the requirement to build a Participatory Testing System that allows developers to participate by testing their applications as the payload. We here at SpikeSource have adopted the following initiatives as part of building this participatory testing system:
Testing Framework Transparency
Leverage existing and popular open source testing tools, integrate them into a common framework to automate tests, and publish the results. Additionally, we develop new tools and contribute to the community. Today, we use 18 open source components as part of our testing framework.
Screenshot of PHP Code Coverage Tool (first ever of its kind) contributed by SpikeSource to the community:
|
Test Results Federation
An initiative and published interface for open source project committers, ISV developers, and customers to share and federate their test results.
|
Managed Testing Services
Let customers and developers write tests, archive them into an "uploadable" format, and run the tests in a remote and managed fashion. This helps customers and developers to verify or validate a patch with their functional applications even before the patch is officially released.
Advanced Testing Techniques
Develop new techniques to test open source projects from an inside-out perspective. A few initiatives that can be leveraged, like Binary Regression Testing, and next-generation testing, exist today.
Everybody Tests
Develop methodologies and frameworks to leverage people with different skill sets to test the software at different levels. For example, a big open source project could be formulated in subcategories, with committers doing unit testing, ISV developers testing interoperability, IT developers testing functionality, system administrators testing stress and load, and academic students contributing to testing that software in their own respective areas of expertise.
Our goal is to leverage the power of participatory testing to:
Validate a fix from any corner of the globe as quickly as possible-- within hours.
Automate mechanisms that can be built to identify source repository changes, to spider different project home pages, and to crawl multiple bug databases. Today, we pick up our bug fixes by tracking 57 mailing lists, 5 USEnet groups, and 62 RSS feeds from the open source community. In addition, the test bed must be flexible enough to accommodate different granularities of testing, ranging from minimal acceptance tests to complete suites of interoperability tests, thus optimizing the "time to test."
Here is a screen shot from our internal build and bug validation harness depicting the latest Geronimo progress:
We internally track bug validation processes through our automated alert dashboard:
|
Guarantee functionality and backward compatibility across different versions of integrated software components.
This matrix explains some of the complexity involved in testing interoperability between functional constructs and component layers:
|
Certify interoperability and manage exponentially increasing combinations stemming from alternate platform choices like:
Operating Systems: 6 Linux distributions, Windows, Solaris, ...
Database: PgSQL, MySQL, Oracle, SQLServer, ...
Web Server: Apache
App Server: JBOSS, Tomcat, Geronimo, Jonas, ...
Languages: Java, C, C++, PHP, Perl, Python, ...
As an interesting note, here are some of the statistics we have compiled within SpikeSource:
To conclude, the traditional and existing open source testing methodologies are necessary and useful, but they do not take full advantage of that "architecture of participation", which makes open source development so powerful. SpikeSource's goal is to facilitate the adoption of open source software in the enterprise through testing, certification, and support services. We innovate, automate, and optimize advanced testing techniques as part of our core competency.
Murugan Pal is the founder and CTO of SpikeSource, Inc.
Return to the O'Reilly Network.
Copyright © 2007 O'Reilly Media, Inc.
