There are a lot of applications left for clusters and grids to explore, as two companies I’ve talked to recently show.

Cleversafe: Better reliability and security by letting go of data

Cleversafe is commercializing a distributed storage technology that first showed potential back in the 1990s, and that was being widely discussed in 2001 when the peer-to-peer craze hit and people first talked seriously of monetizing the grid computing model of SETI@home. The Cleversafe approach is doubly interesting because it is distributed in two senses:

• The data is stored on many systems, geographically scattered around the world to facilitate disaster recovery.

• Responsibility for the system is distributed. Cleversafe provides only the technology (open source) and the service. Actual storage will be purchased from a variety of companies, so that no one entity has control over users’ data.

Remote storage is currently a hot topic; Amazon.com’s Simple Storage Service (Amazon S3) is a relatively conventional example garnering a lot of interest.

But modern bandwidth and processing raises new possibilities in scalability and
security. The traits of storage systems in the style of Cleversafe are as follows:

• Instead of being stored in a central location such as a Windows file server or NFS, data is split into slices stored at different nodes. A large file (like a video or a database backup), which is becoming more and more common these days, can therefore be stored and retrieved faster than a centralized system could, because it can benefit from multiple data paths through a network to multiple systems.

• The data in each slice is compressed, encrypted, and digitally signed with a checksum, to ensure that it is transmitted securely and without corruption.

• Only a subset of chunks are needed to restore the entire file. For instance, the file may be stored as nine chunks in 18 locations, and only five chunks are needed to restore the file. Even if a disaster causes 13 of the stored sites to be unavailable, the user can retrieve his data. The information dispersal algorithms that make this possible can be compared to RAID 5, where multiple disks store
  stripes of data along with parity information that can be used to compensate for a lost disk.

Several experimental filesystems have been deployed with these characteristics;
examples such as Chord and Tapestry are fairly well-known. But Cleversafe has a business plan; it involves leasing space from 11 companies and promising twelve 9’s of reliability. Because its software is open source, it has also launched a non-profit site to encourages research facilities to offer free storage.

In an age where millions of Internet users store their photos on Flickr, their calendars on Google, and their web bookmarks on del.icio.us, I am looking forward to the day when centralized file servers give way to services like Cleversafe’s. The physical network is ready for the responsibility being placed on it, and the concept is well tested.

The company has just announced Cleversafe Desktop, a graphical interface available for all major desktop systems that makes interactions with the storage network easier.

Virtual Iron: making Xen an enterprise solution

We all know that Xen lets you run multiple operating systems. It is currently associated most closely with Linux, and also supports Windows 2003. But raw Xen does not make it easy to turn virtualization into a support system for the kinds of things mission-critical sites need to do. other important applications as wellVirtual Iron is a comprehensive management interface for Xen, providing a platform and GUI interface for all sorts of valuable procedures:

• Viewing the hardware and software characteristics of the hosts on which you’re running servers.

• Adding or removing resources such as CPUs, memory, and disks to and from a guest operating system.

• Moving an operating system to another physical host.

• Setting up policies whereby Virtual Iron reconfigures systems automatically as loads change or hardware fails.

• Access control: defining users who have the right to carry out various tasks.

Imagine getting a laptop with audio drivers or power management software but no
tools to configure them. You’d spend a lot of time tweaking kernel parameters by hand. Tools make useful operating system features available to a wider audience, and this is what Virtual Iron hopes to do for Xen and cluster technology.

So Virtual Iron can be an infrastructure for building high-availability clusters–along with uses in server consolidation, capacity management, and rapid provisioning that separates development, testing, staging, and production.