David HM Spector credits his lifelong fascination with supercomputers to the "spinning tape drives and blinky lights" of the 1960s science fiction computers of his childhood. He grew up to be deeply immersed in computer technology during the dawning of the Internet, when its potential, like that of clusters today, was just beginning to hit the computer world.
In 1991 he was working for J.P. Morgan as a consultant when he suggested they open the bank up to something new--the Internet. Going online meant that Spector was able to fix a system breakdown in minutes, remotely, rather than in weeks. In a business where time is money, the fix netted Spector a vice presidency.
Once senior management was convinced that using the Internet could solve technology problems, Spector's Internet Service Group was challenged to discover ways to use its Internet connection to give J.P. Morgan competitive advantage in the marketplace. The "big bang" happened in 1994 when the traditionally conservative firm surprised the financial world by making their risk-modeling package, RiskMetrics, available on the Internet. It's hard to imagine how shocking this idea was in an era when banks kept their risk evaluation strategies a deep secret. But this variation on the open source advantage immediately propelled J.P. Morgan into the forefront as the benchmark for market analysis. Within a short time all banks were following suit.
Spector's access to supercomputers through his work fostered a love of super-powered research in such topics as Artificial Intelligence and intelligent agents. "I've got this fantasy that I will one day be able to design a small machine I can wear like a pager that will do my bidding without me having to tell it things," said Spector. It's his version of the Victorian gentleman's major-domo, a butler who took care of administrative details and placed fresh flowers in the hall. "I'm not enough of an elitist to actually hire somebody to do that for me; I'd rather have a machine so I won't feel guilty."
Spector's first Linux cluster, and ultimately his new book Building Linux Clusters, arose from his need for an affordable home supercomputer for his research. Building it was a challenge. "I noticed that a lot of people talk about it, but the documentation is really terse. There's no way for someone to start who doesn't have all the knowledge already." Spector wrote up an outline and realized it would make a great O'Reilly book. And, as it turned out, O'Reilly & Associates was looking for just such a book.
Building Linux Clusters is a hands-on guide for people new to clusters. "Like most O'Reilly books," said Spector, "it's to get people going. You can build a cluster right out of the box: You pop the CD in, follow the instructions, and in ten or twenty minutes you have a fully functional cluster."
Spector likes to say that the book tells a story. And indeed, the first chapter is a brief, lively history of the evolution of supercomputers and, eventually, of clusters, including Don Becker's first Beowulf cluster . Built in 1994 to meet NASA's need for supercomputer performance on a shoestring budget, it used Linux and off-the-shelf hardware, saving NASA a couple of million bucks, according to Spector.
The book defines a cluster in its simplest form as "a bunch of computers tied together with a network working on some large problem that has been broken down into smaller pieces."
Clusters have been around for years, with commercial versions from Compaq, IBM, and Sun. But Linux clusters, with their open source code, give programmers a unique versatility in adapting to the special requirements of individual problems. "If you have the source code, you can customize the thing to your heart's content to get the driver working the right way for your application," says Spector. "Same thing for all the other components inside of Linux."
And the fact that Linux is a free, versatile operating system that can run on off-the-shelf, even outdated, hardware puts the power to solve extremely complex problems into the hands of individuals and organizations with limited funding. Better still, Linux clusters are almost infinitely upgradable--you just add more units. Supercomputers, on the other hand, are outdated as soon as they reach the customer.
Interested in learning more about clusters? You'll find two sessions on clusters at the O'Reilly Open Source Software Convention:
- Introduction to Cluster Building--Get an overview of free and commercial solutions available in the Linux environment.
- Installing Linux Clusters and Server Farms with VACCINE --Join in a demonstration of a highly scalable system for setting up large Linux installations.
Not everyone goes that far, but the fact that Linux runs on ten or twelve different CPU types opens new possibilities, says Spector. "Say you're at the research lab and down the hall they've got new 1000 megahertz DEC alphas, and they no longer want their measly 500 megahertz alphas, so they're yours. You throw Linux on them and suddenly you have more power in your cluster."
The easy access to cluster-style supercomputers presents an interesting dilemma. "The U.S. Bureau of Export Administration is scared of the proliferation of supercomputers in the world because they fear that every Tom, Dick, and Harry is going to build chemical weapons and lord knows what," said Spector.
While the United States can prevent its own supercomputers from being sold to foreign companies, "you can't stop clusters," Spector explained. "If you opened up your PC and looked at the imprints on the chips, you'd see mostly Malaysia, Korea, and Singapore. Intel, Digital, HP, and Transmeta have their chips fabbed in places like this.
"So the cat is not only out of the bag, the cat is being delivered from the manufacturing point, which is off-shore. There's nothing stopping someone from picking up a zillion clones in a flea market in Taiwan, stringing them together with a hub, and getting the software, which is freely available."
What kinds of computing do clusters enable? Intelligent agents, of course. "Say you had a web agent that went out and found research reports, digested and abstracted them, and reported back to you, 'Well, I found thirty research reports and the following look like the things you asked me for.'"
Clusters are great for distributing animation tasks in movies like A Bug's Life, with some forty frames per second of film. "The computer selects slices of each frame and farms them out to thousands of computers where each individual slice is done. Then other computers recombine them into a picture."
And clusters are making headway on some previously intractable problems in complexity, like the weather. " The National Weather Service just bought a 277-node cluster from Compaq Alpha processors," recalled Spector. Other applications ripe for exploration include voice recognition, data mining (the search for trends in large amounts of information), and online trading.
"With clusters you can build a very impressive trading system that could simultaneously handle all the stock markets in the world with what's called a money matching system, where you're trying to match buyers and sellers. You can create online systems to do analysis of stocks, which could benefit even small investors."
Spector thinks that the most exciting future applications of clusters are the least possible to predict. That's because learning to work with parallel computing changes the way people look at problems and opens new possibilities for applications. "People have to step out of the mindset of being serial and think about how to do everything at the same time. And that's a hard transition, but once you do that, suddenly new vistas open to you."
In cluster programming, the harder you work to understand the nature of the problem and how it can be broken into parallel units, the more powerful and effective your cluster becomes. You become more aware of the order of sequential and parallel aspects of the task.
"Once people get into the mode of thinking about how what they're doing in the real world breaks down along those fault lines," he said, "they can start to come up with things that scale better and that are better written and less error-prone. And we'll wind up with better software. I suspect that the most surprising effect of clusters will come from some enterprising young person who has some extra machines and a good idea and will invent that next great advance in information tools."
O'Reilly & Associates will release Building Linux Clusters in July (2000).
Beta Sample Chapter 9, Application Examples, is available free online.
Copyright © 2009 O'Reilly Media, Inc.