The Crafty Turk

		Print
		Email weblog link
		Discuss
		Blog this

Tim O'Reilly
May. 15, 2003 01:06 PM
Permalink

After reading my blog entry entitled Why Scripting Languages Matter, Dave Stutz sent me the following email, which he gave me permission to share here (links added):

Graham's original essay

I've always thought of hacking as a *craft* rather than as an art. The innovation and experimentation that we use to produce software is no more or less profound than the inventiveness that has gone into the refinement of things such as weaving, mechanical devices, farming, food preparation, and other practical technologies. As with these other crafts, the tools used to practice them are a natural outgrowth of the raw starting materials. (Interestingly, as I know we've discussed, the techniques that programmers keep alive and the ways in which they propagate their knowledge of these techniques also follow the tradition of artisanal production. The organizational structure of most large website providers, corporate development groups, and open source projects is something close to the guild.)

For a programmer, the choice to use types, for example, is driven by the task at hand - most programmers use both typeless scripting languages and more rigorous compiled languages together. [Sometimes it makes sense to use gravel, but sometimes it makes sense to pour concrete. And of course, gravel can be used as an element of concrete; gravel also makes an excellent base on which to pour...]

Successful web apps, just like every other kind of app, will harden due to necessity. Once they are interlinked using web service techniques and driven programmatically, once they've got millions of barely attentive users who don't *want* the interface to change, or once the cost of operating the franchise becomes critical, the kinds of change that you celebrate will no longer be desired or tolerated. The good ones will harden with lots of flexibility built in. The bad ones will just seize up. And once they are locked down and expensive to run, techniques such as compilation and context-sharing will start to make lots of sense for safety and efficiency reasons. [A favorite example of a really efficient, large, distributed app is any living organism. The information that drives the process can be found in easily hackable source form in its genome, but the expression of this information is most assuredly typed once that genome is expressed!]

The rigidity of shrinkwrap PC ISV software is really an anomaly. Most corporate systems are just as dynamic as web apps, due to integration requirements, changes in exogenous data representation, pure size, or any number of other factors. Not surprisingly, the guilds that are the stewards for these apps make heavy use of databases, scripts, and data-driven integration techniques based on extensible formats like XML.

I guess that the point that I'm making is that I don't think that there is any fundamental paradigm shift involved in open source web apps. The tools and techniques that you are seeing have been used by developers for a long time, although the materials and resources (and therefore the tools) are changing as hardware and networking infrastructure evolve. I love Python as much as the next guy, but I can promise you that a really big, rotting old Python-based LAMP app will be just as smelly and hard to maintain in 10 years as the VB codebases that I've seen that won out 10 years ago and survived based on VB's ability to quickly "sketch" PC apps in a dynamic way. The guilds that used VB to sketch first, after which they channeled the dynamism of the app using structure *rather than code* are the guilds that are still happy and prosperous today. A formula that worked then and still works now is to couple databases/data formats with metadata-driven building blocks - in the long run, this is a far, far, better way to run a business than constant resketching and recoding. Web apps certainly have humans inside, but the less time those humans spend on the craft of programming and the more time they spend on the art of communicating at a higher level, the more valuable their contribution will be!

Just a random ramble - ah the pleasures of being out of work :)

I don't disagree at all with Dave's points about the eventual "hardening" of many web apps as time passes. And I certainly agree that corporate applications have many of the same characteristics as well applications. But because so many of the popular conceptions of software development are framed by the "cast in concrete" shrinkwrap applicationst that define the PC era, I still do think it's worth claiming that a fundamental paradigm shift in today's killer apps like google and amazon.com. While a VB app of ten years ago leveraged the "sketching" aspect of scripting, that VB app was not responding to the same kind of requirements for constant change. An Amazon is listing and delisting thousands of new items every day; they've even involved their millions of customers in the design of their interface. I don't think that the Turk is ever going to disappear from inside Amazon's machine. As long as they need someone managing all that user-contributed data, this isn't going to be an application you can set running and just walk away from. The job of developer and editor and database administrator all converge in one of these apps. As Dave points out, this is also true for corporate applications. (So maybe the real point isn't that this is a paradigm shift, but a reversion to the norm, after the aberration that was shrinkwrapped software.)

And yes, as Paul Graham points out (thinking much like Dave), in large companies, these processes are likely to be constrained, "hardened" if you will. But I do think that the skills that are required to build a successful dynamic web application are different than the skills required to build a successful compiled binary application. And it's important to celebrate those differences.

I totally agree with Dave that, as one friend once said, "you pick the hat to fit the head." Scripters typically use compiled languages as well as part of their toolchest. But there is a snobbishness among some developers of compiled applications, where they don't seem to realize just how important dynamically typed languages are.

I think Dave's comments about craft guilds are also fascinating, but that's a whole other subject... Much more relevant to Paul Graham's original essay.

Tim O'Reilly is the founder and CEO of O'Reilly Media, Inc., thought by many to be the best computer book publisher in the world. In addition to Foo Camps ("Friends of O'Reilly" Camps, which gave rise to the "un-conference" movement), O'Reilly Media also hosts conferences on technology topics, including the Web 2.0 Summit, the Web 2.0 Expo, the O'Reilly Open Source Convention, the Gov 2.0 Summit, and the Gov 2.0 Expo. Tim's blog, the O'Reilly Radar, "watches the alpha geeks" to determine emerging technology trends, and serves as a platform for advocacy about issues of importance to the technical community. Tim's long-term vision for his company is to change the world by spreading the knowledge of innovators. In addition to O'Reilly Media, Tim is a founder of Safari Books Online, a pioneering subscription service for accessing books online, and O'Reilly AlphaTech Ventures, an early-stage venture firm.

What do you think?
You must be logged in to the O'Reilly Network to post a comment.

Return to weblogs.oreilly.com.

Weblog authors are solely responsible for the content and accuracy of their weblogs, including opinions they express, and O'Reilly Media, Inc., disclaims any and all liabililty for that content, its accuracy, and opinions it may contain.

This work is licensed under a Creative Commons License.