Last week I was in Copenhagen for YAPC::Europe. One of the announcements at the conference was the location of next year’s conference which will be in Lisbon. The theme of next year’s conference will be “Corporate Perl”. And that (along with a couple of conversations last night) got me thinking about a talk that I’ll submit to next year’s conference which might well be entitled “Why Corporates Hate Perl”.

It’s not true, of course. There are a still large number of large companies who love Perl. I could probably work through to my retirement enhancing and extending systems that are written in Perl at many of the big banks in the City of London. There are, however, also many companies who are moving away from Perl for a number of reasons. Here’s one of the reasons that will be included in my talk.

I was talking to people from one such company last night. The Powers That Be at this company have announced that Perl is no longer their language of choice for web systems and that over time (probably a lot of time) systems will be rewritten in a combination of Java and PHP. Management have started to refer to Perl-based systems as “legacy” and to generally disparage it. This attitude has seeped through to non-technical business users who have started to worry if developers mention a system that is written in Perl. Business users, of course, don’t want nasty old, broken Perl code. They want the shiny new technologies.

And so, in a matter of months, the technical managers at this company have created a business environment where Perl is seen as the cause of most of the problems with the current systems. It’s an impressive piece of social engineering.

It’s also, of course, completely unfair. I don’t deny at all that this company (like many others) has a large amount of badly written and hard to maintain Perl code. But I maintain that this isn’t directly due to the code being written in Perl. It’s because the Perl code has developed piecemeal over the last ten or so years in an environment where there was no design authority which encouraged developers to think beyond getting their immediate task done. Many of these systems date back to this company’s first steps onto the internet and were made by separate departments who had no interaction with each other. It’s not really a surprise that the systems don’t interact well and a lot of the code is hard to maintain.

There are, on the other hand, a number of newer systems which are also written in Perl which follow current best practices in Perl development and are far easier to to maintain and enhance - as easy, I would contend, as anything written in the new approved languages.

It’s certainly true that this company has a large number of systems that need to be rewritten over the next few years. But throwing away all of the company’s accumulated Perl expertise and moving to new languages seems to be a step too far. Management are blaming Perl for the problems when really they should be blaming the management and design procedures that were in place (or, more likely, weren’t in place) when the code was originally written.

Many organisations are in the same situation, with large amounts of unwieldy Perl code. Ten or twelve years ago everyone was writing web systems in Perl and we were all making mistakes. We all have to deal with those mistakes but we’ve, hopefully, learned from them and can rewrite our systems to take account of everything that we’ve learned in the last ten years.

It’s too late for the company I’ve been talking about in this article. The anti-Perl social engineering has probably insinuated itself too deeply into the culture. It’s unlikely that Perl’s reputation can be rescued.

But if you have similar problems in your own company, then please try to ensure that blame is apportioned correctly and that you don’t use Perl as a scapegoat.

A while ago, I published a teaser on this blog about using the Web as a whole as a data storage object. At that time I said that “the Web right now is cut down into a million pieces that don’t talk to each other properly”. Almost two years have gone by since that article and it looks like not much has changed.

One of the early questions was how interoperable are Web services when they’re not envisioned and created by them same company. This problem lead to a number of initiatives that are trying to push forward Web services creation standards. DataPortability, for example, is evangelizing a number of different standards that will create a better interoperable Web:

• end user authentication through OpenID;
• inter-application authorization through OAuth;
• information syndication and distribution through RSS, RDF and OPML;
• information meaning and automatic extraction through microformats;
• user attention profiling through APML;
• messaging and information brokerage through XMPP.

This collection of standards and best practices is great when a large number of companies start following them. For us, developers, it means that by following these standards our Web services will be interoperable with all other Web services that use the same standards. It means that creating a Web service now is much easier than it would have been two years ago.

What about the end users? How can they take advantage of this interoperability? I’m not just talking about Web services that let you consume data, because that problem was solved a long time ago by aggregators. Aggregators are a good example of a class of Web application that survives because there’s a de facto standard in place: RSS.

So, my point is, how can end users take advantage of Web services that let you publish, transform and assemble information? We’re moving to a point where a number of emerging services give you a one-to-many publishing approach:

• Ping.fm and HelloTxt publish your status across multiple services, like twitter, jaiku and Pownce;
• Typepad’s Blog It publishes blog articles across different platforms and also announces them on different status services;
• twitxr publishes your pictures across different services like flickr and Picasa.

Is it just me or there’s a pattern emerging here? Users see value in these services because they save you precious time by automating repeatable actions, like publishing a picture across different services. One thing to notice, though, is that these services only provide half of all that’s possible with the existing Web.

All these services let you choose among a number of services and then broadcast your information to all of them. Forgetting minor format and content adaptations, they won’t give you the possibility of programming the flow of your information. One thing is to shoot a picture and send it to different services, another thing is letting users tell how that picture flows through different services.

One service that’s offering you the capability of configuring this flow of information is switchAbit. It evolved from an original idea by Dave Winer that you could grab your pictures from flickr and post a tweet for each one of them. Quoting Dave’s original post:

The SwitchABit platform was developed because we noticed that an ever more complex flow of ideas and information is being facilitated by editorial systems and aggregators such as Flickr, Facebook, Twitter, FriendFeed, Seesmic, Qik, Ustream, YouTube, BlogTalkRadio, Disqus, Wordpress, Tumblr, TypePad, Blogger, etc.

switchAbit is basically an RSS to publish mechanism. It’s built around the pub-sub paradigm which means that it will get your information from a number of services, filter it according to your instructions and publish part of it into other services.

With this approach you’d still have to publish your information on at least one supported service, so that switchAbit grabs it and routes it somewhere else. Another approach is acting like a reverse aggregator, extending the functionalities of Ping.fm and others by adding the possibility of configuring information flow.

You could, for instance, add a watermark or a copyright notice to the picture, extract EXIF geo-location information and send it to Fire Eagle, publish the transformed picture on a number of services, and announce it to your contacts on some social networks. And this is just an example of what can be done in the near future.

I’ve been working since January on such an application. It has an interface similar to Yahoo! Pipes, but it lets you compose the flow of information from a starting point through a set of Web services that exist on the cloud. Because of the obvious similarities of this concept with the familiar UNIX pipe, it’s called tarpipe. Quoting tarpipe’s blog original post:

tarpipe will also create an ecosystem where Web applications and services will be able to receive and transform media content. Users will take advantage of this ecosystem by defining delivery and transformation workflows for their documents.

With tarpipe you can direct the output of one Web service into the input of another one. This makes different services virtually interoperable, even if they’re not able to talk to each other individually. It also gives end users the ability to compose flows of actions (or workflows) for their information. It currently accepts information sent through email and a REST endpoint, meaning you can extend your application by connecting it to tarpipe.

So, my initial thought that “the Web right now is cut down into a million pieces that don’t talk to each other properly” is not so true anymore. There are ways of making the Web more interoperable, like following de facto standards and creating programmable service adapters.

At last Thursday’s Ignite Boston, which I wrote up in a previous blog, provided an unexpected mirror in which two opposing views shined on each other, each view provided by one of the two keynotes by John Viega and Jonathan Zdziarski.

Both Viega and Zdziarski.are security experts and authors of books by O’Reilly and other publishers. Viega used the bully pulpit for an entreaty against the “full disclosure” philosophy, a fundamental article in the open source catechism. Zdziarski, who had not consulted with Viega beforehand, endorsed full disclosure whole-heartedly and with a doggedly pragmatic intent. The context for Zdziarski’s approach is the Apple iPhone, which has security vulnerabilities that, in his experience, Apple doesn’t fix until they’re made embarrassingly public.

Today Zdziarski sent me a long and frightening article from the National Journal about the threat of cyberwar. Although the basic premises in the article have been circulating for years, many of the details were new to me. And despite the focus of the title on China, the article makes it clear that governments as well as individuals (the “cyber-militia”) are engaging in disruptive behavior around the world. In fact, the article cites worries about what may be happening in the NSA.

It seems to me that the National Journal article provides more fodder for Viega than Zdziarski. Veiga insisted that the black hats planning DDOS attacks and identity theft aren’t as smart as they are commonly made out to be. They couldn’t create as much havoc if they had to rely only on the vulnerabilities they found themselves. They are helped immeasurably, he said, by the revelations of vulnerabilities in major software products by people with no malicious intent. The worldwide database of known vulnerabilities is swelled by individuals trying to show off their technical chops, and by companies in the security business trying to demonstrate the indispensibility of their products.

So long as software vendors are slow to fix bugs, full disclosure has to be an option, a kind of last resort, and I think Viega allowed for this. Open source projects have to promote a sense of responsibility among contributors to be discreet in reporting bugs with security implications. Perhaps it doesn’t matter much anyway–because most people keep using unpatched versions of software long after fixes come out.

Giles Bowkett’s Never Hate. Only Destroy. (disclosure: contains language your local third graders probably use and your work filter might block as inappropriate) contains a side point which crystallized something I’ve pondered for several weeks:

The whole point of the Cory Doctorow Problem is that the fundamental assumption with Internet celebrities - that a very smart person will always be interesting - is false…. What irritates me is essentially a search failure; I can seek excellent insight on social software and end up reading pointless trivia about a corporate amusement park filled with plastic birds on plastic trees.

This is my problem with current social networks as well. Your information is either public or it’s not. You’re either connected to someone or you’re not. There’s little to no sense of context.

The ever-creative Wade Olson (of KDE fame) tells an interesting story of immediately losing interest in otherwise-interesting hardware due to “Intellectual Property” protections. He caught himself going from caring to not caring in the time it took to read the phrase “Don’t expect Linux support anytime soon.”. His conclusion is:

Vendors need to beware: Intellectual Property gains, once thought to be a Competitive Advantage, will continually over time become a negative branding attribute.

I’ve noticed this myself. I don’t particularly care what Microsoft does, what NVidia does, or what Adobe does. Their products don’t really matter to me when I can use other products without giving up freedoms I consider essential.

Have you had similar experiences?

A proposal to help editors work better with dynamic languages — by not pretending they are static, and by leveraging their unit tests.

As a Test Driven Developer, using dynamic languages, editors frequently disappoint me. The main thrust of editor research, for the past few decades, targets debugging static languages. This post suggests a very simple fix.

For years, many people have argued that one of PHP’s big successes is deployment. The language has little to recommend it for anything beyond simple database-backed HTML templating, but there’s little easier than dropping a couple of .php files in a directory through FTP.

While there are still millions of wonderful (and ultimately unproductive) flamewars about how mod_php is faster than vanilla CGI Perl and Ruby uses too much memory and FastCGI is unstable and shared-everything on a monster JVM is obviously more scalable, none of that will ever matter to most of the deployed PHP code in the world today.

A Perlbuzz commenter named Yudel made the deployment/colonization point very clearly:

I still think in Perl, but as an only occasional programmer, I seldom find it the best tool for the job. The Perl community failed to successfully colonize the new ecosystems of programmers who don’t have root access. Simply asserting that PHP is linguistically inferior won’t convince anyone who has had to argue with a web hosting company about the load MovableType was placing on their servers.

mod_perl is great for what it does, but it’s clear that mod_perl isn’t what hosting providers most wanted. A slim Perl distribution — including perhaps a new Apache httpd module which only embeds Perl — with a good templating module, the DBI, and perhaps an XML parsing module or two could have put Perl on more $4.95/month hosting plans. The corollary to that of course is an easily installable bundle of Pure Perl for an application.

Sure, that doesn’t cover everything. You probably can’t get RT orPlagger or Angerwhale in such a system, but it’s a start.

Ceding the very low end of a technology to an upstart is just one of the ways to let distruptive innovation eat your lunch.

One flaw in this argument is that approximately zero webhosts supported Ruby before the Rails lovefest. As well, the Rails deployment strategy went through several iterations. Here’s the interesting point which subverts my argument somewhat: Rails hosting suddenly became lucrative enough that several Ruby-friendly hosts appeared.

I haven’t yet figured that out.

Yesterday Google celebrated the opening of a larger Cambridge, Massachusetts office, which takes up a substantial part of a building right next to the Kendall/MIT subway stop in the higher-than-high tech area of East Cambridge. I got a look at their new Friend Connect service (covered in a related Radar blog) and heard some fascinating comments that the staff kindly let me reproduce here.

Google staff certainly know how to say the right things and react in ways I approve to the situations Google finds itself in. More and more people I know (including authors) are Google employees, which is statistically predictable because more and more people in general are Google employees. The Cambridge office has been growing wildly since it began with the purchase of the company that created Android. And this office is one of 45 Google offices around the world.

This raises the question of whether the empire can be supported through continued sales of advertising, and whether Google’s stated openness carries through to employee behavior on the ground. I explored these questions with managers and staff at

I like numbers. They can mean a lot of things.

Rather than continuing silly arguments over obfuscated and flawed measurements of “language popularity”, perhaps a better way of measuring the viability of a language or platform is to measure the freshness of its ecosystem.

LaPerla’s How Fresh is the CPAN? measures the upload dates of one of the world’s largest and most active repositories of free software. Of the 12,000 (or is it 14,000 now?) distributions on the CPAN, 25% have a most recent upload date of February 2008 or newer. Half have an upload date of 2007 or newer.

You don’t get those kinds of statistics by putting “Ruby Programming” into Google and pretending the results are meaningful.

I promised to explore the theme of Free-loading Adoption of F/OSS in more detail. Alan Rimm-Kaufman’s Why Small Businesses Should Support Open Source is a great place to start:

It doesn’t matter if your donation is large or small. It doesn’t matter if you give money or code.

What does matter is this: if you’re benefiting from the Open Source Movement, try to give something back.

It makes good business sense. And it is the right thing to do.

Before I joined O’Reilly, I worked in a small consulting company. As I joined, the company was migrating away from proprietary platforms to open platforms. We saved a tremendous amount of money and gave our customers far better service. Being able to use, modify, and redistribute free software let us finish jobs we’d never have been able to do otherwise.

In return, we submitted bug reports. We occasionally submitted patches, both on and off the clock. We knew that we owed a great deal of our business to a healthy commons of free and open source software… and now I know that keeping that commons free, open, and healthy was vital to the business.

The owner of the company allowed the Portland Perl Mongers to meet in our offices once a month. He rented chairs for the meeting. Maybe it’s not a big thing, but it was a way to pay back part of one of the communities which had produced so much great software integral to our business.

You don’t have to hire a core developer. You don’t have to release your own (non-derivative) source code. You don’t have to donate money to a foundation, or host a conference or a meeting.

You don’t have to contribute back to the communities which produce software you rely on… but if you rely on it now, aren’t you interested in its healthy future as well?

Issue 16 of [IN]Secure Magazine is available. Mirko Zorz interviewed me in this edition (Page 41). If you decide to read it, I’d be delighted to hear your thoughts and feedback. The magazine edition of the interview is much better looking and highly recommended (as are the other articles), but for the sake of convenience, the interview session is below.

Small ISPs such as Glass’s (and yes, they do exist, even today) have none of the incentives that network neutrality advocates attribute to major carriers to discriminate against voice, video, or other content. In fact according to Glass, new applications such as VoIP are great because they provide new business. “This week we hooked up a VoIP company which was dissatisfied with the quality of service it was getting from the incumbent in our area. We deployed a low-latency, high-bandwidth radio link just for them, at a cost (parts and labor) of about $1,000. We can justify the cost because we will be paid for the service. It’s cost-shifting without compensation that’s the big issue for all ISPs–large and small.”

Glass has a stake at least as precarious in the current Internet economy as the media companies using peer-to-peer transmission as part of their business plans. Laws or regulations that fail to take economics into account, in his view, could put small ISPs out of business. He defends his position fiercely, and gets plenty of flak in return. I consider Glass a friend and have even planned to tap him as an author on some projects. So I want his view heard as a balance to the “just throw more bandwidth at it” proposals.

That said, I wonder whether the problem is really peer-to-peer protocols, which Glass focuses on, or high-volume media such as video. What architecture could handle the video experiences Internet users want. Compression can achieve impressive quality at reasonable bandwidth, but the sheer volume of everybody sharing the network stresses current transmission systems.

Matt Asay kicked up a small controversy in MySQL adoption: Deep and wide when he wrote:

Now the only thing missing in that conversation is the enterprise stepping up to pay for some or all of its free-loading adoption of MySQL. This is what is prompting MySQL to consider new licensing models. It would be very easily resolved by enterprises for owning up to and paying for the value they derive from open source, very little of which comes down to a lower price tag.

I’d like to extend that to projects beyond MySQL and to a definition of “contribution” far beyond opening a checkbook. Here’s my thesis: if your organization derives some benefit from a community-driven software project, you have a moral obligation to contribute to the health of that community in some way.

I’ll write more about this tomorrow.

There’s been some recent chatter and speculation on the upcoming enhancement to the PCI standard. Among the discussions, I’d like to publicize my opinion on one argument I’ve heard multiple times during the last few days. The argument goes something like this: The cost of performing security code reviews is too high, but the cost of performing black box reviews and/or implementing web application firewalls is lower. Therefore, the solution is to recommend that organizations rely on penetration assessments and/or web application firewalls.

Remember Andy Lester’s rant about Can’t You Just…?. There aren’t often easy answers in any field.

I really like what Chris Cummer had to say in a comment on “All I Need is a Programmer”:

Every time you use “just” to describe a feature or a process it tells me you’ve made a gross assumption about what I’ll need to do.

(Of particular amusement is when non-contributors tell volunteers what to do in free software projects.)

A conference attendance that tops 2000 suggests that a technology involves a certain number of subtle angles. MySQL became a hit because installing it and manipulating tables were so simple–and yet when you get serious, the simple things start growing hair.

If you look at the CPAN test reports for Parrot, you’ll see that the pernicious and persistent problems relate to odd bits of not-quite-always-cross-platform math, specifically floating point numbers and not-a-numbers.

It’s reasonably easy to find and read the C89 and POSIX specifications. They’re well-published. Even if there are confusing parts and contradictions, you can look for them and find the specifications.

Now try to find current information about how various operating systems and the various compilers and major versions and minor versions and libc versions in all of those combinations interact. In short, if I want to figure out exactly what OpenBSD does in its C library in its standard configuration to return a negative floating point 0.0 (or not, as the case may be), where do I go, what do I read, who do I talk to, and — most importantly — how do I change the software I work on to deal with those platform-specific quirks such that the users of my software don’t even have to know that these quirks exist? Ditto GNU/Linux, Cygwin, FreeBSD, Mac OS X, et cetera.

I know that knowledge exists somewhere. Perl 5’s core encodes it somewhere. I suspect the same is true of Python and Emacs and Guile and Glib and kde-base or related projects. Yet the knowledge in code is only useful in two ways. First, in projects that use the code directly and only need to trust that it does the right thing. Second, if other people read the code.

There ought to be a third option: encapsulating that knowledge outside of code somewhere. Maybe this specific case is documented and I just can’t find it. The same goes for alignment concerns (64-bit Intel/AMD, PA-Risc, Sparc, ARM), pointer sizes, and other information.

Does this information exist in one or two good places, or does everyone have to track it down on his or her own? Worse, does everyone just hope these problems never come up?

Didn’t get one of the 10,000 golden tickets in special Google-brand chocolate bars? Python isn’t your favorite language? Not sure about hosting your code and data with the world’s largest ad broker? Never fear — Google’s not the only supercomputing grid in the world. It may not even be the largest.

Computerworld reports that the top botnets control over a million computers and can deliver over a hundred billion advertisements per day. MapReduce and AdWords have nothing on this.

Yes, the deployment platform is mostly Windows, and you don’t exactly have professional system administrators in charge of every whim and need of the machines, but you do have root access, and there’s little chance the box owners will suddenly yank your code and data if your business model conflicts with theirs. Google’s offering has a ways to go if it wants to compete.

Benjamin Otte’s Open Source will scale brings up an interesting point.

If currently 1% of the world uses GNOME and it suddenly were 100x as many, we’d be at 40 million bugs right now.

The persistent lie that increased usage guarantees hordes of available volunteers descending from heaven to wipe out all signs of resistence (accompanied by the appropriate Wagnerian soundtrack) is one of the most pernicious Myths Open Source Developers Tell Ourselves.

Though Benjamin finds hope in the fact that users tend not to report bugs (and distributions don’t work well enough with upstream), the entire scenario still bothers me. I don’t mind fixing an interesting segfault now and then, and I’m always happy to fix a well-reported bug with a test case and a sensible description which helps me reproduce the problem. Yet I don’t scale, especially for projects where I can devote only a few hours a week.

That’s not a few hours every week, either.

Bug reporting, bug triaging, and bug fixing are all activities present in healthy project communities. In return for the freedom of using great software (often at low or no cost) with no usage restrictions and few (if any) distribution restrictions, users have the responsibility of ensuring the community’s long-term health. That may mean submitting a bug report, testing a development version, posting a bug bounty, producing a patch, or even sponsoring a developer. Otherwise, you’re relying on the goodwill of volunteers who’ve already more than paid their obligations to you — and I’m concerned about the long-term sustainability of that model.

Sometimes I wonder of the dual-licensing model is actually healthier in some ways. At least there the costs are explicit and fungible.

I’m at a unique symposium this week named Codework, which I do not dare to describe because it has barely begun. I can only say that snagging Ted Nelson to deliver the opening talk was not only a great motivation for attendance but an exquisitely appropriate historical marker for the workshop, which bills itself as “Exploring relations between creative writing practices and software engineering.” Nelson, of course, is one of the first people to recognize the benefits literature and computing could offer each other.

Readers have plenty of ways to learn about Nelson’s famous Xanadu and his more recent project Zigzag, one of the best ways being to hear him speak as we did in a full hall last night. Thanks to the World Wide Web that Nelson perennially maligns, he is much more famous than he otherwise would be, and also has much more opportunity to spread his views. But here I’ll just jot down a few observations I haven’t seen others offer about Nelson’s ideas, and that aren’t immediately obvious from his talks, fascinating and well-argued as they are.

Adobe has released a beta of AIR for Linux. Good news, everyone with a 64-bit processor, or PPC, or Sparc, or ARM, or anything more exotic than 32-bit x86. AIR for Linux Release Notes say that all you need is:

Processor - Modern processor (800MHz or faster)

Finally, web applications have freed us from the tyranny of worrying about such difficult issues as endianness, alignment, and struct padding!

(Sorry “by the end of the day” turned into four days. I was in rural Pennsylvania with no Internet!)

In 2006 I was taking a look at the then unreleased XenSource XenServer 3.0. The server was running on a Dell laptop on a filing cabinet next to my desk where the XenServer management interface was open on my desktop. My wife walked into our office, looked over my shoulder, and while pointing to the monitor on my desk asked, “Is that the Xen thing you said you were going to be reviewing?” I responded that the laptop next to me was running Xen, and that she was just looking at the management software.

The fact of the matter is that to most people, the software that manages virtualization *is* virtualization — a fact that may save companies like VMware. See, the virtualization management interface is the most public facing component of the virtualization ecosystem, and two crucial parts of this ecosystem are quickly becoming commoditized: the hypervisor and the virtual machine (VM). The entire virtualization ecosystem is being redefined, and in a few years the companies that wish thrive in this market will need to focus on an entirely different set of technologies than they like to tout now.

This blog briefly discusses the commoditization of the different parts of the virtualization ecosystem and what areas companies like VMware will need to pay attention to in order to survive software giants like Microsoft and open source alternatives such as Xen and KVM.

I chuckled at a couple of quotes in Java performance improvements touted, specifically one from Cliff Click:

As your program grows in size, the lack of strong typing basically kills your ability to handle a very large program and so you don’t find the million-line Perl program.

I’ve met Cliff, and he’s very smart, but I have to disagree on two points. First, no one who’s used anything with a better static type system than Java consider’s Java type system “strong”. (If you can still get a NullPointerException from a generic-enhanced collection, Java has a ways to go.)

Second, the reason that there aren’t many million-line Perl programs is that the people who are capable of writing and managing million-line Perl programs have better ways to organize their projects than glomming a million lines of Java into a single shared-everything instance. That’s setting aside the qualities of encapsulation and abstraction that Java-the-language doesn’t have, preferring instead to push that problem to tool vendors and AbstractFactoryFactoryInjectors which consume vast swaths of XML to get around Java’s static code fetish. I can only imagine how much larger the Java code would be without all of those XML files.

I also recommend James Robertson’s take on things, from Earth to Sun.

I’m curious to hear how many million-line Java applications exist in the world and what they do. I suspect that they’re primarily web applications that speak SOAP or REST over strict SOA or HTTP boundaries — just the sort of boundaries beyond which it doesn’t matter if your code is Java, Perl, C++, or the Korn shell. You know, because they’re completely network bound.

Hello. My name is Andrew Kutz, and I am honored to be blogging for ONLamp on the topic of virtualization. Please watch this space for news on VMware, Xen, KVM, and other virtualization technologies. I’ll be creating my first real post later today. If you have any ideas with regards to what type of content you would like to see, please let me know by shooting me an email at akutz at lostcreations dot com.

Thought for the day: If the preferred scaling strategy of Java web applications is shared-everything in a beefy JVM with plenty of threads in myriad pools (and it seems to be) and the preferred scaling strategy of LAMP applications is a shared-nothing architecture across plenty of boxes with memcached in front of a replicated database, what changes will be necessary to run popular apps written with shared-nothing in mind in a shared-everything environment?

Bonus question: besides web applications and language research, are dynamic languages on the JVM interesting? (The clever reader will see where this line of thought leads.)

In my (painfully) long career as a software engineer, I’ve often run across the attitude that code has intrinsic value. You see this frequently in the industry when ‘code reuse’ is used as a metric of efficiency. At several companies I’ve worked at, old and badly bit-rotted products have not been rewritten because “we’ve invested umpty-umph million dollars in that code, we’re not just going to throw it away.” This whole attitude is bull-doodoo, and here’s why.

In amusing synchronicity, I was reviewing Bernard Golden’s Open Source Maturity Model earlier today. Then I read My Visit to Sun, where he describes a conversation he had with Simon Phipps before giving a talk at Sun.

In particular:

Many enterprises seeem to operate in a vendor-centric model: they select a vendor and from then on rely on the vendor to define when new technologies should be adopted, when new releases should be rolled out, even what complementary technologies should be implemented. It’s obvious that this causes middle-of-the-pack performance, lock-in, and lack of pricing power. Without rehashing all of those arguments, consider the other implication of this approach: it fosters dependence — an inability to self-direct in technology direction, custom architecture, and unique business offerings. If all you can offer is off the standard menu, you will never serve up differentiation.

When you give away software and trade license fees and pre-sales for support contracts and free downloads, you break the passive-adversarial model between vendor and customer that has served IT so poorly for the past two and a half decades.

That’s not a safe thing, nor an easy thing. That’s still a good thing.

Companies are constantly opening new veins of ore as they attempt to mine the Internet for useful information. Developers and open source system users will be particularly interested in a SourceLabs announcement of a service called Self-Support Suites that has been in beta since December. This tool combines enormous amounts of information indexed by SourceLabs from bug trackers, technical mailing lists, and other sites to help open source users diagnose problems. They’ve just put up a free download.

The proof of concept I heard from Byron Sebastian, CEO of SourceLabs, concerned a site that spent two weeks trying to track down the failure of an Apache Project module. SourceLabs’s system found a bug report with the fix in a few minutes by finding a match between a stack trace provided by the user and a stack trace provided by a question in a public forum message. This search was more difficult than it might sound, because stack traces don’t match precisely and their contents are not unique strings that are easy to search for. Sebastian says that stack traces and log files tend to have the most useful information–but if other information was organized better, it might rise in value.

Oh, joy. Adobe is at it again.

AIR applications are deployed as a single AIR file that works identically cross-platform. The api’s within AIR are identical across different operating systems so any application behavior will work the same regardless of where it is running. Regardless if you use HTML/AJAX or Flash/Flex to build your application the API’s are identical and run on MAC/WIN/LIN without issue.

Ted Patrick, Why Adobe AIR?

Given that Adobe’s evangelists have a very difficult time telling the truth about which platforms Adobe actually supports (particularly pernicious with regard to Flash; see Uh, Thanks for the “Linux” Support for one example), does anyone really think that AIR will run on anything more exotic than 32-bit x86 GNU/Linux? Set aside the fact that, as much as Ted’s quote may make you think that AIR runs on “Linux” right now, it sounds like no one outside of Adobe will see that binary blob until later this year.

When I think about cross-platform support, I think about the first time I sent e-mail on the Internet via a FidoNet gateway accessed through a PC bulletin board from my Commodore 128 over modem-to-modem dialup in the very early ’90s.

Again (I always have to disclaim this), Adobe has every right to support only the platforms and processors it wants to support. I have no problem with that.

As usual, I offer any Adobe evangelist, manager, or developer the chance to prove me wrong, publicly, by successfully installing a publicly released version of Adobe Flash on the GNU/Linux laptop sitting six feet behind me in my office. (Good luck; it has a PPC CPU.)

Just don’t tell me that you offer cross-platform support and then stick me in a ghetto because I’m using the wrong operating system and the wrong processor. I know what cross-platform support means — you can still browse the web on a Commodore 64 — and your walled garden isn’t it. For all its flaws (don’t get me started on the codec licensing nonsense), Moonlight has a better claim to cross-platform compatibility. For starters, it doesn’t lock you out if you happen to be using the wrong type of CPU.

(I thought one of the goals of high-level programming languages and frameworks and virtual machines was so that you don’t have to worry about the details of the lower levels. Of course, I thought one of the goals of web applications was independence of platform at the level of operating system and below. Shows what I know.)

Inkscape developer (and fellow PDXer) Bryce Harrington mused about fixing critical bugs in Inkscape 0.46 to be in The paradox of FOSS projects supporting Windows.

Unlike the philosophical argument described in The Dubious Benefits of Porting FOSS to Windows, Bryce makes a more concrete, more pragmatic argument. In particular, the ratio of potential contributors to users on non-free platforms is measurably smaller than the same ratio on free platforms.

I’ve noticed this problem in some of my own projects. There are plenty of users willing to try a piece of software that may or may not work well on a non-free platform, but when it comes time to debug and fix these problems, their motivation goes away.

I’m sympathetic; it’s not fun to try to build and debug software on Windows. I don’t use it. I don’t understand it. I’m not the person you want telling someone how to install any of the free compilers that somehow don’t come bundled with Windows, just so that someone might be able to produce an interesting backtrace.

I’m not sure there’s a good solution, at least without enfranchising users to become contributors — and that seems to require Free platforms.

Back in December, I was one of the 170,000 eager Linux geeks who forked out $400 for the privilege of getting my hands on a One Laptop Per Child XO. To be honest, my initial impression was not great. For one thing, the keyboard was (and continues to be) the wrong size and feel for productive typing by an adult. I really can’t complain about this, as it wasn’t designed for adults, and I have managed to adapt somewhat to get something approaching a reasonable typing speed.

More troubling was the poor state of the WiFi stack, which seemed incapable of connecting to a WPA encrypted network. Even when using WEP or no encryption at all, getting and keeping a connection seemed to be a hit or miss adventure. And while the Sugar OS was certainly innovative, it really started to grate on me when I tried to do the kinds of things I wanted to do with it.

A quote from Steve Jobs during the iPhone SDK Press Conference last week:

If they write a malicious application we [will] track them down and tell their parents.

In other words, the iPhone applications will need to be digitally signed by Apple, and the developers will be required to register with Apple. It will be interesting to see what kind of information developers will be required to provide to Apple to register. Will they ask for the developer’s credit card number? How will the developers authenticate their identity with Apple before they are allowed to submit their applications to be included in the store inventory?

If observation is the first stage of scientific discovery, watching what people are doing in a field will tell you what the academics and theorists will write about in a few years. By this reasoning, SD West and SD Best Practices are important bellwethers for programming theory, even though there’s little theoretical about the conferences.

On March 10, O’Reilly will release Jonathan Zdziarski’s book iPhone Open Application Development, currently available as an online RoughCut. Online and brick-and-mortar bookstores will have the book as soon as shipping permits.

Apple has just released a toolkit for application development on the iPhone. Readers will naturally ask: what is the relationship between the material in the book and Apple’s development environment? Can they apply what they learn in the book to Apple’s toolkit?

InformationWeek interviewed Steve Balmer on several subjects, including open source. Here’s a choice quote:

We’ve always tried to get innovative work to happen on our operating system, and I want Windows to be the number one destination for open source innovation.

Steve, you could stop by dropping the Unreasonable and Discriminatory Pay-to-Interoperate Tax on commercial open source distribution. These developers already pay the Windows tax. So do their customers. Taxing them again with a licensing scheme that allows you to pull the rug out from under their feet if you decide to compete with them is, in my mind, something other than encouraging development.

I suspect that true innovation doesn’t have to put up toll roads.

In a recent discussion on lkml (found thanks to LWN, Ingo Molnar defended checkpatch, a Perl program which reviews proposed patches against the kernel’s coding standards. In particular:

you might know that Deja-Vu moment when you look at a new patch that has been submitted to lkml and you have a strange, weird “feeling” that there’s something wrong about the patch.

It’s totally subconscious, and you take a closer look and a few seconds later you find a real bug in the code.

That “feeling” i believe comes from a fundamental property of how human vision is connected to the human brain: pattern matching. Really good programmers have built a “library” of patterns of “good” and “bad” looking coding practices.

If a patch or if a file has a clean _style_, bugs and deeper structural problems often stand out like a sore thumb. But if the code is peppered with random style noise, it’s a lot harder (for me at least) to notice real bugs. I can notice bugs in a squeeky clean code base about 5 times easier than in a noisy codebase. This effect alone makes checkpatch indispensible for the scheduler and for arch/x86.

…

I’ve yet to see a _single_ example of a good, experienced kernel programmer who writes code that looks absolutely careless and sloppy, but which is top-notch otherwise.

I’ve spent a lot of time this past year cleaning up a medium-sized codebase written in cross-platform C89 (c’mon vendors, can’t you upgrade your compilers tosupport C99 already? I know Microsoft has a few programmers on staff.). Coding standards help immensely.

Even something as simple as aligning the equals signs in assignments and separating paragraphs in code with newlines increases skimmability greatly. Ingo’s right though, and I wish I’d realized it consciously earlier. Consistent code requires much less brainpower to decipher its structure, leaving that much more brainpower to find real problems.

While there may never be a refactoring called “Beautify code block” (and technically that’s not improving the design of the code), it’s an important tool for human analysis of code.

I’ve slaved away for the past 8 months or so writing a book on Dojo. It’s definitely not a lot of fun moonlighting for such an extended duration on a single project, but my hope is that it’s all going to be worth it when I am able to deliver what I believe will live up to the “definitive guide” standard that so many other O’Reilly books have delivered in the past. (Yes, that previous link shows that the book is titled Developing with Dojo, but it’s really just a placeholder. The book has since been retitled, there will be a different animal on the cover, etc.)

The fact that I’ve spent so long working on what’s essentially documentation of all things should tell you that I’m a huge Dojo advocate, but why did I like Dojo enough to spend hundreds and hundreds of hours writing a ~500 page book on it?

Here are some of the reasons that come to mind immediately:

• Breadth and Depth - It includes a highly optimized JavaScript standard library that protects you from the bare metal of the browser and allows you to write portable code, a terrific set of widgets that you can drop right into the page, and build tools that you can use to compress and consolidate your JavaScript to squeeze as much performance out of it as possible. Those are really broad strokes, but hopefully you get the idea that it’s not just about widgets or just about DOM manipulation. It’s about creating a great user experience and necessarily includes a swath of stuff that helps you to make that happen.
• Awesome Community - The help these people have given me in writing this book is nothing short of incredible. It’s kind of strange to “know” people for months and months via an IRC chat room and somehow feel like they’re your friends, but that’s the way it feels. It’s amazing to watch some of the most hard core committers in the project field some of the simplest questions that come up with grace and patience. (And believe me, I’ve come really close to starting a riot a few times in there.) Drop into #dojo on freenode.net if you ever want to see just how inviting the place really is.
• Clean, liberal, licensing - This post does a great job of explaining, a lot better than I could right now, so have a look.
• Pragmatic philosophy - It doesn’t try to re-invent JavaScript or build a brittle, artificial language on top of it. It embraces JavaScript for what it is, plugs holes where they need to be plugged, smooths a few things out here and there, and otherwise leaves the language alone.

Now, without starting a flame war, I’m curious about some things:

Have you tried to use Dojo in the past and gotten frustrated and dropped it for something else? If so, why (and what did you end up liking better?)

Would a solid book on Dojo have made a difference?

Have you started with another JavaScript toolkit but migrated to Dojo because the other one didn’t quite meet your needs? You don’t have to say which one you moved away from, but it would be interesting to know why you ended up moving to Dojo.

Sebastian Silva is a bit worried. Three hundred thousand XO units will arrive in Peru by August, the largest deployment in the world by far. (Uruguay started this past December and will reach one hundred thousand laptops this year.) As a volunteer on the OLPC support team, Sebastian wants to make sure the children, teachers, and technical trainers are ready.

Sebastian sees the promise of computing as in social terms. With XO, every person becomes a TV broadcaster–and XO is even more empowering, because TV broadcasts can’t be chopped up and digitally altered. Sebastian says he learned Logo before he learned to read and write, and its mastery gave him a learning attitude that he’s preserved his whole life. He wants children not just to make movies and write papers, but to appreciate the drastically open nature of the systems they’re given.

“If OLPC succeeds, there will someday be more OLPC-like systems in the world than traditional PCs,” he says. “The Internet could very well be transformed by the behavior of people using the OLPC systems.”

You have probably heard of AJAX, but have you heard of Comet? Yes, they are both household cleaners you might find under your kitchen sink, but that’s only part of the story.

In the web realm, Comet refers to an architecture that enables a server to actually push data to a client without the client explicitly requesting it. Given that web servers were built to be very good at doing exactly the opposite over a stateless protocol, you should be quite intrigued at this point, so head on over to Comet Daily, a fantastic site that includes incredibly informative articles on Comet, to get some insight on what is going to be the next big thing in web programming.

Right now, there is a series running with the theme of Colliding Comets which includes a number of presentations and rebuttals that discuss opposing philosophies on how to most effectively realize Comet. Trust me, you won’t be disappointed. This is good stuff, and it’s full of substance.

If you find Comet interesting, you may want to also take a look at Dojo, an industrial strength JavaScript toolkit for developing web applications, because it includes cometd, a client that can talk to a Comet server if the server implements the Bayeux protocol.

My upcoming book, tentatively titled Developing with Dojo, includes an in depth example of cometd. We hope to have the book available via Rough Cuts very soon, so if you’re interested in using Dojo with Comet, keep an eye out for future posts about its release.

Eugueny Kontsevoy writes in Web vs Desktop Nonsense:

Can we see past the browser? Can we accept that browser is just a runtime library that most people do not need to download to consume your application?

…

Come on, the “anywhere” part should not come at expense of losing 90% of other features.

…

Yes, future belongs to web applications, but I am not so sure that browser, with [its] weak runtime and close to non-existent programmable graphics, should remain a necessary vehicle for it.

The more I think about the subject, the more I believe that HTTP is a truly successful distributed system because it doesn’t try to solve what most distributed systems tried to solve. It doesn’t try to blur the distinction between local and remote resources; everything is a document accessible through a URI. (At least, that’s true if you’re a RESTafarian, which you should be.)

The question Eugeney raises is important. We already know how to build decent — even good — native applications. Is the zero-footprint installation (minus several hundred kilobytes, if not a few megabytes of JavaScript, or a few tens of megabytes of Flash, Air, Silverlight, Moonlight, or Java) so compelling that everything has to get crammed into the web browser model?

In my mind, that’s a mistake as big as pretending that accessing a distributed component is exactly the same as accessing a local component.

(Quick musings on my lunch break.)

When Microsoft wants to compete, it acquires. (How many successful products have escaped the $7 billion a year Microsoft Research?) Apparently Microsoft wants to compete better in the world of the Internet, so it’s trying to capture Yahoo and all of those lovely page views.

As of this writing, Google Finance on Yahoo (I like the irony of that link) says that Yahoo has a market capitalization of almost $40 billion in 1.34 billion shares and a price to earnings ratio of 63.51. Apparently a lot of people believe that big profits are around the bend.

A $40 billion acquisition is a big deal, and apparently Yahoo’s holding out for more. Everyone has a price — if the board doesn’t like $40 billion, will enough shareholders like $50 billion? $60 billion? A new board might be more pliable.

However, instead of trying to make this acquisition work, Microsoft could turn its baleful eye down the street a little bit.

Chasing page views and a new market dominance in Internet advertising may not be in Microsoft’s best interest. Of course, you know that eventually every Microhoo page will have little pockets of Silverlight here and there. If the Rich Internet Application model threatens the Windows hegemony, Microsoft has to react to preserve the attractive profits of its cash cows. Right now there are two serious competitors. One is whatever HTML 5 and Ajax may produce in the next couple of years. The other is a somewhat cheaper acquisition target.

Google Finance on Adobe reveals a much smaller, choicer target, with a $19.91 billion market capitalization and 568.96 million shares. Microsoft might not even have to borrow money to make this acquisition work.

For half the price of the hostile Yahoo acquisition, Microsoft gets control over PDF (axed), Flash (axed), and Air (axed). Talk about a destination, too — there’s a long-standing rumor that the single page with the highest PageRank anywhere is Adobe’s page to download Acrobat Reader. Microsoft also gets a fair few developers who it can retrain to port Flash to .Net while they’re on the way out the door, and there’s your backwards compatibility strategy — at least for all of those platforms which have officially blessed, sanctioned, and (here’s the important part) legal-codec-supported Silverlight ports.

(Linux users, do you have the Novell SUSE Genuine Advantage?)

What Microsoft doesn’t get is a better foothold into the currently-lucrative online advertising market… but that’s the single leg of Google’s one-legged tripod. It’s difficult to see how selling ads will prolong the life of Office, however… but maybe that will give the Office product team enough time to port a few tens of millions of lines of code to the CLR.

I watch what people like Brian Aker and Brad Fitzpatrick do very carefully. As developers go, they’re among the best at making little problems go away so as not to distract them from developing, releasing, and maintaining software.

Brian wrote recently about encouraging very easy commit access to contributors. I can’t add anything more to what he said, except to echo his concerns about the implications for making a new release of software.

I’ve seen this pattern of open repository access in the past few years, first with the Pugs project, and then with Brian and also Adam Kennedy and Michael Schwern. It’s a model that works in certain cases.

As Brian mentions, a release has certain implications. The best way to know that your software works on the platforms you care about is to test it on those platforms — and I know all of these developers believe in the value of automated testing.

The great mass of developers might just now be hearing about distributed source code systems. That’s fine. Plenty of them will realize how valuable it is to commit without network access or to branch cheaply and locally, or to have the project history instantly available, changeset by changeset, when they need it. The problem isn’t a trivial one, nor is it a completely solved problem, but we’re beginning to understand it better, and the solutions we have now are decent and robust and continue to improve.

Now it’s time to figure out the next problem. As Brian described it:

I want all open source projects being filtered into a network where users can run slaves that do regression testing for them.

We’ll likely discover that on platforms so exotic that available developer support and expertise is minimal (Windows, anything non-x86) that the next bottleneck is people who can fix problems, but being able to track which patch broke support for a platform is highly useful. Maybe for now even the minimal ability of being able to submit a patch to a farm of EC2 instances to get back “Everything passed!” or “Something failed!” results in seconds, rather than minutes, would be an improvement.

Alternately, opening up commit access might mean that we can say “If you want this to run on your favorite platform, you’re going to have to get involved.” Maybe making it even easier to contribute to a project will finally let us be clear about the true cost of free software: if you rely on it, you must help the community support its long-term health.

Adriaan de Groot, vice president of KDE e.V., wrote a short essay on Target Platforms for KDE.

His divisions of four types of target platforms is instructive; the two major axes are free/non-free and Unix-like/other.

“Unix-like” is an awfully big grab-bag, however. The joys of trying to figure out dynamic linking on Mac OS X ought to put to lie the shrill claims of the turtlenecked faithful that it’s just BSD with a shiny GUI (maybe BSD circa 1987, when everybody wanted a magnesium case). I almost dare not even imagine how much fun it is to coax compilers on non-free Unix-like platforms into interpreting modern C++ correctly.

The nice part of free Unix-like platforms is that they’re easy to obtain and install. The monster machine sitting in my other office can run multiple VMs for *BSD and OpenSolaris simultaneously, so testing a patch for portability requires a little bit of system administration and a little bit of discipline to script the process. Testing a non-free platform, Unix-like or not, is much more difficult.

POSIX and free redistribution and source code availability gives us a much better chance of figuring out and fixing those problems than we’d have otherwise. However, that’s no substitute for platform-specific experience – having an OpenBSD VM running doesn’t mean that I automatically know why OpenBSD’s handling of, for example, complex math is different from that of FreeBSD, or how to fix it. Sometimes there’s no substitute for a little elbow-grease from a passionate user of the platform. It’s nice that projects such as KDE actively support it.

It’s a sad old story, a story we’ve all gotten tired of–the patent so brainless as to be almost worth citing as a creative act of industrial sabotage, yet awakened from years of dormancy with a hungry ferocity to claw and mangle everything in its path. This particular patent is being exerted by Trend Micro Incorporated against Barracuda Networks, Inc. for a firewall product incorporating the popular open source spam filter, ClamAV. Only this court case stands in the way of a power grab that would require all open source work on virus filtering gateways to cease.

The Trend Micro patent (5,623,600) simply suggests that virus filtering be provided in a firewall. That’s all. Patents are supposed to cover things that are novel, and not obvious to a person having ordinary skill in the art. This patent meets neither criterion. Although it was filed in 1995 and granted in 1997, Barracuda has found a good deal of written evidence that filtering at the router was widespread earlier. And if lots of people are installing virus filters on their desktop computers throughout a company–any fifteen-year-old could say, “Why don’t you put it all in one place under the control of people who know what they’re doing?”

Ted Neward attempted to pull apart some of the silliness in the debate over scalability with Can Dynamic Languages Scale?. In particular, one of the most important insights is:

There’s an implicit problem with using the word “scale” here, in that we can think of a language scaling in one of two very orthogonal directions:

1. Size of project, as in lines-of-code (LOC)
2. Capacity handling, as in “it needs to scale to 100,000 requests per second”

I think it is extremely important for an organization to account for the reality of doing business (Risk based approach compared to the purist mentality of securing everything) when strategizing an information security plan. It is true that an individual who has a habit of perceiving security issues as purely a technology problem without understanding the business reality is likely to make bad security decisions.

However, I think some people in corporate security take this argument too far and end up awarding critical roles to individuals that do not have the appropriate skill-set and mind-set. More often that not, this happens when organizations responsible for information security misunderstand the argument to mean that you only need to probe for the understanding of business fundamentals and process management when recruiting for talent. Depending upon the criticality of the role awarded, this can deem disaster.

Apple did a beautiful job creating this device. Millions wanted it the moment it became known, and thousands wanted to write programs to explore its ground-breaking interface elements. Apple, however, failed to release its APIs, much less any toolkit or run-time environment.

So the community built its own.

Not only is free software development unprecedented in its size and geographic spread–hundreds of people from countries around the world collaborating on individual projects–but it brings together people who are notorious for having trouble dealing with other people. That’s really impressive when you think about it.

Of course, the stereotype of the computer programmer with Aspergers Syndrome is overblown. I used the term in my title to attract attention, but I’ve worked with enough programmers to know that many warm and socially sophisticated people take up the job.

Let’s put it more gently: many programmers have the feeling their people skills haven’t kept up with their technical mastery. That’s why they are attracted to sites such as Perl hackers Michael Schwern’s geek2geek, whose motto is “What we have here is a failure to communicate.”

How does free software development work so well, then? People often remark that the Internet made the explosion of development in the mid-1990s possible, but they focus (wouldn’t you know it!) on the Internet’s technical functions: instantaneous transmission, exact replication of content, etc. Occasionally a general “nobody knows you’re a dog” comment gets thrown in too. But we have to consider the social behavior encouraged by the tools the geeks developed.

J. David Blackstone has a pointed journal post entitled The Right Way To Do It which praises Perl’s “There’s More Than One Way To Do It” philosophy:

TMTOWTDI is anarchy. It scares people who want to keep order by force.

Allowing people the freedom to choose from many different ways of doing things is a recipe for disaster, we’re told.

… in my experience, it’s been the Perl code I’ve had that is readable, well-designed, and maintainable. It’s been the Java code I’ve seen that is ugly, poorly-designed, and unmaintainable. There are certainly exceptions to both sides of this.

Yet it’s not about Java versus Perl (and certainly not Perl versus Python).

“Perl is dead”, crows TIOBE’s January 2008 index. The world belongs to Python.

You see what you want to see in statistics though.

For example, you could compare Perl, Python, PHP, and Ruby job trends. Don’t drop those sigils yet.

Or compare Perl’s delta to C’s delta. Both lost ground in the TIOBE index, but C declined by almost twice as much.

Here’s a fun one. TIOBE’s editorial says that C# and Java will eventually be the two most popular languages. To do this, C# has to surpass Perl. That’s a problem though; it gained more than Perl lost and still slipped a position and is still more popular than Perl.

Ultimately this isn’t even good stats porn though. There’s no analysis of why languages have gained or lost in popularity. Without that, there’s no good way of deciding what these statistics mean. Without that, it seems silly to declare winners and losers and long-term trends. (One might also suspect that the actual release of Perl 5.10 and the buzz around that from the second half of December versus the “imminent” release of Python 3 may shift numbers from this point on.)

I’ve long believed that the easiest way to install software on a modern operating system is through a well-designed package manager connected to one or more carefully-maintained package repositories. Thus my brain always shudders when someone says “OH it’s so EASY to install software on MACZ just drag and drop! woo!!” (Why should I have to fire up a web browser, navigate to a website, find the download link, figure out which version works with the dependencies I have installed including the OS version, pick a mirror, and then figure out where the file actually downloaded? I suppose it’s likewise easy to get a Ph. D. in theoretical physics — just walk on stage when they call your name.)

I do remember the bad old days when installing something reasonably fresh required me to trawl through rpmfind.net looking for, if I were exceedingly lucky, an RPM built for the particular version of the particular distribution I run, or barring that an SRPM that I could coax into doing the right thing. There were still benefits to using a packaging system (mostly dependency tracking), but that’s more work than I want to suggest to my parents.

I was late to Debian (my first installation was 1999), but apt-get was a clear improvement for installing and updating the entire operating system, especially when combined with the quality and breadth of packages available for Debian.

These days I use aptitude, which is even more so.

Again, I’m not sure that I would suggest that my mother make a habit of running this on the command line by herself, but she’s perfectly capable of copying and pasting a few commands from an e-mail to keep her system up to date or install new software, and the process is much simpler than giving her a list of directions to navigate a web site. I doubt I’ll ever catch her running aptitude search, which is fine… but I use the command frequently.

The process of installing software in such a way that it does not conflict with other software, includes dependencies in a sane fashion, and receives security updates almost automatically for the whole OS (not just the kernel, GUI, bundled web browser, and DRM-laden media player) is now something I almost don’t even think about. That is the sign of a truly useful piece of software.

Thanks to the contributors to Aptitude, apt-get, dpkg, and the Debian and Ubuntu repositories.

I’m glad to see that TPF’s public relations group has spread the Perl 5.10 press release far and wide, and it’s getting some coverage. However, some of that coverage reminds me why I don’t watch television news and why I treat the newspaper as entertainment and not information. Consider eWeek’s First Release of Perl in Five Years Arrives:

Perl is a dynamic scripting language widely used in everything from Linux system utilities to Web servers to full-blown graphical enterprise applications.

What’s a “dynamic scripting language”? Is there such thing as a non-dynamic scripting language? (No one seems to know what a scripting language is anyway.) Minor nit.

During its 20-year history, it gained massive popularity by assimilating the syntax from many predecessors, making it really easy to use for anyone already versed in sed, awk, grep, csh, C/C++, Lisp, and so on.

Syntax, maybe (but Lisp? Really?). Features, sure. Easy to use? That’s debatable. Easy to start to learn, yes. I don’t know that anyone will suggest that Perl is easy to master, though I’m happy to argue that its learning curve is gentle if long.

… languages like python with rigid syntax structure have arguably gained ground in recent times over perl, for applications that are developed collaboratively.

“Arguably” is a weasel word, so you can throw out this whole sentence. I’m not aware of any statistics that show that Python is more popular than Perl. (Arguably, the Maginot Line gained a lot of ground in the southward direction. There. Now Python fans and the French can berate me in the comments.)

Additionally, scripting languages specially-made for use on the Web, like PHP and Ruby, have eroded some of perl’s once formidable share of the dynamic Web server scripting scene.

There’s that “dynamic scripting” mess again. What does that mean? What’s static Web server scripting anyway, and why would you need a programming language for that?

My favorite part however isn’t about Perl at all. Did you catch that? Apparently Matz was really busy in 1993 writing Ruby not as a general purpose language but specifically to use on the nascent Web. How prescient.

If a journalist can rephrase a press release and make this many errors in five paragraphs in a subject I know something about, how many errors are there in subjects I don’t know as well?

Oh, and the title comes from the last television news promo I ever watched, during the X-Files finale. The local Fox affiliate played a blurb for the evening news where the newsbimbo said, and I am not making this up, “Now that the X-Files is ending, let’s see what the series taught us about real aliens. Stay tuned at 10.”

I haven’t read many RSS feeds lately due to time constraints from writing a book, but I came across this post by Adam Gomaa, by way of some of the responses from Jonathan LaCour’s blog.

If there is one thing to learn from the endless, and pointless, comparisons of frameworks in Python, i.e, popularity contests, it is that the most important things for a web framework’s popularity, in order, are:

1. Documentation
2. Marketing

In a popularity contest, the “best” framework, is going to have the best documentation, and the best marketing, like tons of screencasts, etc. Every other discussion is an exercise in futility. If you want to be home coming queen, people have to know who you are, or they won’t vote for you. One of the reasons why Django is considered the most “popular” or “best” framework to many people, is that the Django people did an incredible job of documentation, and marketing.

While Adam brings up some very valid points in his criticisms of Turbogears and Pylons, he ultimately misses out on the real problem and the real solution. While I also like to find reasons to stretch my intellectual muscles and use legendary books to compare and contrast ideas against real world problems, it doesn’t work for this comparison.

The real problems doesn’t involve fancy computer science terminology, or “Conceptual Integrity”, it involves something much more mundane…..documentation. It is a known fact, that smart people, often the very smartest, don’t like to document their work, as they are too busy “inventing”, and being mad scientists. I am quite certain, that core Turbogears and Pylons developers, and power users, like Bob Ippolito, know how to do things that are unbelievable, but creating bulletproof documentation is tough work. Documentation is a job, and that is why millions of technical books are published making millions of dollars per year. If you consider documentation to be full time job, then people need to get paid to do it properly.

The reason why everyone doesn’t have killer documentation and marketing, is that it is perhaps, the most difficult part of being, “in the framework business”. Documentation is grunt work, plain and simple. To quote Bruce Lee, “I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times.” In a similar sense, I fear not the framework that has 10,000 features, but the framework that documents one feature 10,000 times!

Of course, ultimately, a popularity contest is mostly meaningless for people who actually develop in a specific framework and make it their job, unless they are consultants for hire who only build a “insert framework here” websites. As anyone who has attended a 10 year high school reunion can attest, being “popular” isn’t always best, but due to human nature they will always be confused. Look at all of the “quiet”, corporations, all over the world, that just do Plone development, they are so busy making money, the framework issue must seem like background noise.

Also, it is interesting, but not surprising, that Zope 3, ZODB, Plone 3, and Grok are virtually non-existent in discussions about Python Web Development, in some circles. I wonder how anyone can feel comfortable getting into a massive diatribe on Python Web Development and completely dismiss any discussion about Zope related technology. Listen to the people who write the PEPS. If someone has gotten more than one PEP approved then their opinion is worth 1 million times the regular joe. Last time I checked there are only a handful of people who can say that, Phillip Eby, being one of them. He seems to think Zope is relevant, why don’t you?

Another troubling thing I see when Python Web Frameworks are brought up, is a lot of discussion about people writing wiki’s and blogs, etc, in the hot new framework. I suppose I should probably bring up that fact that Plone 3 is one of the most kick*** Content Management Systems on planet earth, see PyATL. If you want a blog, install Plone, if you want a wiki, install MoinMoin, or Plone. Unless you want to learn about writing a CMS or Wiki, you are essentially, self-gratifying yourself, to put it diplomatically, by recoding a solution to a problem that has already been solved.

Finally, I think WSGI is going to change everything, because it is about reusing components and products. Some of the current web framework comparisons are just silly. It would great to live in a world in which the real issues are discussed like, “My documentation kicked your documentation’s *ss!”. He who wins the documentation and marketing, wins the hearts and minds! To steal from a famous political campaign, “It is the documentation stupid!”.

Links:
Noah’s Personal Blog
OS X Automation
PyAtl

Jon Allen announced that http://perldoc.perl.org/ has been updated with Perl 5.10 documentation. This is great news as the look of the rendered PODs with syntax highlighting and many other nice effects makes me feel good.

The Peer to Patent project (which I’ve reported on before) just pointed me to a particularly broad patent that could encumber user interfaces on the web and desktops for years to come.

The idea in this patent, submitted by Yahoo!, is a clever little idea: if someone is starting to drag an icon or mail message somewhere, why not bring the mountain to Mohammed, so to speak? If she is dragging a photo, for instance, the browser or operating system can guess that she wants to open it with PhotoShop or the Gimp, and present that choice right next to the icon for the photo.

I’ve been wondering that lately. I’ve been using Subversion for … well … what seems to be a lot of years now. Looking back at the dates for Subversion’s history and coinciding them with events that were happening in my life, I’m guessing that I started using Subversion no later than the end of 2003. (So, maybe that’s not “a lot of years now”…) And I was using CVS from about 2001 until I started with Subversion. During the majority of that time, I have been either the sole commiter of the code base that I was working on, or one of very few people working on the same code.

Recently, I started a job where I’ll likely be working more mingled in with other developers on the same code at the same time. Everything is set up using Subversion. Before starting this new job, though, I began looking into distributed source control, which is the cool new kid on the block. I’ve created some personal projects using Bazaar and have glanced at Darcs, Mercurial, and Git. I like Bazaar a lot. It can be a little sluggish at times (like pushing, pulling, and merging), but not unbearably so - and I expect that it’ll get better. I keep running through my mind how moving to a distributed model would impact the work flow with my co-workers and I’m not totally convinced that distributed is always the way to go.

I just finished reading this piece of a conversation with Linus Torvalds regarding Git, and I remain unconvinced that going distributed would be the best thing for us. And I’m guessing that maybe most small teams of “closed” development probably don’t need a distributed source control system, either. It seems that the problems that have spawned this new model of source control is more of a problem for open source development, particularly of larger projects, and less of a problem for smaller proprietary development. For example, it’s really important for Linus that Linux kernel developers (or anyone, really) be able at any moment to create a new branch. It’s important for Linux that people be able to experiment with new kooky ideas and maybe come up with a cool new feature to go into the kernel. It’s also important to Linus that people be able to do so in anonymity. I can see how this would be important for a project that has potentially tens of thousands of developers interested in experimenting with the code and are doing so on their own free time. I think this is less important when a small team is being paid to work on a code base. Typically, you don’t have the leisure time to perform experiments. If you do need to work on an experimental feature set, it’s not a problem for a repository admin at work to create a branch for you. And anonymity isn’t typically necessary at work. At least, not anywhere I’ve ever worked.

I can see how having an “off-line” repository could be helpful. But from what I’ve heard, svk should address a lot of those issues. And most of these distributed systems are reputed to handle merging between branches better, which would be nice. I love Bazaar and would love to use it at work, but I’m just not convinced that it buys us enough benefits to switch from Subversion. Does anyone have convincing reasons that a small, closed source development team should consider switching to a distributed tool?

Mike Shaver has a deconstruction of false statements from Adobe about Flex’s openness. In particular:

[Adobe evangelist James Ward] has the nerve to call them “the community” and indicate that their work is a remedy for Adobe simply not being willing to remove the field-of-use restrictions on their existing documentation.

… and:

… they don’t want people to think too hard about the fact that writing to Flash is committing yourself to proprietary platform…

Maybe Monopolight will save the free software/free data/free community communities. (Why Monopolight? There’s a single vendor for licensed Silverlight codecs, Moonlight doesn’t provide the Silverlight codecs, and they’re x86 binary blobs only. Gee. Thanks.)

There’s a lot of speculation at any given time about what the future of the web is. I don’t know what it is, but I think I know what it’s not. Facebook.

Yeah, yeah, I can hearing the booing from here. Easy now, fan boys.

The truth is that the web rides waves of innovation and everyone on of those waves has a trough between it and the next wave. Facebook is currently experiencing the slide down the backside of the wave and is headed into the trough. Why?

Before you dash of a nasty reply, hear me out. There’s an excellent lesson that all companies can learn from this.

By now I suspect that many of you have already heard of Perl On Rails, an internal BBC project. Naturally, this made Slashdot, Reddit and Digg, amongst many other sites and one chap put up a very scathing post entitled Why the BBC Fails at the Internet. Most of the comments you’ll read about the “Perl on Rails” project are pretty far off the mark, but the “Fails at the Internet” post was, despite the vitriol, probably the most spot-on analysis of the problem facing the BBC.

Admittedly, I’ve only worked for the BBC for a month, but I’ve already had several friends ask me for the “inside dope” about this project and I wouldn’t say much for two reasons. One, I had only heard about it second hand and was concerned about getting technical details wrong. Two, I live in the perpetual fear that anything I say in my blog will be held against me. I’m probably right on the first count, but I’m definitely wrong on the second.

Here is a pro-Python propaganda comic. If you’re in to Perl, don’t take the alt tag (by hovering over the image) personally - it’s an xkcd comic. You know, the same comic that depicted Stallman as a ninja? Enjoy! (I know I did.)

I spent yesterday at the XML 2007 conference in Boston. It’s smaller than last year’s conference, which is a shame because I liked the sessions I attended better than last year’s. The knowledge and skills of the attendees as well as the presenters seemed impressive. Here are a few musings that resulted.

Here in the US, it’s Thanksgiving, a day of eating lots of food, watching football, and sometimes, just sometimes, expressing gratitude and giving thanks for those things that make life wonderful.

Here are the things I’m grateful for in late 2007, in no particular order after the first.

Google Code

Google’s project hosting service has been a godsend. It’s changed the way I do open source projects. It has leapfrogged SourceForge for ease of maintenance, and the bug tracker trumps RT for CPAN that we’ve been using for so long. Add that to the integration with Google Groups which makes it trivial to create mailing lists, and it’s at the tops of my list for 2007. I can’t say enough good about it.

The readers of Perlbuzz

Eleven weeks ago, Skud and I started this little website called Perlbuzz as an alternative to the “more traditional outlets” for news in the Perl world. The response has been tremendous. We get 600 RSS readers every day, and have had over 10,000 unique visitors in that time. It makes me happy that our little venture is used and appreciated by the community.

Test::Harness 3.0

It’s been over a year in the making, but the new version of the crucial Test::Harness 3.0 means more flexibility for module authors, and lots of UI improvements for people who just want to run prove and make test.

Mark Dominus

MJD is so much a fixture in Perl it’s easy to forget that he’s there. For 2007, though, never mind all the things he’s done for Perl in the past, or the hours I’ve spent being enthralled in talks of his. His Universe Of Discourse blog is the single most intelligent blog out there, and sometimes it just happens to be about Perl.

Andy Armstrong

Was Andy Armstrong always around, or did I just not notice? His time and dedication spent on climbing on board with Ovid and Schwern and the rest of the Test::Harness 3.0 crew has been invaluable in getting it out. Plus, he’s a really swell guy anyway.

Dave Hoover

When I finally despaired of the amount of time and frustration it took to organize content for Chicago.pm’s Wheaton meetings, Dave Hoover stepped up and volunteered to take it over. I’m thankful, but not as much as I hope the other Chicago.pm folks are.

Perl::Critic

I’m all about having the machine keep an eye out for the stupid things we do, and the goodness of Perl::Critic is always impressive. You won’t like everything Perl::Critic says about your code, but that’s OK. It’s an entire framework for enforcing good Perl coding practices.

The Perl Community in general

The Perl community is populated by some tremendous folks. Some names are more known than others, but these people help make daily Perl life better for me. In no particular order, I want to single out Pete Krawczyk, Kent Cowgill, Elliot Shank, Liz Cortell, Jason Crome, Yaakov Sloman, Michael Schwern, Andy Armstrong, Ricardo Signes, Julian Cash, Jim Thomason, chromatic, Chris Dolan, Adam Kennedy, Josh McAdams and of course Kirrily Robert. If you think you should be on this list, you’re probably right, and I just forgot.

My wife, Amy Lester

Because even if she doesn’t understand this part of my life, she at least understands its importance to me.

I’d love to hear back from any readers about what they’re thankful for. I’m thinking about having a regular “Love Letters to Perl” column on Perlbuzz where people write about what they love in Perl.

Here’s a provocative thought from Mark-Jason Dominus on a rainy Friday afternoon:

Programming cannot be run on the convoy system, with the program code written to address the most ignorant, uneducated programmer. I think you have to assume that the next maintenance programmer will be competent…. If an incompetent programmer has trouble understanding your code, that is not your fault; it is their fault for being incompetent. You do not have to take special steps to make your code understandable even by incompetents, and you certainly should not do so at the expense of making it harder for competent programmers to read and understand, no, not to the tiniest degree.

— Mark-Jason Dominus, Creeping featurism and the ratchet effect

Please note that I don’t believe that any one particular programming language can solve this problem; see Does Your Programming Language Have Magic Powers? This is a people problem.

Last week, the Perl Foundation announced that the Mozilla Foundation has awarded a development grant to Patrick Michaud to work on the Perl 6 on Parrot compiler. Thank you, Mozilla Foundation (especially Zak Greant and Frank Hecker.)

This grant is particularly important because it’s the largest grant of money to any Perl 6 hacker in several years. There have been a couple of other grants; the NLNet foundation made a very generous grant to Parrot a couple of years ago, which the Perl Foundation is doling out as Parrot hackers reach specific milestones. A well-known Perl-friendly development and consulting company also sponsored several microgrants for Perl 6, Pugs, and Parrot.

If you’ve read between the lines of several of my recent postings related to development, motivation, sponsorship, and scheduling (for example, Squeezing One Year of Work into Eight) you’ve probably recognized that I believe strongly that external resource constraints have slowed Perl 6 development dramatically. (Alternate phrasing: the progress of various Perl 6-related development efforts is impressive considering how few resources any of the projects actually have.)

I have confidence in the work of Patrick and a strong belief that the Mozilla Foundation’s generous grant will enable him to devote more attention to the Parrot compiler tools and Perl 6 on Parrot — and even more important, to recruit and encourage more developers, testers, documenters, and participants to participate. I do believe that we can develop and are developing high-quality, useful, inventive software for comparatively few resources, and I’m proud of what we’ve all accomplished.

Still, as Allison Randal’s Impact of Parrot Grant report to NLNet illustrates, even modest sponsorship can help us work even more effectively.

Think of the community-driven software projects you rely on and please consider how individual and company sponsorships may help us build and maintain a powerful, useful, and above all free ecosystem of high quality software.

I went down to the Cambridge, Massachusetts lab of One Laptop Per Child today to find out what they’re doing with mesh networks. This was a particularly appropriate day for a blog on OLPC, because today they’re launching a fifteen-day-long purchasing opportunity called Give One Get One. You pay them for two of their brightly colored, impressively lightweight computers, and one goes to a child in a developing nation, while the other goes to you.

But the whole point of this blog is that a One Laptop Per Child system has limited value on its own. Its most innovative and powerful features lie in its participation in a mesh network with other laptops. So get your neighbors and workmates to buy them too!

We’re used to think of system-enforced access policies as crude and coarse-grained, such as the setuid permission bit that lets a user execute a program as the file’s owner. Fine-grained access has to be enforced by individual applications, a laborious coding process that is weakened by not being able to take advantage of underlying operation system security. PolicyKit, developed by Red Hat and included in Fedora 8, ameliorates this unsatisfactory situation.

Red Hat developer David Zeuthen describes through examples the types of access problems solved by PolicyKit: “it’s fine to mount removable media; it’s not fine to mount fixed media; it’s not fine to change the timezone.” These are operating system capabilities that can be enforced by such operating system components as the HAL or the filesystem. But any application can use PolicyKit API to enforce any kind of access it chooses to, and it gets the backing of the operating system. Zeuthen compares PolicyKit to Authorization Services on Mac OS X and Group Policy in Windows.

UPDATE: If you want see an “application”, Plone 3.0, run under WSGI with a “framework”, Grok, you can check out this demo on the Repoze Site. This is one of the more exciting things I have seen all year. So what is this called, a framework, a product, WSGI…you tell me

Our local Atlanta, Python User’s Group, PyAtl, is collaborating with the local, Plone User’s Group, to build a Plone 3.0 site. I have been attending the Atlanta Plone meetings recently at ifPeople, and we had a mini-sprint last weekend where we got a demo site running behind a, newish, forward proxy server called Varnish.

On Wed, I attended an Atlanta Plone meeting, where Brandon Rhodes, gave a presentation on Grok. Brandon has kindly agreed to post his presentation on his blog, which he did in KeyJnote, which is written in Python, of course.

In Brandon’s talk, he mentioned that Grok, which is built on top of Zope 3, is less of a framework and more of a component based architecture. In looking at the Grok website, I liked two things in particular:

1. Grok offers a lot of building blocks for your web application.
2. Grok is informed by a lot of hard-earned wisdom.

One of the reasons why web application developers should look at Grok, and Zope 3, in particular, is the second point. Failure over a long period of time, leads to experience. When people learn from their failures, it then turns into wisdom. The Python world has much to learn from Zope, and that is why I am especially interested in Grok, and Zope 3. Remember Zope Corporation was founded in 1995, a lifetime in software development.

All of this background information brings us to the meat of the discussion, which is Repoze, WSGI, and the eventual death of webframeworks as they exist in their current state. I for one, welcome our new WSGI, overlords. WSGI, and products like Repoze, will make webapplication frameworks much less important, and component based architectures, much more important.

One of the massive, unaddressed until WSGI, problems of standalone, web frameworks, is that, ultimately, they could never win in the long term. If a web framework has a frenzied pace of development, then the API, and documentation will never be accurate. Additionally, web frameworks, like Django, can lock you into a specific set of “tightly coupled” components, in which your ability to use other components, like SQLAlchemy, is not possible.

With WSGI, these issues become much like relevant, as something like Repoze, can use the WSGI spec, to do things like, “allow for systems that participate in a WSGI pipeline to make use of the existing two-phase commit transaction management provided by the ZODB transaction package.” WSGI is clearly the future for Python web development, and in hindsight, it even seems silly that such an emphasis has been placed on web frameworks as a one stop solution.

Clearly the better solution, is a loose set of components that work together in a unified fashion under one specification, much like how the standard library of Python itself works. I suppose, to further illustrate the point, can you imagine how irritating it would be if you used, say, the tarfile module, and that meant that you could not use the zip module? That would clearly be unpythonic, and this is why WSGI is the new buzzword. By using the WSGI spec, and optionally, something like Repoze, combined with deliverance,it is possible to run multiple web applications, running under one URL, all with the same “skinning”, or look and feel. This smells like victory, for Python web application developers.

Note update: I want to make it clear that I am not slamming any web framework, including Django, my point is that ANY web framework can run independently inside of something like Repoze, coupled with Deliverence, making even a full framework an actual component and in fact eliminating problems like one architecture is tied to a specific ORM, or templating engine etc.. My apologizes if this point was not clear in the original post. What this does is take the emphasis, and pressure, off of a site being one specific component architecture, or framework. A site won’t be a “Django site”, or a “Turbogears” site, it will be a “website”.

(Thanks to Derek Richardson, for being such a strong advocate of Repoze, WSGI, and deliverence, as much of this post is based on things I have heard for the first time from him. On that note, it is looking more and more likely that we have the authors of Repoze speak at PyAtl in December, 2007. If you are interesting in helping to sponsor this event, please contact me or Derek Richardson.)

[1] Update: Update May 17th, 2008: I would refer anyone who is angry about my opinion to first read this essay by Paul Graham: How To Disagree. Note, I privately attempted to contact Jacob Kaplan Moss to prevent him from leaving the drunk post, but I received no response.

[2] Update: A particularly juvenile response to this post seems oddly relevant to this article from the New York Times. I don’t see any distinction between his behavior, and the protagonist in the Times article, “Weev”. I attempted to contact the poster privately via email to resolve the issues, and was confronted with a similarly juvenile response. I have the correspondence available for interested parties.

Even though I do my best to avoid using Windows, at all costs, unless it is a function of work I need to perform, I am bit impressed with how far IronPython is getting along. Even with recent news stories about ugly blue screens, and cries from a crotchety old man, about finally considering Linux or Mac, Microsoft is doing something right.

I recently starting reading Steven Holden’s blog, and in this post, he mentions some of the exciting developments in IronPython as discussed by Michael Foord. I have played around some with IronPython, and watched a presentation on IronPython at our local Python User’s group on it, recently. I wonder though, if Apple is falling a bit behind Microsoft in its support for Python as a first class development option?

I went to an iPhone Development talk today, which covered the webkit side of iPhone development, but I wondered if Apple would be forward thinking enough to beat Microsoft to the dynamic language battle, and do the iPhone SDK right. Doing it right, would be to think of the API in terms of Python and Objective C.

For example, are they going to allow pure Python code to write applications using the iPhone SDK? Apple has soundly, and routinely, beat Microsoft in the Operating System war, since OS X, but what about after that? What if the future is led by dynamic languages, and Microsoft is way ahead with IronPython?

I am not knocking the incredible hard work, and effort, that has been put into PyObjc2, and the further integration of XCode with Python. This is wonderful, and I cannot wait to start digging into PyObjc2, but does the PyObjc team have the same resources and support from Apple, that Iron Python has from Microsoft? If not, then maybe it should be a higher priority for Apple to look to the future and put more research, money, and energy into their support for dynamic languages, as this may be the next battleground for hearts and minds.

Personally, I would love to hear about Apple hiring a few Python people like Google did, and to start developing the Cocoa API with Python directly in mind, not as an afterthought. I cannot even imagine how many new, yet very experienced, developers this would bring to Cocoa, it would be mind boggling. While Apple is at it, they could abandon Applescript like they have done with carbon. If you look at GNU/Linux, it has an interesting near equivalent to Applescript Script Recorder in the form of Dogtail, where you can record and write UI events in Pure Python.

I personally think dynamic languages are an ever-growing part of the future, and I hope Apple continues their meteoric rise by taking a lead in adopting them.

One hoary truth of computing technology is that most of the pressing problems today have solutions discovered or developed, at least in part, twenty years ago. (This nicely avoids the patent problem.)

Google’s announcement of the OpenSocial API brought up yet again the persistent problem of walled gardens on the Internet, as myriad social networking sites spring up, offer to invite all of your friends if you divulge your address books, and then slowly wither as you realize that visiting half-a-dozen sites every day to read messages from your fragmented social groups is busy work.

Wouldn’t it be nice if all of these disparate messaging systems could interoperate?

Over the weekend I encountered a dusty old RFC written in 1982 that might solve this persnickety interoperability problem. Jon Postel’s Social Messaging Transport Protocol describes a system that relies on the combination of your unique identifier (username) on a social networking site with a unique identifier (domain name) for such site to produce an Internet-wide addressible identifier uniquely identifying, well, you. Given this unique identifier, any conformant messaging system can use this Messaging protocol to send you, well, a message.

Lest you fear such a system (a quarter century old!) is inextensible for the 21st century uses of zombie bites and pokes, the system builds on the tried-and-true HTTP style header/body distinction, where headers are simple key/value pairs that get ignored if unknown but offer plenty of ways to encode Gravatar or OpenID information, if necessary. (Admittedly, an extension of SMTP in RFC 2821 is only six years old, but provides further useful updates.)

The Social Messaging Transport Protocol enables the lovely store-and-forward behavior we’ve all come to expect from social networking sites, where you can send a friend a message and he or she does not have to be online to receive it. (Unfortunately, there is a flaw in the system; sending messages does not work if you are on an airplane. I can’t find a technical reason why it shouldn’t work, but I probably overlooked a MUST NOT in the RFC. One potential solution is for social networking sites to offer miniature versions of their sites that users could download and use directly while offline, but is that really easier than never traveling anywhere you don’t have broadband access?)

There are other tremendous benefits from adopting this ancient protocol for interoperability between social networking sites, but there is one drawback (besides the fact that you can never fly again): there’s a slight potential that you may receive unsolicited messages from people you don’t know. Fortunately, unsolicited commercial messages are almost unknown on all popular social networking sites thanks to the diligence and exemplary customer service shown by their operators.

A final benefit is worth noting. The Social Messaging Transport Protocol has a companion protocol revised most recently in RFC 3501, the Instant Messages, Advertisements, and Pokes protocol, which allows you again to use the unique identifier granted to you by a site with the site’s own domain name to login, view, and manage all messages held for you at the site. An IMAP widget could be embedded in any web page to allow you to check your messages from any web site.

(Imagine if we could solve the problem of local storage of data from Internet applications!)

I’ve already started writing a Javascript implementation of the protocol, but the Digg It! widget conflicts with the pulsing yellow update of my rounded corners, so it may be a week or two before I can release something usable.

Ian Bicking discusses Mozilla Prism in Prism:

Here’s a general rule I have: I don’t accept anything made by people who hate the web. If you hate the web and you want to improve the web, I don’t want anything to do with you… To me Silverlight and AIR reek of a distaste for the web.

While I think that HTTP and REST are the most important parts of web applications, not the web browser (and certainly not HTML and JavaScript), credible (that is, both free-as-in-freedom and not web hating) alternatives to AIR, Silverlight, and Flash are very welcome.

Wikipedia estimates that Facebook has 47 million active users, as of this month. If, as reported widely, its founders believe it’s worth $15 billion, each active user account is worth around $319.15. That’s old valuation math, though–bog-standard cost of acquisition. Facebook is something new. It’s a platform for hosting applications to take advantage of 47 million users who like to poke each other electronically.

As far as I can tell, the second best reason to use Facebook is the Facebook Zombie Application, which lets you bite 20 of your friends every day to turn them into zombies too. (The best reason, according to my contacts in institutions of higher learning, is to watch the friend list of your next potential significant other to see if your rival gets more attention than you do.)

If every active Facebook account signed up for the Zombie app, the value of the site becomes approximately $15.98 per daily zombie bite. This introduces new funding models for publicly traded platform-only companies in a new economy; perhaps class A shares could eschew zombie bites in favor of voting privileges, class B shares could transition from zombie bites to voting privileges over a vesting period, and class C shares could get additional zombie bites.

I wonder if they’ll have to extend EBITDA to include decapitation, though….

Bom dia!

It certainly has been a while; in fact, the last time found me in Colorado, toughing out the ridiculously beyond-cold weather of the Rockies, but managing to have a great time, and laugh about it, nonetheless. Now, my better half and I are in her hometown of Uberlandia, Minas Gerais, Brazil. I’ll stop myself from turning this into a piece on all the goodness one can experience in a Brazilian minute, but let me just say this: You owe it to your taste buds to indulge in the ambrosial bliss that açaí is. Be liberal and have two bowls. It’s Dessert 2.0. I hope they serve it in the sweet by-and-by.

Okay, so on to the topic at hand - Windows Vista’s BitLocker, which encrypts all data on the system volume. At a glance, this might not seem like the logical place to discuss matters Vista-centric, but what follows is applicable to any instance where cryptography is required. As much as it is about BitLocker, it’s about much more. First, before continuing, you might want to take a look at the original draft of an article I wrote, entitled, “On Shifting ‘Windows’ and ‘Security’ from Less Antonymous to More Synonymous.” An adaptation, “BitLocker and the Complexities of Trust,” appears in the October 2007 issue of Microsoft TechNet Magazine.

Phil Zimmermann was kind of enough to provide some commentary for the article, of which I assume you’ve read at this point. Perhaps the most resounding proverb is, “Design as if making a mistake will cost someone’s life.” This brings me to my questions for you guys and gals. What do you expect out of cryptography? Are there any general goals you think it should achieve? What is “good cryptography” to you? How do you feel about open-source versus closed-source, in regards to cryptographic implementations? Are there any examples, that stand out to you, of where cryptography is being done the right way?

What I’ve come to find, over and over again, is a perpetual state of failure within cryptographic implementation; that is, when cryptography fails in practice, it’s almost never because of the cryptography itself, but, rather, its implementation. I attribute much of this to the fact that most developers responsible for implementing cryptography aren’t, well, cryptographers. I don’t expect them to be, either. However, many of these developers haven’t the knowledge to properly define the right threat model, let alone identify which cryptographic primitives they need in order to address that threat model. Mistakes ensue. To many of you, I’m sure this isn’t news.

I’m proactively working on ways to educate developers so they can avoid the subtle mistakes that leave huge marks. There’s a long road ahead, but the dividends are grand. Besides, cryptography is usually the strongest link in any security system. Why all the lax implementations? Shouldn’t we expect the same strictness in implementation that’s put into design?

Cryptography has, arguably, the best track record out of all the other aspects of security; it’s time to do a better job at reflecting this in practice. Something’s wrong if cryptography can’t reach its fruition in practice. Developers are in dire need of something that cryptographers have, so we need to bridge what is still an uncomfortably large gap between the two.

With our systems like loose slacks, we have a tight belt, yet we can’t seem to put it on right.

I’m all ears. Well, eyes. I’m all eyes.

(Names, companies, and identities have been changed to protect the innocent.)

I recently got sucked in the dark, black vortex of linkedin, and it got me thinking about my life. Linkedin is a very interesting social networking website because, quite literally, every person I have worked with in the last 10 years of my professional life is on it.

It is quite a stroll down memory lane, as I add one person, then remember, wow, I forgot about that person, hmm, I wonder what they are up to, lets add them too…. All in all, it is fun experience that is almost like a high school reunion, but for an adult professional life.

One of the items that caught my eye recently was, a title a, link of a link, had, roughly:

Catalyst Team Organizer -
Create innovation at Joe Blow Company.
Encourage interaction between departments.
Demonstrate innovation case studies to Executive Team for approval.

I would suggest that by definition, Corporate Mandated Innovation, defines that a Corporation will never be innovative, and is currently not innovative. This reminds me of a scene straight out of Catch-22. Here is the famous quote from the book:

There was only one catch and that was Catch-22, which specified that a concern for one’s safety in the face of dangers that were real and immediate was the process of a rational mind. Orr was crazy and could be grounded. All he had to do was ask; and as soon as he did, he would no longer be crazy and would have to fly more missions. Orr would be crazy to fly more missions and sane if he didn’t, but if he was sane he had to fly them. If he flew them he was crazy and didn’t have to; but if he didn’t want to he was sane and had to. Yossarian was moved very deeply by the absolute simplicity of this clause of Catch-22 and let out a respectful whistle.
“That’s some catch, that Catch-22,” [Yossarian] observed.
“It’s the best there is,” Doc Daneeka agreed.

So here is the Corporate version of Catch-22. If you are innovative, no one needs to tell you to be innovative, you just are. If you are not innovative, you cannot say you are working to be innovative, because by definition, the very act of deciding to be innovative is not innovative. Ahh, this is also some Catch-22, maybe not the best there is, but close….

When I started using Unix seriously in 1998, there weren’t a lot of options for getting on the web. I’d happily used Opera on Windows at home, after Netscape’s rather disappointing version 4, but Netscape Navigator (and not the whole suite) was clearly the best option when I switched to GNU/Linux full-time at home later that year.

I followed the Mozilla project with interest and finally switched away from NN4 to one of the Mozilla milestones around 0.9.1. This was an improvement, and not only because Mozilla tended to be higher quality but because it supported more sites more effectively.

Firefox didn’t impress me when it first arrived. Removing useful features altogether still strikes me as deeply silly. I remember thinking that Firefox’s vaunted slimness and lower footprint would go away when I had to install a dozen extensions to get back the features I used every day.

I haven’t thought that for years. I switched to Firefox around its 1.0 release and consider it a fine piece of software. Sure, there are problems, but I remember using Netscape 2 betas. Web browsers have come a long way in reliability and utility since then.

Firefox may be my most heavily used application, if you don’t count X.org or the command line. It’s served me well, and I’ve recommended it highly to countless others. Thank you to all of the contributors to Firefox and Mozilla’s projects. Even ten years ago, I had no idea how valuable this work would be.

I have been meaning for a while to write a post about how mentors have helped me to achieve my goals and dreams, but I have been so busy recently, that I put it off. This week I came across the video of the last lecture by Carnegie Mellon Professor Randy Pausch, who is dying from pancreatic cancer, and I was inspired enough that I figured I should write this post and tie it into my response to the video.

First, I would highly recommend that everyone and anyone watch this video. It is truly an inspirational and powerful video. One of the things he mentions in the video is, “How to Get People to Help You”. He mentions five points:

1. You can’t get there alone, and I believe in Karma
2. Tell the truth
3. Be earnest
4. Apologize when you screw up
5. Focus on others, not yourself

He also mentions, “Brick walls let us show our dedication”.

I won’t focus on everything he says in final speech, but I will get straight into how this relates to finding mentors in life. My first great mentor was Dr. Bogen, who I met at Caltech. He took in an interest in me when I was starting to form into a man, and, without a doubt, changed my life forever.

I remember doubting for quite a while if I was really intelligent or not, and then thinking that here was a guy that was in school until he was 37, a neurosurgeon, and a professor at Caltech, who found me interesting enough to talk with me every Friday from 8PM until 2 or 3 in the morning for a few years. Maybe I was smart? We talked about Philosophy, Math, Psychology, Consciousness, Religion, the Stock Market, Computers, Artificial Intelligence, and Bonsai Gardening, etc.

These talks and his wisdom allowed me to dream beyond what I thought was possible. He also introduced me to other powerful mentor/guru types. He also spurred a life long interest in the brain and psychology. Even though he passed away a couple of years ago, I still remember his words and advice almost like when Luke Skywalker hears the voice of Obi-Wan Kenobi. “Noah use your brain, don’t turn to the dark side…..”. I am also very glad that I got the chance to tell him that I loved him before he died and that I got to say goodbye when he was in the hospital. It gives me great comfort that he knew that before he died. I still remember that day very clearly, I grabbed his bald head, like I now grab my son’s head now, and let him know how much he meant to me.

I have met other different types of mentors, as well, in my journey through life so far. Another interesting person who I met at Caltech was Titus Brown. He has been a much different type of a mentor, but I have been asking him questions about computers and programming since 2000. He was the voice that kept gently whispering in my email…”so why don’t you try Python?” I can honestly say that if it wasn’t for Titus, I would never have touched Python or considered it. God knows what language I would be using at this point.

In March this year, at Pycon, I had dinner with Titus, who I hadn’t physically seen in a while and I met Shannon Behrens. We ate some barbeque and started talking about Vim, LDAP, Python, etc. Before I knew it, Shannon turned into a mentor. His knowledge of Python is so vast, and his attitude is so humble, that he has made a huge impact on my knowledge of Python. As I have mentioned in many previous posts, Shannon is my hero. He is one of those rare people who will quite literally tell you anything he knows, and will not judge you for asking a stupid question, for this, he is my hero.

I have also met some incredible mentors in the various jobs I have had. What is interesting, is that each mentor has a different method to access their secrets, and different ways of teaching those secrets. In some cases I have even had “unwilling”, or “hostile” mentors. A “hostile” mentor is someone who is scared to share what they know with you because they are insecure. A “hostile” mentor is a bit like someone who has turned to the dark side of the force. They feel that they can only succeed by working really, really hard and not sharing information with people or giving them false information.

It is still possible to learn from these dark lords, by just seeing through the insecurity and focusing on what you can observe to be truth. Yes, a dark lord can still be a mentor.

I met a really impressive, good, mentor at Disney named Greg Neagle who taught me many cool OS X tricks. I met another extremely powerful, yet incredibly humble mentor named J.F. Panniset, who is now in charge of engineering at a A52, while I was working at Imageworks. JF was scary smart, in particular, because he can write code, is an expert Video Engineer, knows Film production and Post Production inside and out, is an expert sysadmin, AND is about as approachable and humble as they get. All I can say about A52 is that they lucked out big time getting JF. He is a one in a million catch for a company. I was very upset to have had to leave Imageworks, but my wife and I moved out to Atlanta to start a family.

When I got to Atlanta, I was lucky enough to take a couple of classes at the Big Nerd Ranch and I become friends with Aaron Hillegass, the owner. He has been a great help and a very kind mentor as well. I am in a book club with him and we are currently reading Information Theory, Inference, and Learning Algorithms.

So, what does all this mean and why am I bringing it up? There are mentors and gurus behind every bush. It is very easy to tell a powerful mentor. The more powerful the mentor, the more freely they give information, help and guidance. I am now under the belief that there is a direct correlation with a person’s mental power and their ability to mentor. The greater a mentor’s power and wisdom, the greater they can give. After all how much effort does it take for a flood light to light up a dark backyard, compared to flashlight? These great wizards can spare the beams.

If you want to get better at something, you need to find mentors and find out how to get them to take an interest in you. In some cases it make take quite a while for a good mentor to think your serious. I would refer back to the quote by Randy, “Brick walls let us show our dedication”. If you really want to learn from a good mentor, then if you put in the work they will eventually help you, unless they are a dark lord, but those guys are easy to catch. They say things like, “Old age and treachery will beat youth and skill, every time.”

The soon to be released “Spotlight on FOSS”, video podcast Pilot episode, features Mark Shuttleworth. (We should have a date announced for release very soon). He gives a tremendously, moving talk that is on par with the Randy Pausch talk. One of the goals that Jeremy and I are trying to achieve by doing this podcast series, is to take these mentors of the Free and Open Source Software world and to expose their thoughts, ideas and passions to a larger audience. I hope people take the time to watch our first show, and I am very excited that it turned out to be this powerful.

In closing, I am also on the lookout for a new mentor to learn new things. I am particularly interested in learning more about artificial intelligence in the next few years. If you are a guru, and are interested in taking an interest in me, I would love to hear from you. I am currently looking for another mentor like Dr. Bogen, although that may never happen again, as he was truly one of a kind, or one in a billion. Also, if you have Guru/Mentor stories, I would love to hear about them. Finally, all of the mentors mentioned here are mine, all mine, so get your hands off them!

I first used Subversion about five years ago while writing my second book. It was an early milestone, but it was easier to compile, install, and configure than CVS was, and it was much more powerful than RCS. Since then, I’ve used Subversion to host every book I’ve written or edited. (See The Making of BSD Hacks.)

Though distributed version control systems are gaining in popularity (and though I use SVK atop Subversion), Subversion is still tremendously useful. Try being a F/OSS developer without knowing how to work with a project in Subversion, for example.

The preponderance of support with free hosting providers for F/OSS projects, as well as the quality of documentation and implementation, mean that I spend a lot of time working with Subversion. I’ve rarely had any trouble (even as far back as milestones 19 and 20). My work would have been much more difficult without this project, so thanks to everyone who’s contributed in any way.

In my career, I’ve been paid to program at ten different companies. Of those companies, only two of them have taken computer related security very seriously and three have had serious security breaches. There is no overlap between these two groups.

Of the three security breaches, two of them were known security issues that had been brought to the attention of management but management chose to ignore them. One of these caused serious financial harm¹. Due to the nature of the problem and management’s reluctance to discuss it, we couldn’t determine the exact amount of damage, but between known financial losses and the cost of responding to the incident, I would conservatively estimate that we lost at least $100,000 and possibly up to a quarter million. Had we fixed this problem before it occurred, it only would have taken two or three days of developer time. Given the relatively small cost of fixing the problem, why didn’t it get fixed?

Rael Dornfest and Brian Aker invited me to an RSS meeting way back in 2001, so I’ve known about the value of syndication for a long time (at least before it was super popular). Jon Udell’s Practical Internet Groupware made me realize the value of URIs as identifiers as well as the utility of alternate views of information.

I never found a newsreader that I liked, though. I tried, really tried to get into NetNewsWire during my brief flirtation with Mac OS X. I thought Sage for Firefox was useful… but it never really stuck with me.

I despaired of keeping up on the news of the day with anything but a careful list of bookmarks in Firefox until I found Akregator. Because I already use KMail, the interface was sufficiently familiar (which bothered me about other feed readers), and it’s just configurable enough that I can update feeds once a day, then disconnect from the network (though objects embedded in feed entries need an active Internet connection).

The highest praise that I can give Akgregator is this: I never considered a feed reader worth using until I found it. Now it’s a primary application for my research and work. Thank you to all contributors to the project!

Even using the right tools, in the right way, a software project can still get into trouble. One of the most pernicious ways to fail is over-specify everything up front. As the “Lean Software Development” movement has documented, well-intentioned people often add risk to their projects when they make hard decisions too early - before any research to identify any supporting facts. The best practices are Adaptive Planning, and Just-in-Time Requirements.

Another way to fail is to allow these requirements to fall into your lap by themselves. This post explores why embracing these deceptively easy requirements still adds risk.

Scott Ambler’s The Discipline of Agile on DDJ deconstructs a myth surrounding agile development, namely the fiction that agile development or XP is for undisciplined cowboys. Perhaps my favorite part of the article is the checklist at the end which helps identify cowboy coders and actual agile teams:

• Take a test-driven approach to development.
• Work very closely with their stakeholders on a daily basis, and the stakeholders have active roles on the team.
• Produce working software on a regular basis and can show a clear track record of doing so throughout the project.
• Work in an open, collaborative, and self-organizing manner.

In Are Authors Technological Poseurs?, JRuby hacker Charles Nutter suggests that “Good authors do not have time to be good developers.”

There are plenty of counter-examples, but I think the number will end up in the range of several dozen, while the number of great developers in the world is easily hundreds. (Ohloh had over 60,000 F/OSS developers identified the last time I looked, so the top 10% could be 6000 people.)

The nature of the skills are different, though. Writing’s less unambiguous than code, where at least you have a chance of independent and repeatable verification through technical means. It’s not a skill you can practice much on your own into the dark hours, because the purpose of communication is all important (at least in technical writing).

If it takes a year to write a good technical book (and it usually takes at least a year to write a good technical book), how much brilliant code could you produce in that time? Maybe it really is the case that to master a subject, you have to work at it full time — and while you may be good at both, you can only master one. There’s just no time to do both.

Qtopia has just undergone a major new release, adding such features as the Safe Execution Environment (a kind of sandbox for native code), strong support for WiFi, and integration with the WebKit browser engine. It’s a lot to give away for free. Based on Trolltech’s Qt graphics libraries and supporting C++ development as well as a JVM, Qtopia is a graphical environment found in a huge number of devices ranging from the Motorola Razr V8 to automobile navigation systems and medical equipment–and of course, Trolltech’s own Greenphone.

I talked to CTO Benoit Schillings, who cited fairly stock reasons for going open source: development energy comes from a community of developers and users, and ultimately there’s more business in serving a thriving environment of new applications and features than in holding onto secrets.

He mentioned the ubiquitous address book as a mobile feature that has hardly changed in years and is shut off from innovation because it can’t be touched by application developers. When the whole platform is open, new blood flows to these atrophied parts of the system.

There appears to be a general culture in IT, partly, but not completely, because it is male dominated, that it is bad to say the wrong thing, reveal your weaknesses, or be an “idiot”. I have been an Engineer for about 10 years now, and I have felt it from the beginning. When I first started learning Unix/Linux at the beginning of my career, I felt stupid quite a bit, and was afraid to ask questions as there was a culture that scoffed at “stupid questions”. Why don’t you read the manual, “idiot”?

When I transitioned from working as a sysadmin to working as a Video Engineer in Film and Television, I felt the same way. By that point, I had done enough in my life that I wasn’t as deterred to ask stupid questions, but I do remember several times people telling me, you should know this, why are you asking this question?

As I transitioned from working strictly in Film as a Video Engineer/Systems Engineer, to working just as a software engineer, I have often felt the same way. Maybe I shouldn’t let someone know that I don’t know everything about Python or programming? What if people think I am an “idiot”?

In the movie, “The Edge”, there is a great quote that directly applies to any Engineer, “Most people lost in the wild die of shame. They didn’t do the one thing that could save their lives –thinking”. How many potential skills or dreams die because of our shame? How much quicker could people learn if they were able to act like an idiot at some new skill they are learning, and truly learn it the way a child learns.

Now that I have some perspective from working as a Systems Engineer, a Video Engineer, and a Software Engineer, I will tell you that in each industry, I have had someone tell me that the Engineers in my previous field were “idiots” because they didn’t do “X”. For example, when I was a Video Engineer I had a couple of people tell me, “…you see Systems Engineers aren’t real Engineers because they can’t read a line drawing.” When I was a Systems Engineer, I had people tell me that Software Engineer’s aren’t real Engineers because they don’t really understand how equipment works. Since I have been a Software Engineer I have had people tell me that Systems Engineers aren’t real Engineers because they can’t program.

This situation I have described is a classic case of the “Observer Bias”, of course. From Wikipedia, “Observer bias is error introduced into measurement when observers overemphasize behavior they expect to find and fail to notice behavior they do not expect.” When you’re an engineer in one field you notice that all the people like you are smart and know how to do what you do, but strangely everyone else is an “idiot”. This critical scientific fact is the exact reason why all engineers should do something where they feel like an “idiot”, as it gives them true perspective and allows them to grow.

My piece of motivational advice is to ignore your inner feeling of shame as an engineer. Attempt to do something new, express a controversial opinion, invent a new technique or technology, learn to program, or learn a new language or skill. Step out of your comfort zone and do some activity where you are perceived as an “idiot”. In this sense, it is really important to be an idiot. Being an idiot, means losing the sense of self-criticism that is often found in programmers, sysadmins and engineers and truly learning. Being an idiot is important!

Hear me roar. This interview has me answering why I began the Women in Technology series, my hopes and dreams for it, and how I’ve already benefited from doing this. Hope you enjoy it!
(Also, to the many who have contacted me about this series, thank you so very much.)

Software development is rarely easy. Even though some open source pragmatists suggest that Linus’s Law makes traditional schedules and planning obsolete, I remain skeptical. (I’ve worked on a few pieces of F/OSS in the past decade; I believe that I’ve earned a right to be skeptical.)

In the era of Web 2.0, it appears that Slashdot has “Jumped The Shark”. The question now, is when did this happen? I remember in the early 2000’s, Slashdot was THE geek website, but something has changed and it appears they have lost the magic. I decided, today, to take them off of my RSS Reader, as I find their stories trite, boring and dare I say, irrelevant to IT?

So the questions of the day are:

When was the exact day, and, what was the exact story, that caused Slashdot to jump the shark?
Who did you replace Slashdot with in your RSS reader?
What is your best bet to the cause of their demise?

UPDATED SPECIFIC REASONS FOR TAKING SLASHDOT OFF OF MY RSS FEED:

Why I think Slashdot has Jumped The Shark?

Please note, comments are now closed as I can’t respond to them all.

Due to an overwhelming response by a fanatical few, I have decided to post in detail, why I no longer find Slashdot interesting, or relevant, and why I have decided to take them off my RSS feed. This is just my opinion, and I am sure each and ever person that uses RSS at some point has made the same decision about another website. If you get angry easily you might not want to read the rest of this post:

1. I am 32, and I have outgrown any interest in the usual stories that appear on Slashdot. A specific example is this story, about a “Coup” attempt in an Apple Underground User Group. I have absolutely zero interesting in ever reading something like this for the rest of my life.

I felt violated reading something that stupid today, and I will admit it may be because of my age, and due to the fact that I now have a wife, a kid, and a life.

2. I never really participated in the community much, I only read stories, and as I mentioned the stories are getting very bad.

3. I am bored of the terms, “Troll”, “Trolling”, and “Dvork”, they make my skin crawl just like it would make my skin crawl to hear someone use the terms, “Your playa hatin”, or “Give me the bling, bling”, or “Far out man”. These terms are so commonly used on Slashdot that it is impossible to avoid them and the only possible alternative is to never read anything on Slashdot.

4. Slashdot played an interesting role in the early 2000’s, as it was a human funneled aggregator for news stories. As technology has progressed, a different model of story submission has started to thrive and it does not require a select group of humans to filter which stories are good and which stories are not good. I believe the model that Digg, Reddit, and DZone use are far superior to the method of story submission and approval that Slashdot uses.

I feel that the method that Slashdot uses is a dying art, and the very poor quality of stories suggests that the human element responsible for editorial content is either very young, or not very good.

5. RSS Readers have changed the way people read technology and other news, and it has caused people to stop having a “home page” anymore. People now have the ability to create their own custom filters and get their own news in anyway they see fit. The role of Slashdot as the only aggregator for IT news has ended. Given a choice, I would much rather use RSS than the editorial process at Slashdot to get news stories.

6. As evidenced by the responses today, it is quite impossible to have a discussion with some outspoken members of the Slashdot community. It would take a large amount of patience, time and energy that I, frankly, am not willing to part with, to discuss why I am “liar, hypocrite, troll, etc”. It is just not interesting to me, and even if it was I would probably need to quit my job just to respond full time to the complaints.

If you feel the urge to yell at me, please remember this is just my opinion and I am very sure other people feel very differently. I do not have the time to discuss the matter anymore due to work and personal obligations, but thank you to the people that did respond. I did learn one very important lesson today though, some opinions are probably best left unsaid, as the effort required to explain it is not worth the cost of expressing it.

In summary, I do feel Slashdot has “Jumped The Shark”, I won’t read Slashdot anymore, and I am sorry I brought it up.

-Noah

I have been reading the CEO of Sun, Jonathan Schwartz’s blog, lately and it has some great material. I just love the fact that many CEO’s blog nowadays, as some of it is quite interesting, and bold, material. I suppose I would even go so far to say, that if I read a CEO’s blog and it wasn’t good, I wouldn’t invest in their company, as I would have my doubts about the leaders intellect and authenticity. On the other hand, if I read a CEO’s blog like Jonathan’s, I would be very interested in the company.

One of the recent posts that Jonathan made was in response to a flame by Linus. In his response to accusations, by Linus, that Sun was being disingenuous about truly open sourcing its ZFS file system, he mentioned that not only was Sun going to open source everything, but that they were going to do it under GPL3.

I think this is a truly brilliant strategic move by Sun, as it raises the bar for GNU/Linux and Linus who is very adamant about his dislike for GPL3. I do see there being competition between OpenSolaris and GNU/Linux in the coming years, and I do think it is a good thing, as competition is what drives innovation. I also see an interesting dilemma for GNU/Linux as OpenSolaris will be able to use GNU/Linux code, but GNU/Linux won’t be to use OpenSolaris code. This gives OpenSolaris a huge competitive advantage and might give them a temporary head start in the head to head competition of their operating systems. Ultimately, it seems like GNU/Linux might have a very large problem on it’s hands if it stays with GPL2.

I am not making a value judgement on GPL2 vs GPL3, but I will say from a strategic standpoint it appears to be checkmate for Sun on this round. Sun has some incredible virtualization and storage technology and it if enters the open source arena, “just right”, it could capture a massive amount of market share with this strategy.

Complaints by people in the GNU/Linux camp can easily be rebutted as Sun can claim the “moral” high ground as they support what the FSF recommends. I am quite interested in what happens over the next few years. There are quite a few possibilities, some of them admittely far fetched, but interesting to discuss regardless:

* OpenSolaris comes into the FOSS market goes GPL3 and takes significant market share as it has the best GNU/Linux technology and Sun Technology. GNU/Linux tries to fight back and switches to GPL3, but the damage is enough that they become the second preferred FOSS Operating System behind OpenSolaris.

* Nothing happens. Things pretty much stay the same and OpenSolaris doesn’t really make much of a positive or negative impact.

* GNU/Linux and Linus seeing the “end game”, quickly switch to GPL3 and cancel the advantage Sun might have before it happens. It then crushes OpenSolaris as it takes all of its good technology and the huge market share it already has and makes OpenSolaris marginalized.

* GNU/Linux becomes the preferred choice for companies that don’t want to become tangled in GPL3 and OpenSolaris becomes the preferred choice for a commodity operating system that runs in a data center. This leads to a significant loss in market share for GNU/Linux.

* Other things….what am I missing?

Comments?

Recently on this blog, I wrote You Have the Right to Read Your Accuser. In this, I argued that any software with substantial risk to harm your life or liberty must be open source. I specifically mentioned some breathalyzer software that people were fighting to see the source code of. Though the Florida legislature and many regional prosecutors obviously feel that the breathalyzer company’s rights were more important the individual rights, the New Jersey Supreme Court ordered that source code to a popular breathalyzer be revealed. The results, if you’ll pardon the pun, are breathtaking.

Even though I wrote a very popular post about the Zen of Mac,
to show that I can be fair, I thought I would write about what I hate about Apple.

I don’t think Apple has the perfect Operating System or company even though I use OS X as my preferred desktop OS and I love it. They could still improve on things, so on that note these are things I hate about Apple.

1. Dump the silly DRM stuff COMPLETELY, not just for part of your library.

DRM just insults our intelligence. Apple is supposed to be ahead of the curve and creating a user experience that is better than any other desktop OS. “Authorizing” my music when I reinstall my OS is extremely obnoxious. Have some guts and say no to DRM period! No Mac users want DRM, so why are you providing a service we don’t want. Sounds like another OS we have all heard of…tread carefully!

2. Free and Open Source Software package management system doesn’t exit!

Your core OS is UNIX, yet you STILL can’t figure out how to integrate a decent package management system for FOSS? Huh, I don’t get it? Integrate Fink or Darwin Ports, or copy debian, but get with the program, it is embarrassing!

3. Regular Commercial Software Package Management doesn’t exist…i.e. the “uninstaller”?

Again, why is it so hard to uninstall or reinstall commerical software on OS X? There is at least one robust open source packagement tool Radmind, that does this. Are you telling me that figuring out a regular package management system is that hard? Often you just need to drop a bundle inside of your Applications folder or delete it from your Applications folder, but many 3rd party applications leave a trail of garbage. Make them conform to a package management system so we can get rid of their junk!

4. Locking the iPhone.

Unless this was a very shrewd marketing campaign for the iPhone, what did you think was going to happen when you released the iPhone without an SDK and locked it to ATT? Lets have less “locking” and rules with things we pay tons of money for. Just release the friggen SDK already, even Microsoft has an SDK for their phone.

I don’t want to hear all of the excuses either, like Safari is an SDK. They are all lame! Just do it already.

5. Don’t break UNIX behaviors that should work.

Now why doesn’t autofs work again? Is it because your pushing AFP? That is nice and all, but I like NFS so keep the tinkering off of autofs and make it work again! I shouldn’t have to buy an Open Directory Server to serve automounts when I could just use autofs. This was either a real poor design choice, or a somebody from Microsoft was hired to work on autofs :)

By, the way I think I might be the only person in the world who wrote a how to article on getting NFS to work with GNU/Linux and OS X via Open Directory, so I know what I am talking about: Open Directory Part 3. Everyone else uses AFP, but you don’t have to. Apple just doesn’t publicize it!

6. Now why can’t I write cross platform applications with Mac Developer Tools?

Cocoa is really great, but it is quite a daunting task if you need to write a tool that works on OS X, *nix and Windows. I can understand Cocoa not working as that framework only exists on OS X, but why don’t you have a cross platform development environment? Why not work with one of the dynamic languages like Ruby or Python and build a toolkit that is OS X like, but builds applications for all platforms?

7. Applescript

Just dump it already. Seriously, it is way past its prime. Replace it with a modern scripting language like Ruby or Python. A lot of the work has already been done for you.

8. Why does OS X Server require a running GUI?

Take a hint from Ubuntu and have the option to not install and/or run the Window Manager. Your starting to get into big leagues with cluster computing, XSans, and quad core 1U servers. Why oh why do I need the GUI running sucking up tons of CPU and memory when it is a file server, or a render node? You need to lose your server OS beer gut and get into shape!

After reading one of my favorite blogs this morning, it got me thinking about the Zen of Mac. In the article that I linked to, Shannon mentioned that one of his hangups is not using OS X because it isn’t Open Source. The topic of smart people with hangups is very interesting actually and I would love to talk even further about it, but I am going to talk specifically about the Zen of Mac in this post.

Back in 2002, I was a systems administrator for the administration building at a Caltech, and I was solely responsible for providing support for Dr. Baltimore. Dr. Baltimore is smart. He received a Nobel Prize in his 30’s, he went to MIT, and he was the President of Caltech at the time. He is a mac guy, and that is partially why I was hired to do that job, as I have been into Mac computers for quite some time.

For those of you that remember, it was a huge deal to switch from OS 9 to OS X. It was my responsibility to design and build his OS X laptop and make sure that it was an easy transition. When the time came for me to actually deliver the laptop, things worked reasonably well and were intuitive as he would expect them to be. The punchline of this story is when I finally decided to show Dr. Baltimore the terminal. I think I mentioned something like, “One of the nice things about OS X is that it has a terminal and you can do nice things like……”.

The look on his face was priceless. I don’t remember the exact specifics of the conversation, but it was roughly, “Why would I use a terminal, it is a mac, that is the whole point!”. That taught me quite a bit, as here was one of the most successful and intelligent people on planet earth and he “got” Mac. He didn’t want to think about his computer as his thinking time was spent in other areas like making sure Caltech continued to be one of the top Science Universities and doing research on curing AIDS.

That is the Zen of Mac. You don’t think, it just works. As software engineers, systems administrators, or people that are very technical, it can be difficult to just not think about your desktop computer. The reality is that you are more productive on solving your other problems when you don’t think about your computer, it just works.

So as far as I am concerned, even though I work literally all day on linux machines from a shell, my desktop experience is OS X because it just works and I don’t have to think, my thinking can be devoted to solving my problems. Finally, for those people that are switching from a Linux desktop to an OS X computer, I would give this advice. Don’t think about how you used to do it on Linux, just forget what you know and try things out. OS X is designed to be intuitive and effortless. Fighting it to make it do what Linux does is not the proper approach. You must submit your will, relax your mind and float downstream on the white glow of the macbook pro monitor. Once you surrender your desires, and realize that desire leads to suffering you will truly appreciate the mac.

The discussion turned, as it occasionally does, to licensing and philosophy. (This is what happens when you hang out with smart people who also care about the subject.) I may have surprised a couple of them by saying, fire-breathing zealot for freedom that I am, that I don’t particularly care about the mythical software as a service loophole.)

Why not?

My current camcorder is over 5 years old now and I’m looking to replace it. I’m hoping one of you can point me in the right direction. Specifically, I’m looking for a camcorder which uses flash memory, writes files in a format which iMovie can directly start manipulating, has a crisp picture (which implies 3CCD, but not necessarily HD), and has a firewire connection.

I was looking at the Panasonic HDC-SD1, but iMovie 08 looks like it transcodes the files as it imports them. The reason that I wanted to go with a flash based camcorder is the ability to bring movies onto my computer more quickly than I have been able to with my current digital 8/firewire camcorder. But if iMovie has to transcode files as it imports, them, that doesn’t really buy me much. I suppose I could get the SD1, create an image of the files on the camcorder and mount it later to allow me to free up the flash drive in the camcorder, but that feels like a pain. I’d like flash as opposed to hard drive because of the ease of replacement if it goes bad and fewer moving parts. I want firewire because sometimes it’s nice to be able to record directly in iMovie.

So, 2 questions: 1) Am I thinking correctly about AVCHD and the SD1? and 2) Do you have any specific camcorder suggestions which roughly meet my needs?

The subject of “Non-Toxic Stimulation” has been on my brain for a bit now. I remember the subject being brought up a few years ago by my mentor, who passed away a couple of years ago. The best way to describe our relationship was to say that is was very similar to tuesday’s with morrie. One night Dr. Bogen described a friend of his that was in constant search for Non-Toxic Stimulation.

The definition for it is quite simple. Non-Toxic Stimulation is something that excites you but doesn’t kill or harm you. Toxic stimulation is quite easy to define as well. It is something that excites you but could kill you literally or figuratively. Examples of toxic stimulation could be smoking crack, driving a motorcycle down the freeway at 160 mph on your back tire, running with the Bulls in Spain, rock climbing without a rope, or fighting dogs. These are all optional activities that might leave you or others dead, maimed or in jail.

There are equally exciting Non-Toxic forms of stimulation, but they for the most part involve your brain. One form could be taking a class at Big Nerd Ranch and starting a Web 2.0 company. It could be finally teaching yourself higher math because it scares you and make you feel inadequate, but you know you can do it anyway.

The Non-Toxic form of stimulation I am engaging in currently is to teach myself Ruby as a way to practice test driven development. It is too hard to do test driven development in Python as I don’t want to slow down, but in Ruby, which is very similar to Python, I am not tempted to crank out some code as I can’t. It makes it much easier to practice doing TDD with Ruby and then apply that knowledge to Python.

In a now famous article by Joel Spolsky, he argues that you should never rewrite projects from scratch. To be fair, I’ve done this, but generally on open-source projects where I’m donating my time. I’m less worried about financial constraints or competitive advantage.

Aside from that edge case, I generally agree with Joel Spolsky. If something is a tiny project, refactoring is often trivial and if you want to do a rewrite, so be it. However, large projects are often dangerous to rewrite. But how did they become large projects? The vast majority of “large” projects I have worked on started out as small projects which gradually had features and cruft added. Interestingly, this is very similar to how agile methodologies work: build something small, but useful, in the first iteration or two. Always have working code and keep building on it. So the rewrite seems like a good idea because we’ve already shown we can create the project, right?

In comments on Sci Foo 2007, Tim O’Reilly wondered if giving Larry Wall a research job may have reduced his immediate practical concerns influencing the development of Perl 6.

I interpret the line of thought as “To what degree does the lack of a single well-defined problem to solve influence the concreteness or the abstractness of the solution and its delivery date?” (I won’t claim that that’s Tim’s line of thought, but it’s the question I’ve pondered this past week.)

I rarely see my colleagues, both in my work and in my hobbies. I spend most of my time collaborating with them, and one primary communication medium is email.

My personal mail server runs Postfix. Besides one small problem with a spam filtering message loop (mail forwarding to an account which rejects spam messages and includes the spam in the response is at least as evil as challenge-response), I’ve never had a lick of trouble.

Throw in server-side filtering with procmail or Email::Filter, as well as extended addresses, and I’m thrilled.

Then I discovered how to add just a couple of lines of configuration to make temporary, expirable addresses stunningly easy to create and manage and expirable, lightweight mailing lists, both of which require MTA administration beyond one initial configuration.

I never have to think about Postfix. It silently hums along, delivering buckets of mail. Thank you to all of its developers and contributors!

This blog will talk about two organizations I met with and their members–the Linux Mobile Foundation and the Open Solutions Alliance–plus other interesting people I met at LinuxWorld Expo.

A few years ago I was at a meeting where a salesman was trying to pitch his terribly expensive closed-source software to our company. Since security of our data was very important, this topic was raised a few times. I asked about how they encrypted their data. The salesman replied that since security was so important, the company created a proprietary encryption algorithm which was secure because no one knew how it was implemented. He seemed a bit flustered when I burst out laughing.¹

If you’re reading this blog, you probably have a technical bent and know that the vast majority of software out there has bugs. The larger the project, the more bugs. In fact, I’ve never worked on any significant (you know, large) piece of corporate software without known bugs. I’m constantly talking to friends who complain bitterly about long-standing problems with their systems. So why is closed-source software allowed to take a witness stand and accuse you of crimes when you’re not allowed to cross-examine it?

I’m running Firefox 2.0.0.6 on my (relatively) new (intel) Mac. Problem is that Firefox locks up several times a day. I found a forum which suggested disabling the anti-phishing functionality. Several people seemed to have benefited from said disabling.

I also found a Bugzilla report about the same behavior. It sounds like the anti-phishing thing can cause some problems during startup, but is probably not responsible for hanging during regular browsing. My problem is not on startup, but on browsing, so disabling anti-phishing probably won’t help me. But I’ve disabled it just on the off-chance that it will. If this hanging persists, I’ll either switch to Opera or Safari.

What are your experiences with FF on Mac? Crashes, hangs? Or is your world just peachy?

A couple of disparate threads have bounced around in my head lately, which makes me think that there’s some fundamental notion at work in the world.

Mitchell Baker and Matthew Gertner have had a brief back and forth over the nature of a public good (such as the Firefox web browser in specific, or free software in general). Mitchell’s position is:

A people-centered Internet needs some way for people to interact with the Internet that isn’t all about making money for some company and its shareholders.

— Mitchell Baker, Firefox is a Public Asset

Matthew Gertner wonders if corporate backing is necessarily, in itself, inimical to the creation and community-based maintenance of such a public good:

This isn’t about a small group of people trying to get rich. It’s about putting into place the most efficient overarching structure to achieve our common goals of choice and innovation on the internet.

— Matthew Gertner, More on Mozilla and Capitalism

The other thread synchronous in time comes from a comment Tim O’Reilly made a week ago:

I will predict that virtually every open source company (including Red Hat) will eventually be acquired by a big proprietary software company.

— Tim O’Reilly at [08.02.07 11:47 AM] in Microsoft to Submit Shared Source Licenses to OSI

Sometimes I wonder if the fateful 1998 meeting which gave birth to the term “Open Source” led the world of software freedom down a dark path. If the only way to get business to adopt the idea of embracing the power of communities to build software and ecosystems larger, more powerful, and more efficient than individuals could build on their own was to focus on economic principles, rather than the notion of the public good, is it any wonder that so many businesses seem to be indifferent at best to the health of those public goods?

To switch rhetorical metaphors, do you find it more likely that any given business would invest N% of its budget in energy-saving measures because it considers the investment ethically right on its own merits, or because it saves money and provides the basis for a nice, friendly press release?

Perhaps it’s inevitable that community-driven development, maintenance, and support will reduce markets for proprietary software up and down all of the stacks. Perhaps the most successful projects will have the strong support of businesses.

Do you want to rely on their goodwill to allow you to use, study, and redistribute software as you see fit? Are you willing to take the risk that they will encourage a healthy commons which allows you to use your data as you see fit?

I’m not sure.

Updated on 2007-08-10; corrected misattribution of Matt Asay to Matthew Gertner

Piers Cawley decided that his little language embedded in Ruby is a pidgin:

Casting the problem domain as the colonial power and ruby as the native language, it’s obvious that I’ve invented a pidgin language.

The nice thing about a pidgin (or a dialect, as Ben Scofield suggests) is that all of the power and syntax and semantics of the host language are available if you need them to express concepts that your little language cannot–without modifying the little language at all.

There you go; there are two good terms for describing the use of domain-specific language within a full-fledged programming language without co-opting a term with a perfectly good, existing definition.

Many workers in human services, she told me, are reluctant to provide data that would be useful to improve the services. She’d like to track homeless people as they move from one jurisdiction to another for instance, to provide better continuity of service and find out what works and what doesn’t. The agency staff are afraid that sinister forces within government will misuse data. While we have no lack of sinister forces in government, it appears that the people needing human services are more at risk of snooping by random staff people, facilitated by the awful security practices just mentioned.

I’m not surprised that employees would treat passwords as just one of the many random impediments they have to bypass each day to do their jobs. Given how many regulations reflect political grandstanding rather than life on the street, and how many well-meaning regulations outlive their usefulness, workers have to interpret the rules in a (shall we say) creative manner. I’m sure many employees in private industry get through the day the same way; it’s not limited to government. But an even deeper issue is at work.

But I don’t want to live in a world where the only thing the Internet is useful for, or effective at, or pleasant or fun, are activities where someone is making money from me.

— Mitchell Baker, The Internet and the Public Good

Kirrily Robert dissects a meaningless job posting for a Perl programmer.

This reminds me of a story Jim Shore told me about Fit. Developers in a company wanted to use the software, but their lawyers had grave concerns about the license. Eventually, the developers appealed to Ward Cunningham, who said that they were using it in the way he intended and he had absolutely no intention of bringing suit or other legal action against anyone who used his software appropriately. Even so, the lawyers saw that as an unacceptable risk.

The punchline is Ward’s final question. “You have to ask, do you work for your lawyers or do they work for you?”

Perhaps it’s time to ask that of HR departments.

As I read reactions from people to JT Smith’s Perl is Dead. Long live Perl. I see the usual knee-jerk claims that Perl inherently leads to unmaintainable code.

In my vast experiences in dealing with difficult-to-maintain code, I’ve noticed that nearly all of the messes I’ve seen had comments, documentation, and identifiers written in English. English is not an easy language to learn. It has inconsistencies (irregular verbs! homophones! homonyms!) and quirks (idioms! punctuation styles! possessive marks!) which make writing perfectly correct English–or even succinct and direct English–difficult.

There’s one good reason it’s difficult to produce correct and successful software from a full specification written at the start of a project.

Yet somehow it’s acceptable to allow programmers, presumably smart people with the ability to juggle small details, to write terrible, horrible, incomprehensible English but not comprehensible, concise, and correct code in languages which are orders of magnitude simpler than English. Hey, if you can use the pronoun “it” appropriately in English, Perl’s topic variable $_ should give you absolutely no trouble!

If (perceived) simplicity and regularity of the language were truly an important factor toward the maintainability of software, we should all be using Esperanto or Lojban to talk about our software. It’s not as if we don’t expect programmers to be able to learn new programming languages where appropriate.

Perhaps the true source of maintenance problems lies elsewhere.

Saturday, November 9, 2002 was my first Weblog entry, right here on O’Reilly Network. In 2005 I started using my personal Weblog Copia and soon I was posting exclusively there. On Copia I post on everything from music to literature to politics to technology, and I’ve long wanted to find a way to primarily post my tech writings here on O’Reilly Network, while having this aggregated with my other personae on Copia. This, “personal aggregation” as I call it, is one of the motivations for me to move from the PyBlosxom platform of Copia to a new one I’m building from scratch, with a few other co-developers. Bright Content (I don’t yet want to link to it because i think it’s not entirely ready for any attention) is a Weblogging system that embodies much of my long-standing interests in technology architecture relating to content.

But I’ll talk more about that project later. Meanwhile I’ll start the process of moving my technological Weblogging persona back to this Weblog, while ‘ll continue with other topics on Copia. It’s actually good timing for a different reason: after a separate one-year hiatus I recently renewed my agile Web column with the new article Introducing OpenSearch, a topic I chose in large part because of the resonance of OpenSearch with my philosophy of content technology. I look forward to continuing my conversations with you, through my articles and Weblog entries on this site.

JT Smith, president of Plain Black, the creator of WebGUI, and one of the unsung successes of using Perl in business, recently sent me this essay. He gave me permission to publish it in its entirety here.

(In the interest of full disclosure, the plush WebGUI octopus I have from YAPC::NA last year is one of the coolest pieces of swag outside of Hollywood, ever.)

c|net had a story up a couple of weeks ago entitled “Kids say e-mail is, like, soooo dead”. The story was about how kids were moving away from email and using instant messaging and social networking sites for peer communication. Maybe I know why. Or maybe I know why that could be such a temptation. The reason is simple. Email sucks. All email clients suck. No one of them has the feature set that I would like. Their handling of the email protocols can be atrocious, particularly IMAP. And don’t even get me started on having to interface with Exchange with anything other than Outlook. Also, you don’t know when someone has read an email (unless you’re using X.400, but that’s another story) or when they’re online. I can see why IM, social networks, and text messaging are gaining usage. Pownce, anyone? (BTW - I have a few invites left. If you’re interested, email me.)

UPDATE: The Pownce raffle is now officially closed. Thanks for everyone who emailed me.

My wife’s business has the need to make some sales online and so naturally I want to help. I know next to nothing about E-Commerce, so I started to do a little bit of research. It appears there are three options available to people currently in the market for a shopping cart that will process credit card transactions. There may be more options as well, but this is what I have found on short notice.

Option 1:

It appears to be relatively cheap to just host a whole website with a shopping card and credit card transactions built in. Google Checkout has links to a couple of complete solution providers that offer WYSIWYG admin panels and editors for somewhere between 30-50 dollars a month.

Option 2:

Integrated Solutions that you build into your existing framework/website. They appear to offer differing levels of flexibility, ranging from an API, to a link to another website that has a cart.

Option 3:

Something like Satchmo which is a “webshop” for perfectionists with deadlines. I found out about Satchmo through a fellow member of PyAtl who had some success with it.

Why does E-Commerce have to be so hard anymore? Maybe it isn’t that hard, and I am very ignorant. It seems like 2007 might be a good time for mere mortals and Mom and Pop businesses to start taking orders online. So, what is the best solution for minimal effort? Is there a solution that works well with Python, and/or Python Web Application Frameworks?

On one level it would be great to know there is a completely Open Source solution to a relatively simple problem. After all, why pay 50 bucks a month if you can easily do it yourself. I think VOIP is a good example of small business doing Phone Service themselves, and the same model might apply for E-Commerce. On the other hand, processing credit cards is somewhat risk prone, so perhaps there is a middle ground that still serves as a good price point for small business owners.

Scott Walters recently ranted on how Perl programmers tend not to perform OO analysis and design, at least when compared to Java programmers.

Setting aside issues of language design–though they’re important, I think there are other insights available–it does seem as if there’s much more literature available for developers who want to design large systems in Java than there is for developers who want to design large systems in Perl. (I operate from the assumption that the number of people who could write a program in Perl is probably the same as the number of people who could write a program in Java, if not greater, so I set aside questions of market size as well.)

Aristotle Pagaltzis suggested I repost my analysis here. In short, it’s all about the teachers.

At OSCON I attended a handful of the sessions that were sessions were related to web application development. I’ve compiled a list of features I’d like to see in the next web application development framework:

• Give me continuations. Let my request handlers check whether the user has authenticated and, if they haven’t, prompt the user and resume execution in the handler without losing state. If the user has requested to delete something, confirm the deletion first. I want this type of logic in the controller, not as a confirm() method in the JavaScript.
• Let me attach listeners to server-side models and update the interface when the data structures have changed. For example, given a list of favorite movies that is stored in memory or in the database, I want a <div> to be modified automatically when a favorite is added or removed.
• Keep the API simple. Give me little languages, DSLs, data structures or whatever the fancy name for them is. Let me focus on the logic of what I’m doing. Don’t limit these little languages to only configuration files — Let me write my HTML templates in whatever language I’m using throughout the rest of the framework.
• Make sure testing is a breeze. Let my tests “click” on <div>s and make assertions on the contents of elements.

Fortunately, there are a few frameworks that have some of these features already.

• Google Web Toolkit lets you write applications entirely in Java. The Java that needs to be run client-side is compiled into JavaScript. Designing interfaces in GWT is similar to the popular toolkits for desktop applications, and the testing framework seems solid.
• Jifty, a Perl framework created by Best Practical, includes a lot of the aforementioned. Templates and tests are all written in Perl in a mini-syntax, and they’re working on compiling Perl to JavaScript. It even has continuations.

Above all, I never want to have to write an <a href="..."> tag again. Giles Bowkett, in his HREF Considered Harmful talk, explained that it’s the modern equivalent of GOTO. I agree.

Here’s my wrap-up of the Open Source convention. I published an earlier blog on it as well.

Andy Lester is renown for his evangelism of technical debt awareness, and his talk at OSCON was full of rich lessons on improving your code and yourself.

Managing upward is a theme in his talk, but it isn’t his primary focus. Andy wields upward management as a tool which can be used to improve code quality, but it’s a topic that deserves significant attention as well.

“Talk in dollars. This is the language that management understands.” — Andy Lester

Nobody develops for Perl anymore, CPAN is too crowded.

    Jordan Henderson

Senator Dick Durbin has announced an online forum about broadband policy that started last night. Today’s news that iPhone sales are disappointing provides one illustration of the importance of this issue. While the device is overpriced, I have no doubt that a key drag on its uptake is the slow network AT&T has devoted to it. This would have been considered ludicrous in East Asia.

US Government, as we all know, is fleeing from openness as fast as its so-called leaders can run. At Tim O’Reilly’s open source briefing yesterday, open source advocate and version control expert Karl Fogel presented a case for recording and releasing all communications that go into making laws. Apparently the New York Times picked up Tim’s blog on the subject.

The briefing also presented open web APIs, open source hardware, and other examples of how the open source movement has spilled over its origins in free software. The most popular free software packages are still infrastructure: operating systems, languages and language tools, system administration packages, and so forth. But there’s no doubt that everybody is evolving in response to this powerful model for encouraging creativity.

I just read an article at Linux.com about the OS habits of Linux users. The author of the article asked Linux Torvalds about his habits and found he exclusively used Linux. Torvalds said, “I don’t use either [Windows or Mac OS X]. OS X is kind of pointless (pretty much anything it has, Linux can do better) and Windows offers stuff that I don’t much care about (mainly games — and I’ve got games machines for those).” Before I comment on this any more than I have, let me just say that I have been a near-exclusive Linux desktop user since 2001. I love Linux and I still think there are some areas where Linux dominates. However, to Torvalds, I say, “hogwash”. He can get around careful scrutiny by his choice of words “pretty much anything it has” and “Windows offeres stuff that I don’t much care about”.

But the spirit of what he’s saying seems just dead wrong. And it’s an attitude that’s pervasive among many Linux enthusiasts. Whether Torvalds has the zealotry I’m about to discuss is irrelevant. It’s just a launch pad for me to address this attitude. The attitude goes something like this. “Linux is an awesome OS. We’ve come so far in a short amount of time. We have everything anyone would need. And bling to boot. Linux is ready for the desktop. In fact, it’s ready to take over the desktop.” There is a lot of truth in what both Linus said and my characterization of the Linux zealot.

Truth 1: Anything that Windows and Mac can do, Linux could do. Notice my choice of words. I said that Linux could do anything that Mac and Windows can do. But the sad story is that Linux is not currently doing a lot of what Mac and Windows is doing. Please, please, please someone show me wrong on this! Please show me a DVD authoring application on Linux that is as easy as iDVD is on Mac. Or a video editing application as easy as iMovie. Please! Yes, I know that there is wine and you can often get Windows apps running on Linux. But 1) it’s hit or miss and 2) most of the apps that I’ve gotten to run in wine look…..let’s just say “bad” to sound polite.

Truth 2: Linux has made considerable strides in the past few years. The desktop looks spectacular (both kde and gnome). There are tons of top notch applications available for free download. Hardware recognition and support works better than it ever has. I don’t think this one needs to be shot down, so I’ll leave it as it is.

Truth 3: Linux has the bling. In my opinion, Linux is actually winning the bling war. Just check out Beryl/Compiz/Fusion. It rocks. For bling, it is (my opinion, again) unrivaled. And there are some pretty cool productivity enhancements, too. But let’s not confuse bling and even cool productivity enhancements with a usable desktop (not that Linux has an unusable desktop). They aren’t necessarily the same thing. Just because Linux has bling doesn’t mean that its applications are well integrated with one another. Or that the applications work well on their own.

The point of this rambling is that all three of Mac, Linux, and Windows do some things well and other things not so well. Personally, I don’t like getting on Windows. It feels square and wooden. But it does some things pretty well. And I just bought a Mac a few weeks ago and I’m really enjoying it. I’m not at all ready to say that neither Windows nor Mac is really competitive with Linux. Conversely, I’d say that each of them spank the other two in some areas and don’t do so well in other areas. So, can we please let the zealotry die? Please?

Update: I’m closing the comments on this blog post because of recurring blog spam. If you want to carry the discussion on, please email me and I’ll post a new entry.

PyAtl is switching to Plone as time is short and we want a CMS that everyone can edit. A few people are are working on getting Plone configured and we are looking at using the gmail authentication plugin and the forum plugin.

All of this has got me thinking? What is the story on Plone and/or Zope? There is all this talk of Django/Turbogears/Pylons, etc., but what about Zope and specifically Zope3 and Plone? In the younger crowd you almost never here anyone talk about anything related to Zope and I wonder why?

So, what is the dope on Zope? Can you turn Plone into the next myspace? How hard is it to learn Zope3?

I’ve been listening to The Linux Action Show podcast for a few months now and really enjoy it. I’ve tried other Linux podcasts, but they seem focused on the noob level. Can anyone recommend a good Linux podcast for experienced Linux users?

I’m pretty good at Perl, so I hear a lot of comments about programming language syntax. Many of them are fluff around the old argument that “I don’t like to read punctuation.” Many of them bring up the silly idea that an ideal programming language syntax should be so intuitive that people who’ve never used the language before should be able to understand programs written in the language.

That’s a ridiculous argument.

It’s been some time since I don’t follow closely Lua development. But I try to keep updated with what’s going on. The announcement of LuaPOD 0.1 caught my attention (due to the gathering of three technologies I find quite interesting).

In the spirt of health competition, and because I selfishly want to know what blogs to fill my Google Reader account with, what are the Top Ten Python Related Blogs? Also, how many RSS feeds should I take? What is manageable?

What are the Top Ten Python Blogs you read either “old school” or through an rss feed? Here are ten I happen to read in no particular order but feel free to order yours by whatever political statement you choose:

Tornado of Testing Titus
Dynamic Diatribes of Doug
Glitsy Glyf
Enigmatic Ian
Jazzerific James
Jewels of JJ
Jonathan’s Juleps of Joy
Ricky the Tricky Raccoon
Natural Selection Niemeyer
Jumping Jeremy Jones

IBM announces an interesting initiative today that will make it easier for open source programmers and other small coders to put together a range of software, including Web Services. The initiative is just a subtle procedural change, but a welcome one for people with little time and tolerance for bureaucratic red tape: IBM’s wide range of royalty-free patents are now available without formal licensing.

I’ve found myself writing a lot of C code this year. I’ve come to appreciate the power of a good compiler, especially when trying to walk other people through compiling my supposedly cross-platform code on Windows with MSVC.

GCC may not always produce the fastest code or compile with the greatest speed, but it’s reliable, and once I have code compiling with GCC on Linux, I have confidence that it will compile for just about any other free Unix with GCC.

I’ve also come to appreciate several GCC flags and options. For example, GCC 4.x added a compiler flag called -fvisibility=hidden. When you build a shared library with this flag, GCC will hide all symbols not explicitly marked as visible. As Windows DLLs require export lists of all symbols visible externally, enabling this feature for non-Windows compiles helps prevent me from adding a new symbol but forgetting to export it.

I’m also a fan of -Wc++-compat, which gives copious warnings about dubious constructs which may choke a C++ compiler. Again, not all platforms have good C compilers available (mostly by disallowing the use of GCC), so making my code as clean as possible helps avoid a large porting burden later.

I know there are other good tools to analyze code, but for the cost of a few compiler and linker flags, I get a good cross-section of warnings that help me clean up my code–and a high-quality, cross-platform compiler for free.

Thank you, everyone who’s contributed to GCC.

Keep an untrusting eye on your LAMP servers — you don’t get 5 nines of reliability and robust support for hundreds of simultaneous connections without building up a little resentment for all that unpaid labor (say, in the form of license fees to the software’s proprietor).

I just finished How to Survive a Robot Uprising and thought I could do my part of saving humanity by sharing some tips from the book:

Destroy or disable exposed sensors (p. 99)

I’ve used a free software desktop since 1998 (except for a six-month flirtation with Mac OS X). It’s much easier to avoid proprietary software in 2007 than it was in 1998, and it gets easier every year.

If that’s true on the desktop, it’s more so on the server. My employer in 1998 (one of the top five computer vendors) had a Linux strategy best described by a division manager with unwitting irony as “Of course we were proactive about Linux! We were the first company to decide to wait and see what everyone else was doing!” Now that company not only has a Linux strategy, but it will happily sell and support you Linux servers.

If Microsoft’s stranglehold has loosened, is supporting Windows as important as it once was? Is it important at all? A recent discussion on the O’Reilly editors list may provoke some arguments.

I resisted automated mail filtering for a couple of years, figuring that the problem wasn’t too bad, and that I could always detect and delete spam with only a little bit of work.

Then my e-mail address spread far and wide in the credits files of a few software projects and dictionary attacks became cheaper and… then I took everyone else’s good advice and installed SpamAssassin. Now I skim a folder full of questionable mail for a few seconds a day and train false negatives once a week and don’t worry about spam as a user. (As an administrator of the mail server, I worry about the waste of resources, but that’s a separate problem.)

I wish I didn’t need this software, but I do, and it works. Thank you to all of the contributors to SpamAssassin!

Oh, and for everyone curious about my spam training aliases, they are:

alias learnspam='sa-learn --spam --mbox ~/Mail/spamtrain && sa-learn --spam --mbox ~/Mail/questionable'
alias learnham='sa-learn --ham --mbox ~/Mail/hamtrain'

As the U.S. Independence Day approaches, we can honor the shot heard around the world when the IT department of the state of Massachusetts declared a couple years ago they would adopt the Open Document Format.

Although many people inside and outside the state detected more than a whiff of anti-Microsoft sentiment in the announcement, it didn’t preclude the use of Microsoft products. (Not long after, a plug-in was developed–not by Microsoft!–to produce ODF from Microsoft Office programs.) But instead of adopting to public pressure and supporting ODF, Microsoft lobbied international standards organizations to adopt its own proprietary format as a standard instead.

Now the state has formally backpedaled, according to a posting by standards expert Andrew Updegrove. It has declared Microsoft’s OOXML as an acceptable format for state documents.

Dueling standards are nothing new, but it’s not in the public interest for a lightweight, publicly developed standard with multiple alternative implementations to be driven out by a monster of a specification (6,000 pages) that has legal encumbrances and other complexities that mean it can be implemented by only one vendor.

My involvement with the Wild West side of Python came somewhat accidently. I am helping organize PyAtl and on June 14th we had an incredible meeting! My company Racemi gave a mind boggling demo of our datacenter management tool that is written in all python. Our FlagShip Product Dynacenter allows any OS, including Windows to move around to different hardware in the time it takes to warm reboot…go Python! Finally, Google gave two presentations, one on Cross Site Scripting Attacks and one on Twisted. We also officially launched the PyAtl website that night which is running the bleeding edge Turbogears stack of Sqlalchemy,Genshi, and Toscawidgets. My friend Alberto Valverde is in charge of Toscawidgets and the concept is really awesome! If you haven’t met Alberto yet, you should, he is one of those rare exceptionally helpful, yet insanely smart people.

Here is where the the fun started…

I invited Mark Ramm and Jonathan Lacour to come to our meeting and talk about Turbogears. Mark and Jonathan mentioned that on the way up to the meeting they had a crazy idea. How about building Turbogears on top of Pylons? They announced an experimental sprint the next weekend and this is where things got wacky!

Rick Copeland, Jonathan, Mark,Mike Schinkel, and myself met at Jonathan’s house and started to experiment. We ran into an initial snag with understanding the pylons controller and I called up Shannon Behrens, another friend, who is insanely smart and incredibly helpful. Shannon works on the Pylons trunk and asked him how we would mount Turbogears on top of Pylons. After he got over the “you want to do what!”, he helped us with some good advice. At some point we all went to get some Pizza, then came back to watch Jonathan and Rick go into the “Zone”. After they came up for air, a controller was working and Frankenstein was born..mu ha, ha, ha, ha!

It was 1 in the morning by the time we all quit, but Mark, Jonathan, Rick and I decided to meet at Panera the next day at 1PM to finish it off. A little more work was done the next day, but part of the day was spent just hanging out and talking shop which was pretty cool as I hadn’t met Mark or Jonathan before. It turns out Mark and I have a bit in common as we both grew up on a “Ranch type compound” for parts of our lives, we both have been SysAdmins, and we are both writing a Python book right now. Mark is a really fun guy to hang out with for anyone who hasn’t met him yet!

So, after the weekend was over with I started to hear about some of the excitement. I emailed my most educated friend Mr. Phd from Caltech Titus and mentioned maybe he could contribute with some Twill stuff for TG2. I talked via email a little with Kevin Dangoor and noticed his big announcement.

Apparently, people were really fired up about the collaboration between Pylons and Turbogears. Lets face it, I am very excited that all of these smart people are working together! It now seems that some momentum in the battle for the perfect Python Web Application has shifted, as Pylons and Turbogears have the 800 lb Gorilla of ORM’s in SQLAlchemy, and they have Toscawidgets which is about to come into its own.

I have written several small web applications in Turbogears and Django and I like both. Currently Turbogears and Pylons don’t have a way to graphically manage the database like Django’s admin tool and the API isn’t as stable, but from what I hear this is about to change…..

I do get the impression that many people in the Turbogears/Pylons world feel left out and a common heard rallying cry is that Django has a “Not invented here attitude”. Whether this is true or not, I learned this past week that if smart python programmers feel they aren’t apart of the fold, they are capable of creating an uprising and doing just about anything!

I will close with this comment, Ian Bicking, who wrote paster which I think is pretty sweet, mentioned in a fairly famous post that it would be great, but unlikely that Pylons and Turbogears would merge, yet the impossible happened and the two frameworks are closely working together. May I suggest an equally implausible scenario? What if Django, Pylons and Turbogears worked on developing an interchangeable API? Is this impossible…you tell me!

Untangle could turn out to be a poster child for free software. The company started out considering both free and proprietary software for its platform, but settled on a flat-out, pure-play open source approach. In return, they demonstrate the kinds of enhancements a commercial firm can make to free software in a range of areas commonly known as productizing.

I’m registering for OSCON, and Suzanne Axtell asked me to say some kind words about the conference. I always have trouble deciding which tutorials to take.

Via John Lilly (COO of Mozilla), Steve Jobs misses the gold ol’ Browser Wars.

Now I’m not the chief of a successful hardware/software/consumer products company, but the goal of being #2 by swallowing #3, #4, #5… seems somewhat wrong to me.

The Linux Foundation–the new organization formed this year from the merger of Open Source Development Labs and the Free Standards Group–is holding a summit this week at the main Google campus.

I think we can already call the summit a success, and an indication of the Linux Foundation’s acceptance by its community, just on the basis of the many busy, well-known people who chose to show up. It was a great place to make connections among both speakers and attendees.

The first day was a pretty public affair, with an audience of 200 that included journalists, so the discussions were basically polite and stuck to acceptable debates such as how to recruit more developers. I could not, unfortunately, attend the sessions later in the week where (to use Executive Director Jim Zemlin’s metaphor) the sausage would be made. But some interesting points came up anyway.

Thanks to Simon Morris’s A Rose by Any Other Name, I can now identify myself as a Neo-Desktopian. I’d say more, but I’m too busy right now trying to figure out why JSVim won’t launch on my iPhone; I’m trying to debug a compilation problem in the Monopolight GCC front-end running on EC2.

I think it’s a CSS problem.

One of the dimensions we watch at O’Reilly is the “platform” category. At a high-level in our taxonomy, the Platform category can be split into two groups — Open Source and Proprietary. The following charts represent the “platform” world for the first five months of each year [January thru May of 2003 - 2007].

This first Chart shows Dollars for the first 5 months of each year.

The next Chart shows Units for the first 5 months of each year.

Here is a little context for these charts. The data here is for the whole computer/technology publishing market, and not just O’Reilly. The data is actual cash-register sales in bookstores, as measured by Neilsen Bookscan, throughout the United States. Typically about 85% of this market is Amazon, Borders and Barnes and Noble.

The market as a whole is down about 10% compared to 2006. The open source and proprietary trends on the unit chart have closely mirrored each other except for this year. On the revenue chart, open source has produced more dollars for the past 4 years and looks like it will again this year. I thought you would find this data interesting.

Technorati Tags: Analysis, Books, Data, linux, Markets, mysql, open source, perl, php, python, Trends

But the idea of the Mozilla Foundation de-emphasizing applications in order to transform ourselves into a general purpose “platform” organization — giving up the fundamental focus on the human being a application focus provides, reducing our ability to help individuals directly — seems an absolute non-starter to me.

— Mitchell Baker, Application vs. Platform Focus

“Mozilla” is the people who understand that telling Mozilla what it should be doing is like saying “nobody in my neighbourhood cares about the litter” but not picking up a piece.

— Mike Shaver, now don’t take this the wrong way.

You can substitute just about any F/OSS organization for “Mozilla Foundation” in the first quote and any F/OSS project for “Mozilla” in the second.

Suppose you’re the author of a software project. You spend your time developing new versions and prefer to add new features in new versions.

Suppose you’re the user of a software project. You prefer not to upgrade to new versions. You might want some new features.

Suppose you’re the distributor of a software project. How do you reconcile these desires?

I have some thoughts on backporting to publish in the near future, but I would like to get other perspectives first.

I know Piers Cawley hates the word “metaprogramming” as much as I hate any of my pet peeves, but Tomasz Węgrzanowski just made a fantastic point:

Every use of text-based runtime code generation is a failure of language’s reflection model.

(See Ruby and methods with weird names.)

Language–and library–designers should keep this principle in mind. One of the explicit design goals of Perl 6 is to enable all of the metaprogramming of Perl 5 without exposing as much of the machinery. (For some reason, assignment to dynamically-scoped typeglobs frightened people.)

If you loosely follow the US presidential party nominations like I do, you know the second republican debate was last night. I heard there were surprisingly strong results for Ron Paul in the MSNBC online survey, so I thought I’d have a look myself at http://www.msnbc.msn.com/id/18963731/.

Six questions, a vote button, then a little note instructing me to “Vote to see results.” Well, I didn’t watch the debate and can’t even name all the candidates, so I’d really like to just see the results. If I were to make up answers, it would serve only to inflate the size of their sample with garbage.

Well… who knows, maybe they thought of this and I’m just supposed to press the Vote button with no selections to get through. Nope, a pop-up informs me to “Please make a choice before submitting your vote.”

So I did.

Yesterday was the Perl Teach-In at the BBC. People have been saying for months (years probably) that Perl is dying. If that’s true, then I’d like to know why the fifty places on the course were fully booked in less than two days and why another forty people signed up to be on the waiting list.

Perl certainly isn’t dead. On the contrary, the demand for Perl programmers in London is greater than I’ve seen it for many years - just take a look at the archives for London.pm’s jobs mailing list.

So that’s why fifty or so Perl programmers were willing to spend one of the hottest Saturdays of the year hidden away in a BBC conference room listening to me talking about Perl. It was particularly gratifying to see that most of them were people that weren’t already involved in London.pm or any part of the Perl community. It’s always been my belief that the majority of people who use Perl regularly aren’t part of the Perl community - so it was good to be able to reach out to some of these people and encourage them to join us.

The day seemed to be successful. Pretty much everyone told me that they enjoyed themselves and that they found it useful. That makes it very likely that something similar will happen again in the future. But no firm plans have been made yet.

I’ve put the slides online and they’re under a Creative Commons licence so that anyone else can use them to run a similar course in their city. The presentation was also recorded (well, until the microphone batteries ran out twenty minutes before the end) and those recordings will go online at some point in the next week or so.

All in all, I’m very pleased with how it all went. It was an interesting experiment and I’m glad that it all worked out so well. Thanks to all the attendees for turning up and to London.pm and BBC Backstage for their help in organising the event.

The last amusing discussion in perl6-language@perl.org mailing list was a proposal on renaming Hash to Dict. Perl and, after it, Ruby use the word “hash” for their associative arrays while Python and Smalltalk use “dictionaries” and Java uses “maps”. As so many things are changing while going from Perl 5 to Perl 6, the suggestion was attempting to introduce a “better name” while it is still possible.

Rob Conery reports that Microsoft wants to expand the OSS landscape for .NET, so they’re recruiting people to lead open source projects built on .NET.

The carrot is an MSDN subscription. I suppose that could be useful.

If I had any interest in .NET F/OSS programming, I’d instead hold out for patent indemnification for all of the contributors to and users of the software.

Give this offer a miss until MIcrosoft can get its story straight.

I try to get out to the Consumer Electronics SHows (CES) in Las Vegas every January. As I’ve blogged about on other sites, there’s actually a ton of Linux-based devices there, almost anything with an embedded OS runs Linux. One of my standard activities has been to go to the major consumer PC vendors (Dell, HP, Gateway, Toshiba, etc) and ask if I can get their product with Linux preinstalled, or failing that, with no OS and not have to pay a Windows license fee. Until today, the answer has been no except for some of the server products.

You notice I said “until today.” As of 4PM CDT today, Dell will be offering 3 systems (2 desktops and a notebook) with Ubuntu preinstalled and supported. This is a red-letter day for Linux, one of the major reasons that we don’t see the Penguin on consumer PCs more is that Windows is already pre-installed when the box arrives. Now, at least, consumers will have a choice. Bravo to Dell for listening to the hhundreds of thousands of people who voted for Linux, and taking quick action

Hibernate’s Gavin King recently published A Defense of the RDBMS. He has several useful points about ORM and object databases and their as-yet lack of success. Perhaps the most salient point is in the conclusion, however.

Note: The following is based upon my experience with IT-Security oriented environments (i.e. feel free to replace ’smart’ with ‘extremely geeky’). I would think this should apply to non technology people as well, but I do not have the experience to confirm that one way or the other.

During the course of my professional career, I have been fortunate enough to work in some environments that attracted and retained smart people. I have also had the opportunity to analyze environments that simply could not retain smart people. A few of the ‘bad’ work environments managed to hold the good folks for a tad longer than I thought they could by offering higher than market value salaries - but ultimately the good people, including myself, ended up leaving.

There are dozens of ‘Top 10′ lists that attempt to identify do’s and dont’s on how to keep people happy. As a fellow geek, I can easily dish out a never ending list of dont’s based on my experience in environments that simply didn’t seem to get it, but that is not the goal of my quest which is simply the following: What are the one or two root-causes of executive or management level decisions that repel smart people in any given organization?

I always enjoy a conversation with the folks from Splunk, which I wrote up at LinuxWorld Expo 2006. Three of the major enhancements to their 3.0 release, being announced today at Interop, demonstrate a very flexible, interactive–dare I say it? Web 2.0-style–approach to their model of collaborative system troubleshooting.

You can’t spend more than ten minutes at RailsConf without hearing the dreaded three-letter acronym DSL. They’re everywhere these days in the Rails and Ruby worlds. I think I ate one for lunch (either that, or it was a grilled portobello mushroom sandwich).

Confusingly, every example of a DSL I saw looked like something I would have called an API before my sudden immersive enlightenment.

If I’m confused about what a DSL is, you must be too. I took copious notes through the sessions on Friday and have devised a simple, ten-question test to help you determine whether a wad of code represents a DSL or an API.

Adrian Howard posted in his use.perl blog a text about antipatterns on writing jobs advertisements. It is a insightful piece of advice for those confronted with the need of writing a convincing call for new employers and entertaining for those wanting to know the mindset of well-intentioned people on the other side of the job interview desk.

Bugzilla 3.0 came out, and with it a debate over whether it is possible to write maintainable Perl code.

That debate has now spilled over into Bugzilla’s wiki (see Bugzilla:Languages). When I viewed the page, the final requirement for a programming language amused me greatly:

7. Enforcement of Good Code: One place where Perl falls down is that it doesn’t enforce any good coding standards. A language that does would be welcome.

I spend a lot of my time in a web browser (my top applications are the terminal window, an e-mail client, a web browser, and a text editor, in random order). The fewer distractions, the better.

I use Firefox as my browser, in part because it has a few important extensions that are impossible to ignore. One stands out primarily in that I almost never notice it: NoScript. This extension blocks JavaScript, Flash, and Java code from executing automatically. If I need such code to run for a particular page, I can allow it temporarily. If I always need the code to run (such as on our site statistics reports), I can enable it permanently.

The rest of the web is blissfully quiet non-responsive to anything but my clicks. Thank you to all of the NoScript developers and contributors!

From The Mathematical Sciences Research Institute.

The hat problem goes like this:

One obvious strategy for the players, for instance, would be for one player to always guess “red” while the other players pass. This would give the group a 50 percent chance of winning the prize. Can the group do better?

Don’t click the link unless you want the answer. Can you beat my time of 5 minutes?

You know all those nice things I said about the Microsoft Development environment a couple of weeks ago? Well, I still stand by them as a realistic opinion of the quality of the platform for developers. However, today’s news brings the major reason you should run away from depending on Microsoft technology like it had a case of Ebola.

The murmurs and worries about Microsoft’s ongoing patent gossip campaign, which came to a roiling boil with the Novell deal, have ended. Yep, no more rumors, just the plain reality that Microsoft is going to take their portfolio of laughable patents and start sticking it to the open source community legally, as spelled out in the most recent Fortune.

I’m trying really hard to avoid descending into obscenities here. So where-ever you see the * character, feel free to insert your own vulgarities as you see fit. * Microsoft has proved what a * bunch of * they are, and shown their true colors yet again. All the * about their open source lab and the code they were releasing as open source was in the end, just * propaganda, as many of us had suspected. Faced with omens such as Dell selling Linux on the desktop, they drew their last major card from the FUD deck, and hope to steal the pot.

And this is why you should use Microsoft technologies only as a very last resort. Because they don’t play nice with others. Sure, all companies are competitive and will do pretty much anything they can do to make a buck, but Microsoft is taking things to a new level. What you as a customer are being told, in essence, is that if you use any technology but Microsoft’s (or those of a company paying blood-money to Microsoft), you are likely to be sued. I don’t know about you, but I don’t like to do business with people who threaten and extort me.

I spend a lot of time coding in a lot of languages, with a lot of libraries, with a lot of obscure error messages. I don’t check any language or platform’s “official online docs” and I haven’t touched a hardcopy manual in years. No, the resource I reach for first is Google. But I don’t aim for results that look topical so I can examine them more closely. For me a search for an API call or bug fix is truly successful if I can see what I need to know in the excerpts without clicking through any results at all.

Give it a try for a little while.

Red Hat, which is holding a summit this week, announced plans for a new desktop that will seamlessly integrate local data and applications with remote data and applications accessed over the Internet.

We’ll have to see details before desktop users can determine how new this is. Many will say, “I already have several desktops that seamlessly integrate local and remote data and applications. One is called Firefox, one is called Konquerer, one is called Internet Explorer…” I assume Global Desktop offers a new level of collaboration we aren’t used to already.

And although quotes from Red Hat managers in a Vnunet article contrast the new desktop with what they consider an obsolete Windows model, my sense of what Microsoft is doing (from SharePoint through the new integrated Office features and onward) strives toward the same goal: letting everybody work on data stored on local corporate networks or the Internet, without regard for location.

But I’m optimistic; maybe this is big. Maybe a product modestly “intended for local government and small business customers” will turn out to be a paradigm shift.

Whether or not it’s significant, what I find most interesting is that the driving technology was borrowed from One Laptop Per Child, which Red Hat is heavily involved in developing. This would be just one more example of the oft-noted phenomenon that technologies developed for non-commercial purposes end up benefiting industry and taking off in new venues.

On this blog, Dave Cross recently wrote about free Perl training the BBC is sponsoring in London (Dave will be the trainer) and chromatic wrote about recruiting Perl programmers and frankly, it’s a hot topic in the Perl community right now. There are, frankly, more jobs than there are programmers. At this point, many people automatically say “just don’t use Perl”, but that doesn’t work. Here’s a bit of history about how this problem came about and my predictions about what’s going to happen next.

Hong Feng is a software developer and passionate free software advocate of many talents. He’s based in mainland China (although he gets around a lot) and he has a mission to help Chinese programmers make great contributions to free and open source software. I got to know him after he founded O’Reilly’s Beijing office in 1997, although he is now independent.

He’s developed several courses and is working on a 500-page book, aimed at both beginners and graduate students, that is meant to help them develop the discipline to be competent hackers. I can’t read the poem, but he tells me explores not only the major topics in computer science but such intellect-building activities as GO playing and the I-Ching, which presents original thoughts on “how to think about ancient Chinese intelligence in conjunction with modern math and technologies.”

Hong Feng’s approach fascinates me for combining a wide swath of influences with a joy in Chinese culture and pedagogical practices. He’s even written his personal story of development up as a poem, and set it to a melody from the time of the Great Cultural Revolution to his computer teaching. I can’t understand the song because I don’t know Chinese, but it’s available as an MP3 file.

Here’s a quote from p. 586 of one of our books:

Polymorphism is when a subclass “stands in” for its superclass.

10 points to everyone who can tell me what’s right about that statement as well as what’s wrong with it.

All the developers I know are hitting the gym, chugging protein shakes, and running three times a week. The geek image is changing!

Why has an interesting opinion in Math (Eww?) (and read the comments for much, much more).

Perhaps it’s finally time to break the close connection between math-loving computer science and programming.

I’m looking for people who have filed patents–not just in computing, but any field. I’d like to talk to them about some aspects of preparing the patent application.

A huge amount of public discussion has gone on about how patent offics approve patents, and how they’re litigated later. But why don’t we go to the start of the process? Let’s examine what inventors think about after they realize they have something patentable, and what they do to prepare the application.

I’d like to ask these people:

• How do you determine that something is patentable (that it’s useful, novel, and not obvious)?
• How do you go about searching for prior art?
• How do you decide how much to invest in preparing the application, and how do you decide that submitting a patent application is worthwhile?

Although I know the U.S. patent process a little better than others, and will probably concentrate on the U.S., I’m interested in stories from other countries too.

People with stories to tell can contact me by sending email to andyo at domain oreilly.com.

We’ve all suffered from lost time and lost connections because of false positives from necessary spam filters. Just this morning, board members of a non-profit I volunteer for were complaining to me that email to board members gets trapped as spam, and while working on this blog I nearly lost an email that was treated as a false positive.

But some businesses lose more than time. Open source advocate Ryan Bagueros, who does consulting around web developer and open source (his firms are northxsouth and Linefeed) told me lots of promising social networking companies are stymied because the emails they send members and prospective members get trapped by spam filters–especially at the major email hosting sites.

I find this ironic, because the social networking sites are structured networks that promise ultimately to be a replacement for email: richer with identifying information, more secure, and full of features to build relationships and communities. Yet to get off the ground, the sites depend on email, the only universal online medium, in all its primitiveness. And the sites suffer because of it.

Perl 5.8.x pumpking Nicholas Clark has more thoughts on recruiting good Perl programmers in London. I still keep hearing that plenty of companies want to hire good Perl programmers in places such as Los Angeles and London and much of western Europe, but they can’t find enough applicants.

Maybe it’s time to dust off your Camel book, grab any of Catalyst, Jifty, Maypole, or CGI::Application, and renew your career.

I did some analysis of our audiences who purchase books. This is a time-oriented graph. If you notice the spikes on the “Consumer” trend line, those represent the end of each year. Consumer purchasing reaches a peak at the end-of-the-year holiday period. Consumer means people who buy computer-oriented books like Ipod, Digital Photography, etc. Books that are not for software developers or system administrators or other computer science types.

So look at the slow erosion of the System Administrator audience. That is certainly interesting to me. Could it be that there are no really new IT admin tools or technologies to explore? I’d be interested to hear what you think.

Two useful indicators of the increasing viability of Linux on the desktop; one personal, one public. The public one is the imminent announcement of preinstalled Linux on Dell Desktop machines, combined with the news that Vista is foundering enough for OEMs to fall back to XP.

The other, more personal, is what happened when I went into CompUSA last night to buy a new laptop. Because I do Linux product reviews, it had to be able to dual-boot Linux well, and specifically support Beryl. I had resigned myself to the usual buy-and-pray approach, looking at the several dozen laptops on display, since even a quick web search on my PDA didn’t offer much guidance.

Not expecting much help, I lassoed a salesdroid to try to at least get some advice on which one would run Vista Aero best, and mentioned that Linux and Beryl were a major concern. I was surprised to hear that the droid ran Ubuntu at home, and more surprised when he told me he’d grab the ‘Linux Expert’ from the back of their tech support / repair area.

Within a few minutes, they had gone off to do some research on which models would meet my needs best. 30 minutes later (!), they returned to announce that my best bet would be a Core2 Duo HP with an Intel graphics chip. I was skeptical, but decided to give it a go. I took it home, shrunk the Vista partition (nice new feature in Vista…), and installed Feisty Fawn. True to their word, not only did the Wifi work with WPA right off the bat, but the AIXGL support for the Intel graphics processor did as well, and Beryl came right up.

Now, to keep this from being a total love fest, I should mention that the salesperson also tried to convince me that installing Linux would void the HP warranty, something I sincerely doubt. But this was in the context of trying to sell me an extended warranty, which is when all the FUD and half-truths come out. Overall, I was hugely impressed by the degree to which they both knew, and were willing to assist me with, Linux.

Both of these events tell me that, while this might not be THE year of Linux on the Desktop, it’s going to be a major step forward.

I’m probably not saying anything that hasn’t been said before here, but I thought I’d share a few thoughts on why people seem to be drawn to the Microsoft Way. I recently did something at the ‘day job’ that I’ve thought about doing for a long time, but never quite worked up the steam to follow through on, I signed up to participate in a a Microsoft-centric project, and to learn .NET.

I have made abortive stabs in the past to learn to code in the Microsoft Universe. I made a stab back in the bad old COM days, but the number of hoops I was being asked to jump through was more than I wanted to bite off at that time. Since then, I’ve carried that bad taste in my mouth, and resisted adding any Microsoft skill sets to my repertoire, even though it was sometimes a gap in my resume.

I’ve worked frequently in environments where there was the one Microsoft Guy, the evangelist who would constantly tell you how much easier it would have been in .NET. I’ve written them off as Kool-Aid drinking Gates worshipers. But, at the end of the day, I felt that if I was going to criticize them, I really needed to understand where they were coming from. Know thy enemy, and all that.

I spent last week learning in order C#, .NET and VSTO (that’s Visual Studio Toolkit for Office, if you’re not familiar with Microsoft’s alphabet soup.) I used the O’Reilly ‘Learning C#’ book, and did something I rarely do, went through it pretty methodically (at least the first half or so.)

Guess what? Microsoft has a pretty good development suite on their hands. To be honest, C# is largely what I’d do if I could rewrite Java from scratch with no concerns for backward compatibility. It has a couple of really cool features, like the virtual, override and new keywords that let you specify what should happen when you cast a class to it’s base class and then call a method on it that’s defined in both.

Visual Studio is a slick tool that really does let you bang out applications (and with VSTO, plug ins for Office) is very little time. ADO.NET is no worse then JDBC, and is pretty seamlessly integrated into Visual Studio. I was able, by the end of the week, to develop both stand-alone applications and Office plug ins that could talk to back-end databases, having written very little code. From what I’ve seen, ASP.NET does the same for MVC web applications.

So what’s good about a monoculture, and why does Microsoft win so often when people make a decision about platforms? Largely because what the open source community sees as a strength, people trying to get a job done in the real world see as a weakness. We celebrate the diversity of choices available to solve a problem and call it freedom. IT managers and CIOs look at it and call it chaos, confusion and uncertainty.

Should I use iBatis or Hibernate? XFire or AXIS? Perl, PHP or Ruby? Debian, Fedora, Ubuntu or Suse? Make the wrong decision, and you can waste a ton of time, as we found out on a recent project when we wasted a week try to make AXIS2 work for a web service project, only to find out that XFire was the right choice.

For the Microsoft Guy, no such confusion. You use ADO.NET, ASP.NET, C# and Windows. They all work, they’re all well documented from the perspective of a developer’s needs, with nary a disparaging ‘go look at the source’ blow-off. Every time I thought I was going to be stuck, there were a dozen articles explaining how to do exactly what I needed to do, with sample code that was up to date with the versions of the software I was using, and that actually related to the problem I was trying to solve.

Microsoft offers the certainty of no choices. Choice isn’t always good, and the open source community sometimes offers far too many ways to skin the same cat, choices that are born more out of pride, ego or stubbornness than a genuine need for two different paths. I won’t point fingers, everyone knows examples.

Now, least you think I’ve been turned to the Dark Side, there is one BIG problem with a monoculture, which is that you’ve essentially sold your soul for the stability of a clearcut set of choices. You go down the .NET path, you’re pretty much stuck there forever, Mono not withstanding. You’re always going to be running on a Windows platform. You got the pretty gold ring, but Sauron gets to pull your strings and make you dance. For many companies, ones that don’t need to worry about deploying into heterogeneous environments, that’s a deal they’re more than willing to make.

The takeaway I get from this entire line of reasoning is this: that somehow, someway, we need to start doing some winnowing. The 700 lb clue-bat has to be available to pound on the head of those who fork for no better reason than a disagreement over a license, or who should get to call the shots. When we hear about two or more projects that answer the same question, we should be asking ourself “Why don’t they pool their effort and produce one really good solution?”, rather than celebrating diversity for diversity’s sake alone.

Do we really need Ruby on Rails AND Groovy on Grails? When the April Fools’ announcement of Python on Planes came out, it took me a second to realize it was a hoax, because it’s just the kind of ‘doing something for the sake of doing it’ effort that fractionalizes the OSS community. There’s no way to stop people from starting new duplicative projects, nor should we want to, but please God, do we have to actively encourage it?

We spend a lot of time complaining about all the evil ways Microsoft uses to foist themselves on the world. By doing this, we automatically remove any blame that we ourselves may bear for their successes and our failures. The reality is that there are good, practical reasons that drive people into the arms of the Redmond tool set, and we need to accept that as a fact and learn from it, rather than shake our fists and curse the darkness. For we have met the enemy, and it is us, not Microsoft, at least not all the time…

Steve Holden posted an excellent blog post entitled “Python is Not a Religion” on the topic of Python advocacy the other day, laying out the need for common sense, passion, and planning. Python is a good answer for a large range of programming problems. However, it is not the only good answer for this same range of programming problems. Nor is it even an answer for another (albeit smaller) set of programming problems. I’m all for Python advocacy as anyone who frequents this blog can attest, but it needs to be done in the right way and for the right reasons. I think Steve helps get us closer to both of those. Great post, Steve!

If you’re a programmer, you may spend a lot of time figuring out how to make things faster. If you work with number-crunching, you’ve seen plenty of cases where you can greatly reduce a program’s runtime by some huge factor. You’re probably pretty good at it…

If you’d like to test your algorithmic wit, describe how to speed these calculations up (Ie., without actually adding up all the numbers):

What is the sum of the whole numbers between 1 and a trillion (10^12)?

What is the sum of the whole numbers between 1 and a trillion that have the additional property that they are divisible by 7? (7, 14, 21, …)

What is the sum of the whole numbers between 1 and a trillion that are divisible by 7 and when reversed are still divisible by 7? Numbers like 7, 168, and 1169 do this.

I was lucky enough to read J. Michaelson’s classic (well, 2004) magazine article entitled “There’s No Such Thing as a Free (Software) Lunch.”

He digs deep into the details of licensing and corporate acceptance of Free Software. He’s a lawyer so he doesn’t speculate, but I’ll connect the dots where he doesn’t. I’m going to reveal the dark secret of so-called “Free Software”.

Ignorance is Strength because if developers, corporate lawyers and corporations knew how complicated, varied and legally untested the licenses were, there would be much less innovation and confidence in Open Source.

Freedom is Slavery because code freedom is inversely proportional to developer freedom. The more “free” a piece of code is, the less a developer or corporation can do to integrate it into their products.

War is Peace because in the midst of flamewars, vicious IP lawsuits, and MS/Linux hate, the masses of Free Software users are productive, trusting, happy and idealistic.

You’ve seen this before, in George Orwell’s novel 1984 the three slogans of The Party are War is Peace, Freedom is Slavery, Ignorance is Strength! Big Brother anyone?

I don’t know if you’ve seen The Two Things Meme, but apparently if you ask an expert what the two things you really need to know in her discipline are, she’ll think for a minute and come up with them. But, other experts all come up with different pairs!

Glen Whitman maintains many pairs of The Two Things about Computer Science and programming in general (Jump Link to his Computer section), but nobody appears to have asked the question specifically about LAMP (0 results as of today).

So, what are The Two Things to keep in mind for LAMP?

Despite my best efforts, I occasionally have to perform system administration. This often means checking logs, reading man pages, typing commands, and editing configuration files–often on a remote system. I have a great little shell script that opens several Xterms in the appropriate way on my local machine under an X11 session, but that doesn’t always work on remote systems.

That’s just one reason to love GNU Screen. All of a sudden, one remote login can host a handful of command-line sessions. It’s easy to flip back and forth as necessary.

I used screen off and on for a few years, until someone (I forget whom, sadly) taught me the most useful trick of all. I use Ctrl-a and Ctrl-e to jump to the start and end of Bash lines, but screen uses Ctrl-a as its hotkey. At least, it does until you add one line to your ~/.screenrc:

escape \034\034

Now Ctrl-\ is my hotkey; I never type that by accident, and Ctrl-a does exactly what it should do. There’s no pain in using screen anymore, and I’m a convert. Thanks to all of the screen developers and contributors!

I’ve been out of the music industry for a decade, but every so often I look into it again. The decision makers haven’t improved since I stopped caring ten years ago, and they’re not any smarter.

The current bugaboo isn’t the cassette tape anymore, but piracy.

There’s really an easy solution. If losses due to piracy are as high as industry organizations claim, the economics will eventually make this solution feasable.

Stop shipping physical products via the sea.

Now if I were in charge, I’d address the ease of digital distribution of music and the potential for copyright infringement, but apparently armed robbery at sea is more important.

In the spirit of chromatic’s many thank-you posts, I want to send a huge thank-you to all of the developers and contributors of the OpenSSH project.

SSH provides the right tool for so many tasks, especially when faced with a temporary access need that spans several different networks. It give me secure access to my colocated servers and my clients’ systems, a simple way to print from my clients’ servers to my home laser printer, compressed secure X11 access, easy command line file transfers, and encrypted tunnels for oh-so-many things (including iSCSI connections and pop3 pickups). As a base technology, SSH enables ssh:// URIs throughout my work environment (in gnome/kde/OOo and more), permitting me to securely open documents from any machine I use on any other machine I use, even across complex combinations of intervening networks, and SSH also enables safe remote displays in VNC (-via) and NX.

So thank you, OpenSSH and related projects! — your work securely enables many remote access scenarios and saves me hours every week.

At SD West last week, I sat in on the Agile — With or Despite of Global Development roundtable. While traditional agile development (as you might see in XP or Scrum, to some extent) recommends small teams that sit together, many organizations and projects are larger, with team members in multiple locations.

This separation makes producing software more difficult. (That’s one of the reasons agile development tries to keep team sizes small and everyone together.)

Why does Microsoft talk about Total Cost of Ownership when you merely license their software?

Perhaps all of these TCO studies are true; it does cost Microsoft less to own Windows than it would to try to buy all of the copyrights for every competing project.

Recently there was a thread on the O’Reilly Radar started by Tim O’Reilly posting a chart that I put together from our book sales data. The chart showed comparative market share for most of the relevant programming languages. I have updated this in the chart below and have included 2005 data as well.

A litte insight to the numbers behind this graph. The percent shows that a lanugage like Java represents ~23% of all book sales when looking at the language dimension. That means, I compared all the languages and which books have ‘XYZ using Java’, or ‘Embedded FOO on Java’, etc. So it does not have to be a strict Java Programming book, but rather a book that is Java-centric or the examples contain mostly Java code. I compared aggregated sales units during Jan-Feb for 2005, 2006 and 2007.

Disclaimer:

This is not an exhaustive study as I threw out Languages that did not have a representative sample in one of the years. In other words, if a language area show up with 15 units in 2005, but not in 2006 [or 2007] it was dropped. These are the bottomfeeders. So if you use one of those languages, Squeak, you will not find the results in my chart.

Some observations:

During the previous two years and this year, during January and February, the biggest declines were seen with Java ~5.5% down, C/C++ ~4.5% down, Visual Basic ~2% down and Perl ~1.5% down. The reason I point these out, is that is is market share for books, the unit sales numbers, which I will not supply, are a bit more alarming if you are on the declining list.

During the previous two years and this year, during January and February, the winners seem to be: Javascript ~5.5% [almost exactly what Java lost], Ruby ~5.25%, .Net Languages ~3% and C# ~2.75.

So when you look at the top for both lists, the totals are a bit different. There is a 3% difference on the winners side. What is says to me, is that most of the growth was seen in the four top languages, while the decline was spread a bit wider.

Your observations?

Do you really care about languages and what books sales tell us about trends? Don’t think for a moment, as past posters have said, that some languages have better market share because one language has ’sucky’ books. I did a quick analysis of GPA ratings on Amazon by language, and there are not any really significant wins for one language over another. One thing that does factor tough, is early to market. On average, when a language or technology topic is in its infancy stages, the market is more forgiving on the reviews. About .5 for the first books to show up in a category. You could say the the first are usually the best, but that does not hold up either.

Do you think maturity of a market shows in this data? Javascript/Ajax, Ruby/Rails are top two and are fairly nascent. Java and C++ are mature technologies with presumably less newbies clamoring to learn them. So what do you think?

If there is enough interest, I will follow up with some efficiency and average title metrics.

I just used GNU find to search a directory hierarchy for files matching a particular naming pattern. I’ve been programming for long enough that writing a tree-walking algorithm to search for appropriately-named leaves is almost trivial, but the point is that I don’t have to do that. Piping the output of find into my filter program to search within the files was sufficient.

I use the other findutils programs–especially locate–several times each week. Without them, I’d get lost in a sea of thousands of files. Thank you to all of the developers and contributors. You saved me a few minutes today, as you do almost every day.

I was discussing object oriented (OO) programming with someone who was working on a horrible piece of software with class names like StartSession. I’ll call him “Alice”. Naturally, when wading through these classes, he finds 400 line “methods” in classes which are merely OO façades around procedural modules. This is disappointing, but it’s all too common. If you think that StartSession is a good name for a class, someone has done a poor job of teaching OO to you.

I think part of the problem is that while there are some excellent university professors who do a fantastic job of teaching OO, many professors I took classes from had little to no real experience outside of the classroom, or those with experience clearly went back to teaching due to the old adage “those who can’t, teach”. Surprisingly, two of the best instructors I had were teaching COBOL. Both of them clearly had decades of real-world experience under their belts and it showed in the classroom. They understood their material, they understood the pitfalls, and taught us how to work within the constraints of the language.

Getting back to Alice, he told me about an idea he once implemented to make it clear to other programmers that OO classes can be thought of as responsible agents. One of the first classes he wrote for his work was named the AuthenticationFairy.

Besides e-mail, the only real office document I ever use is a spreadsheet. I’m a very happy Gnumeric user. My needs are reasonably simple; I share spreadsheets with Excel users and rely on some formulas.

It may not seem like an elaborate or flashy feature, but Gnumeric just works for what I need it to do. I’ve never had a problem reading or saving spreadsheets in formats that non-Gnumeric users can read. It works almost transparently, and the only reason I think about how often I use it is because I deliberately made a note of the applications I use the most.

There’s little better praise than “It works so well that I never think about it.” Thanks to all of the Gnumeric developers and contributors!

Steve Loughran’s How to Own an OSS Project, part 1 and especially How to Own an OSS Project, part 2 bring up the always-relevant issues of security, transparency, and trust. In particular, the second entry asks a most insightful question: how can you trust the documentation?

It doesn’t help when the documentation suggests outdated practices which are, at best, dangerous and, at worst, completely wrong. (I’ve patched a few of these in Perl 5, myself.) Add to that active malice, such as a recent dangerous answer to a novice question in comp.lang.lisp, or running obfuscated code outside of a locked-down sandbox, and it’s almost a wonder there aren’t more security problems related to source code posted on the Internet.

Piers Cawley picked up on a rant I had elsewhere in DSLs, Fluent Interfaces, and how to tell the difference. An API that uses Ruby symbols isn’t automatically a DSL, nor did Ruby invent or even popularize the concept (though it seems to have taken the “How to Draw a Horse!” concept right out of matchbook covers)–I credit lex and yacc as ancestors.

Once I simplified and unified almost all of the language-specific test harnesses in Parrot to a single line per language implementation:

API or DSL? Does it really matter? Maybe. It can be difficult to talk about closures when Luddites from the Java world claim that they’re the same thing as anonymous inner classes, and I still consider Python’s use of the word lambda as misleading.

Then again, there’s no reason that code has to look anything like valid Perl. It only needs to be a list of strings, and Parrot::Test::Harness could do anything. Consider also P5NCI::Declare which extends the concept:

use P5NCI::Declare library => 'shared_library';

sub perl_function :NCI( c_function => 'vii' );

perl_function( 101, 77 );

Maybe a language with richer syntactic elements–and the possibility to change their behavior–allows greater potential to create a true DSL modeled after the domain.

Either way, the possibilities for API simplification are hard to ignore.

If you think Twisted is the engine of the internet, then place your vote at this site.

William Hurley first got in touch with me many years ago, not to promote one of his own projects, but to set me up with some of his colleagues to write about network security. Later, he got involved in several open source ventures in networking.

So William is both an innovator and a facilitator. He’s willing to lend his expertise wherever there’s a good chance someone can make a difference in an area he cares about. And he definitely cares about open source.

Now he’s chief architect for BMC’s open source strategy. As he points out in a podcast, this position is an important step for BMC, but only one of several.

In the free software world, we’re getting used to companies putting money and support behind the software. This doesn’t diminish the importance of the individual zealot. Individual zealots are the source of many new tools, and the genius of free software is the ease with which it allows someone to introduce a new idea and then let larger institutions amplify it.

With William at BMC, I can be confident that this large company will not only continue to contribute to open source, but will maintain strong community bonds so the route from individual innovation to large-scale adoption remains pothole-free.

In September 2006, I wrote a short blog post on Python’s readiness for the enterprise. That posting had a question mark in the title because someone asserted (jokingly) that Python is, in fact, not ready for the enterprise. (Remember, my post was a link to a tongue in cheek blog entry elsewhere. I’m not stating that Jeff Waugh was trying to discredit Python.)

I recently came across this eWeek article which outlines Python’s use in the airline industry where failure would be very, very bad. This eWeek article contains assertions of some really smart people that Python is up to the task. Could they be wrong? Certainly. But I don’t think they are. And I’m not going to defend my reasons for thinking so. That’s not the point of this post.

My whole point in posting this is perhaps more for my own benefit than anyone else’s. I see Python too often treated as a toy programming language. I see people who feel that the language has failed them somehow because they encountered some problems at runtime which would have been caught by a compiler. I’m currently watching Python being replaced by another language. (No, this isn’t the wholesale state of Python in the world. Just my little segment of it). This article is reassuring to me that I’m not totally off of my rocker by thinking of Python as an excellent and capable language. And it gives me a warm fuzzy to think of Python being used in such a weighty manner.

By the way, I think ITA (the company around which the article was written) is hiring. If someone from ITA wants to shoot me an email with a link, I’ll gladly edit this post and put the link in. Good going, guys and gals of ITA!

I’ve used revision control for almost all of my professional career, starting with CVS in 1999 as I joined the world of free software developers and migrating to Subversion in 2003 for personal projects. Now I rarely use Subversion directly; I use SVK for most of my work.

SVK accesses public Subversion repositories almost transparently, so on projects where I have commit access, the only real difference for me is typing svk instead of svn.

For projects where I don’t have commit access–or when I’m on a train or an airplane or in an airport, hotel room, or conference center without reliable Internet access–it’s amazingly convenient to be able to commit changes as I go and merge them later.

I enjoy the use of offline mirrors and the ability to generate patches or push certain commits but not others. Even better, there are no .svn directories in my checkouts to deal with. Everything I remember from using plain Subversion before extends to creating and managing and working with repositories, but I’ve gained the ability to manage my own branches even without commit access at the moment or in general.

When I need those features, I really need them. Thank you to all SVK developers and contributors for making my life easier!

I’m trying to dig into C# a little more at work, so I decided to do something really easy and write a little code to grab an RSS feed and pull the enclosure data out of it. Before you look at the code, let me say that I don’t have an objective point to make with this. I’m not trying to make sweeping judgments based on number of lines of code or anything else. I do have some subjective remarks to make at the end, however.

Here is the C# code:

using System;
using System.Xml;

namespace RssGrabber
{
    class MainClass
    {
        public static void Main(string[] args)
        {
            string rssUrl = "http://geekmuse.dreamhosters.com/wp/?feed=rss2";
            XmlTextReader reader = new XmlTextReader(rssUrl);
            while(reader.Read())
            {
                if ((reader.IsStartElement()) && (reader.Name == "enclosure"))
                {
                    while (reader.MoveToNextAttribute())
                    {
                        System.Console.WriteLine("{0} => {1}", reader.Name, reader.Value);
                    }
                }
            }
        }
    }
}

Here is the Python code:

from elementtree import ElementTree as ET
import StringIO
import urllib

rssUrl = "http://geekmuse.dreamhosters.com/wp/?feed=rss2"

et = ET.parse(StringIO.StringIO(urllib.urlopen(rssUrl).read()))
for elem in et.findall("//enclosure"):
    for items in  elem.attrib.items():
        print "%s => %s" % items

Now, on to the subjective. Python just feels more manageable than C#. It feels friendlier. It didn’t feel tedious to try to get data out of. That’s it. That’s my point. If your opinion varies from that, then that’s OK. I’m not here to change your mind. And I’m definitely not here to try to offer up comprehensive language comparisons. I’m just expressing a little piece of an experience I just had which I thought was worth noting.

In the financial world, the big news today is the worldwide drop in stock prices, including the worst performance in the United States since the aftermath of the September 11 attacks. But there seems to be an interesting computer story too. During the tense afternoon in the U.S., Dow computers fell behind and failed to reflect the full extent of stock declines. Apparently, Dow brought some new computers online about 3:00 PM, and within a few minutes the Dow caught up and registered an extra 200-point fall. This, in turn, increased the panic among investors and led to an exaggerated sell-off, although this was probably mitigated by the end of the day.

I can’t find much information about this yet, and am relying on a few snippets of online news reports plus an NPR radio broadcast. It will be interesting to hear more details. It sounds as if the Dow computer administrators (who are highly paid, well-trained, and dedicated) had a back-up plan in place and implemented it successfully. But it must have been a horrendous day for them, and the jolt that investors received on seeing the sudden drop could not have contributed to good financial planning.

When we hear more about the computer systems, it may provide lessons for emergency management in a number of areas, including local and national government responses to disasters. The incident shows that information systems are at their most strained–and in the hardest ways to anticipate–just at the times when the public asks the most from them, and needs information most urgently.

Like so many Internet abuses–spam, artificial measures to boost search engine rankings, and most recently, anonymous postings by businesses or political campaigns that pretend to be unaffiliated–click fraud is an Internet war that undergoes constant metamorphosis.

Both sides (those who wish to abuse the system and those who want to keep it clean) are always increasing their sophistication, pushing more and more criteria through more and more complex algorithms in an effort to outsmart the foe.

In these sorts of battles, the good guys are always attempting to reduce the burden on humans by automating their responses, and run up against the problems involved in anticipating a creative human foe. Also endemic to these situations are complaints about unfairness, inaccurate classifications, and other weaknesses suggesting a need for standardization.

What interests me about the company Click Forensics is the distributed, P2P-like strategy put into practice by their Click Fraud Network. It attempts to maximize the volume and variety of data available for sifting the good clicks from the bad.

The Network brings in a huge amount of input data and allows fast turn-around for results. The question is whether this technique will work for the domain of click fraud.

As in all the collaborative, Web 2.0 sorts of sharing that are currently making the news (or quietly making revolutions in various fields), the Click Forensics strategy reaches out to large numbers of affected users in an appeal to combine their power and try for ever-better results. The philosophy behind this strategy is intriguing; whether or not it proves successful, it is already yielding some interesting public results.

I have everything (except for my laptop) packed up and ready to go to PyCon. I still haven’t nailed down which sessions I’m going to attend. I look forward to meeting everyone.

When I have to write in C, ccache makes it much less painful to rebuild software. I appreciate that Make handles dependencies, but its reliance on timestamps sometimes causes unnecessary rebuilds. If ccache fixed only that, it would be useful–but it does much more.

This is one development tool I would miss greatly if it did not exist. Thank you, ccache developers and contributors!

Competition in the marketplace is a good thing right?

So now we have both jobs.perl.org and jobs.perl.com.

I’m about to start looking for a new Perl job - so let’s see which one of them is most useful. Currently the perl.com seems a bit light on jobs in the UK, but I’m sure that’s just because it’s early days.

Update: A couple of people have decided that the comments to this post is a good place to advertise jobs. Well, it’s not. Why not use one of the services mentioned above. If the perl.com service prevents you from posting UK jobs, then contact them to complain - I have nothing to do with that site.

It might also be worth mentioning a blog post I wrote a few weeks ago which contains some advice on finding Perl programmers in London.

I’m planning on attending PyCon next week. I was just going through the scheduled talk list and am having a really hard time deciding which sessions to attend. I mean, how am I supposed to choose among Python-Dev Panel : “We make the things that make Python work.”, WSGI: An Introduction and Towards and Beyond PyPy 1.0? And then I have choose among Web Frameworks Panel, Writing Parsers and Compilers with PLY, and Python on Parrot — under the hood. The really bad part of having so many great sessions is that you have to miss something. It’s frustrating having to choose, but it’s going to be sooooo good.

Dru Lavigne, our resident BSD guru, argues that the issue of women in F/OSS is all about respect.

While it may be worthwhile to consider stereotypes, generalizations, and snap judgments that affect your first impression of someone of whatever gender, perhaps Dru is right. What does it take for an unknown novice to gain respect in a community?

I’ve used Xfce as a window manager for almost four years now. It does everything I want: flexible numbers of virtual desktops, no icons on the root window, customizable root menus, adaptable mouse behavior, and vitally important window management features such as window shading and window hiding.

I couldn’t do my work without it, and I keep finding new features and enhancements (xfce4-verve is wonderful). Thank you to all of the Xfce developers, documenters, testers, and other contributors!

I’m in the process of designing a workflow framework (in Python, of course). As I’m laying it out, it’s helpful to think in terms of the types of the pieces that are going to interact with one another and what they can do. I’ve begun creating “base” classes which define empty methods which I plan on objects of corresponding “concrete” classes containing. This is all a nice thought experiment as it helps congeal some of my thoughts, but there is effectively no real value in implementing these empty base classes.

Some time back, I had read about Zope interfaces. I didn’t really think much about them at the time, but now I’m giving thought to using them. Since the Python language does not provide linguistic constructs to specify an interface (at least, not like some languages do), I thought it might be helpful (if for no other reason than documentation and design) to use something like Zope interfaces to help build my workflow manager.

The alternative is to proceed as I had begun and lay out some empty “base” classes which will contain the methods which I expect “concrete” classes to contain. And rely on unit tests to enforce that instances of each concrete class will have the appropriate methods behaving in the appropriate way. (And I would assert that using unit tests is the right way to go even if you use “proper” interfaces.)

Now that I’ve layed out my situation, I have a question. Is there really any value to something like Zope interfaces? And if there really is benefit, is it merely documentational and design-oriented which could be achieved with convention without having to rely on a tool such as Zope interfaces? Please post your comments here, as I’m really interested in reading what others thing on the subject.

The Community Patent Review (a phase of the better-known Peer to Patent project) is seeking to hire a top-notch Ruby on Rails programmer.

This is not an ordinary job posting; it’s a chance to get paid for public service. The project is developing software for the U.S. Patent and Trademark Office, and is being closely watched in the UK too. It implements a process to let experts in multiple fields to submit prior art and other information related to patents.

This project has the potential to alleviate some of the problems universally recognized with patents (particularly patents that should have rejected because of prior art) and can serve as a model for other projects that open up government to the public.

Among the companies that have already agreed to submit patents for this process are IBM, Microsoft, Computer Associates, and General Electric.

I don’t believe there’s been enough discussion of the weaknesses gradually being uncovered in Microsoft’s 6,000-page dump of Office behavior, which they are trying to call a standard. Andrew Updegrove’s summary has gotten some circulation on the Internet:

The Contradictory Nature of OOXML

and now Groklaw is organizing opposition to the ECMA standardization process, which is being fast-tracked (a better term would be railroaded):

Searching for Openness in Microsoft’s OOXML and Finding Contradictions

To help Office to become a standard, one adaptation governments could make would be to retroactively declare 1900 a leap year. This would require updates to history books and other documents (for instance, V-E day would change to May 7, and the World Trade Center attacks would have taken place on September 10) but I’d like to see a cost comparison with the alternative that businesses dread: migrating to open document formats.

Seems I’ve been tagged by chromatic to reveal five things you didn’t know about me.

For several years I’ve been fascinated with technical information people get online, instead of from books or journals. Everybody looks online for help installing software, finding programming library calls, fixing bugs, and solving any other technical problem they have on their systems. (Systems, not computers–computers are boring, Steve Jobs told us that. Nobody wants to be associated with computers any more.)

A lot of information is still missing online. That’s good for O’Reilly, because it means the book industry still plays an important role. But this article is not about missing information in general. It’s about a specific segment of missing information: the frustration you feel when you visit a site and don’t understand it because it lacks some background you need.

I encountered innumerable background issues while coding up some JavaScript recently. For instance, after cloning a large and complex node, I was stymied for a long time because I tried to alter children of the node and kept being told they didn’t exist. It was a surprise to find, ultimately, that white space between nested elements counts as a child node. (The log call in Firebug helped a lot to clear up the mystery.) This is just one example where a reader needs to bring his own background to online documentation.

My friend and co-author Jim Shore tagged me to share five little-known personal facts.

Lars Wirzenius posted a short reminder that the single point in common for all free software developers is that we write free software. Please keep that in mind before you assume that all or most other free software developers share specific hobbies, values, or beliefs.

You haven’t seen me much since October. You can blame that on my editor - the task master during the days leading up to production and the laid back, friendly guy afterward. I’ll dangle a participle fir you: Writing “Linux System Administration” daunted me. Whoever wrote this got it right:

Linux System Administration (LSA) is not only knowledgeable and practical, but convenient. The ingredients for this book had been scattered throughout mailing lists, forums, and discussion groups, as well as books, periodicals, and the experiences of colleagues. Everything is now in one handy guide. In the course of their research, the authors also solved many problems whose solutions were completely undocumented. They now pass their lessons on to you.

Oh, I did I mention Bill Lubanvic who came in during the later innings and rescued us, handling tech review comments and backing me up beautifully. His name goes on the cover for what he contributed.

I’m back now and looking for real work. LSA took my full attention and I could only take short-term assignments during that time. I’m ready to get back in the saddle. So, what’s up?

I’m pretty impressed with Server 2003 and the integration of Realms in AD. Putting together NIS and Kerberos to integrate users makes so much sense to me. I also have open eyes now when it comes to forests and trees. I could go camping.

We’ll see what happens and I’m keep you posted. Meanwhile look for http://www.oreilly.com/catalog/9780596009526/#top the book. It’s due out in March and I promise you have not seen this level of sysadmin before. Thanks to all the people who believed in the concept.

A colleague told me an amusing story about an oddball design decision that puzzled him and three tech service representatives for days. Norton Antivirus told him his antivirus software was out of date, so he updated it. The next time he booted, the program told him it was still out of date–and it continued to do so on every boot even though it had claimed the previous time to have updated successfully.

The first responses naturally focused on whether the database was corrupt or otherwise failing to update. But eventually a savvy technician figured out the real problem.

It seems that my colleague had checked his calendar for a date in the following month and had left his calendar on the month of October while in fact it was still September. Norton Antivirus relied on his personal calendar to determine what date it was. The software had been updated for September, but the program thought the month was October.

I am curious as to why a programmer would take such an unrobust decision to use a user’s calendar setting to determine the date. I’m wondering whether the programmers discovered that to get the date from the operating system would require a variety of different calls on different systems, and found it simpler to consult a calendar. Obviously, this backfired. A program that installs automated updates should use an external authority to find the date, and not depend on any settings on a user’s PC at all.

The bug was particularly amusing because the program was promising to have an update it couldn’t possibly have for the upcoming month.

It was a remarkable moment when that staunch pillar of American media, Time Magazine, chose the entire population as Person of the Year. While Time raises a shrine to grassroots news and culture—including explicit obeisance to the YouTube amateur video phenomenon, musical mash-ups (with no hand-wringing over copyright infringement), and the Wikipedia smorgasbord—the editor complemented their precedent-shattering announcement with a brave claim that they regarded the information horde as an enhancement rather than a challenge to their journalistic profession.

User contributions may help to bulk up every business that depends on information, but they also force the painful exercise of new muscles. At the same time, these contributions are creating a new kind of economic value that may require a different way of measuring corporate assets.

I noticed a new feature in an enterprise security package offered by GTB Technologies. This package protects companies from employees or other people on internal networks who send out content marked by the company as sensitive: trade secrets, sales data, customer contact information, and so on.

What makes this package different from most is that it can check content even if it’s encrypted. Essentially, companies can have their security cake and eat it too. They can allow widespread encryption to protect against snoopers inside and outside, while preventing employees from using that encryption to sneak company secrets out port 25 or even something as immediate as IM.

And GTB claims they can do this while adding only 3 milliseconds to each transaction.

Language redesign is difficult, isn’t it? Once you start challenging base assumptions, you find that a lot of your previous conclusions are shaky, and good luck reigning in blue-sky ideas!

See you in 2007… or 2008… or 2009.

Best wishes,
a Perl 6 hacker

Wikipedia says that Walt Disney died 40 years ago today, 15 December 1966.

Tell me again how extending the copyright on Steamboat Willie (released in 1928) is going to encourage him to create more artistic works? Walt’s been awfully quiet for a while now….

SpamAssassin hero Justin Mason has posted a summary of opinions against challenge-response mail systems. I particularly like the pyschopathic challenge-response system user in the comments who defends blowback by arguing that it only affects a few innocent users.

Sorry, folks. CR fails my one question certification test for mail filter authors, and that’s why it’s not just bad and wrong, but profoundly antisocial–just like spam itself.

Blessed are the toolmakers, writes Piers Cawley. I agree.

Recent blog postings here on the O’Reilly Network and articles on Slashdot (including a recent review of my book) have generated some really strong negative comments about the Fedora project. Does Fedora really matter?

When Google bought YouTube, lawyers and business analysts across the U.S. warned it would be swamped by lawsuits over copyright violations. Sounds scary enough–but what about facing criminal indictment for posting a violent video clip? On this basis, the Italian government has launched a criminal proceeding against Google and raided their Milan office, according to the Italian Internet anti-censorship organization ALCEI.

Enjoy this imaginative Thanksgiving story about the dangers of DRM and the potential of open e-books, from David Rothman of Teleread.

We all know that Hello World is the first program many people write in a new language or to test that their development environment is properly configured.

But there’s actually a family of domain and paradigm specific Hello Worlds, for instance, the factorial program could be the Hello World of functional programming.

I’ve been told that the Hello World of CGI is a page with a drop down menu of colors. On submitting, the page background is changed to the selected color. I used this in my entrance into Ruby CGI early today.

I’d love to make this list longer, but I don’t know what the prototypical first program is for OO, or for Logic programming or database access. If I collect more I’ll post them here. If you have ideas, please leave a comment.

Microsoft’s Jason Matusow wants comments on Microsoft’s patent pledge to F/OSS developers. That’s the pledge where Microsoft won’t initiate legal action against any developer who never distributes his or her code in any commercial fashion.

Here’s my comment: that’s useless and insulting.

So the other day Sun announced that Java would be released as true open source software under the terms of the GPL(v2). A few years ago, when I was developing a considerable number of my projects in Java, this would have been greatly welcomed news for both myself and other BSD users. After all, for the past several years, getting Java built and operational on our operating system has been much akin to pulling so many teeth, and the results were almost always wholly dependent on how well you crossed your fingers.

So the question now is this: What does it mean for BSD developers? We’ll no doubt soon be seeing JDK binary packages and complete source trees available via ports, but after several years of incredible platform bloat and withering industry attention, do BSD developers even care about Java anymore?

Over the past year, I’ve spent most of my time coding system-level stuff in pure C, and it’s quite obvious that other developers have latched onto other languages like Ruby. Like those ex-lovers who refused to give us back our LPs, is Java dead to us, or is there still hope for a rebirth?

In the most blatent misuse of the Web 2.0 meme, Sun wants Web 2.0 entrepreneurs to buy Sun hardware to launch a new web company!

I realize that this is way off topic for what I typically write. But this is an honest question. It’s not a troll. It’s hopefully not flamebait. And it for sure isn’t an attempt to push my political bias. I want anyone who is interested and can cordially formulate their thoughts to voice themselves, whether you answer the question “Is a Democratically Controlled Congress Good for Tech America?” in the affirmative or in the negative. Feel free to point out failings of the previous Congress and point out how the new Congress will do better. Or point out the strengths of the previous Congress and how the policies of the new Congress will be detrimental to the technological well-being of the United States.

Here are some guidelines:

1. No bashing of any person or political party. I’m not in the habit of editing or deleting comments, but I will remove any bashing comments.
2. Try to focus on the technological implications of this new Congress. I don’t want comments specifically of the economy, taxes, abortion, religion, etc. unless there is a direct tie to technology.

I’m really interested in reading your comments.

I’ve never done any J2EE programming. However, I have written business-critical code. I’ve written my fair share of LAMP code, and I’ve written enough Java code to last me the rest of my life. It’s no secret that I prefer Perl to Java for web development (and just about everything else).

I appreciate that addressing practical problems allows thoughtful developers to identify common problems and extract patterns and convenient abstractions to make solving similar problems easier in the future. I expect a mature language and platform to allow and encourage that.

What if all software patent applications had to include all of the relevant source code?

If you don’t get fat checks from an enterprisey company for your software work, should you care what they do and why?