In comments on Sci Foo 2007, Tim O’Reilly wondered if giving Larry Wall a research job may have reduced his immediate practical concerns influencing the development of Perl 6.

I interpret the line of thought as “To what degree does the lack of a single well-defined problem to solve influence the concreteness or the abstractness of the solution and its delivery date?” (I won’t claim that that’s Tim’s line of thought, but it’s the question I’ve pondered this past week.)

Constraints are Good

I’m in favor of constraints.

One of my proudest creations outside of programming is a 300 word horror story. I was proud to write it. I was prouder to edit it, removing 10% of its length (always a good goal in editing), and then to add just enough new material to return to 300 words. It’s stronger for the editing, and even more impressive that I hit 300 words exactly both times. I even spent a nice afternoon writing a microfiction assignment generator which assembles a random group of constraints into a short daily writing assignment.

You can take constraints too far, though; when was the last time you read a good sestina? When was the last time you read any sestina? (How many of you know what that word even means?)

In creative writing, constraints can be liberating. Take today’s microfiction assignment. “Given the character of a talking frying pan…”, you have a character. You have a skeleton on which to hang a story. You know what a frying pan is and what it does. You have hints as to what it considers important, how it spends its day, and how it might talk. Also, “Your word limit is 240 words,” or a couple of paragraphs. That’s a few sentences. If you fail, you’ve spent a few minutes thinking creatively and writing a few sentences. There’s almost no risk.

Constraints also focus the mind. In my horror story, I had to weigh every word carefully. I knew the story I wanted to tell. I had a distinct theme. I wanted subtlety and irony. I had no room to wander, to circumlocute, or to distract. Every word had to be there for a reason. I had to find lightning rather than capturing lightning bugs and calling that good.

Of course, those constraints are all artificial. I admit that. I chose them on purpose for there benefits.

Constraints in Software

Software projects also have constraints. Some are even artificial, but most aren’t. You probably have customer demands to satisfy. You likely have a limited number of available programmers, a finite amount of time, and exhaustable resources. You probably have to work with other people who have different visions of what’s important and what the eventual successful outcomes might be.

Sometimes these focus the mind. Sometimes they don’t. I’m not sure why it can be difficult to communicate the idea that the pure thought-stuff of software, while much more malleable than physical construction materials, for example, still has peculiar gravities and inertias. Perhaps we programmers are better at demonstrating the magic of one- and two-line changes which produce stunningly good effects than we are the weeks and months of experimentation, thought, and redesign which go into making such achievements possible.

Perhaps we’re bad at managing constraints–even so bad as to recognize them far less frequently than we should.

I really like the observation in Extreme Programming that there are four dials you can manipulate in software projects: Time, Scope, Resources, and Quality. (I avoided the possibility of increasing or decreasing quality in Extreme Programming Pocket Guide, but Jim Shore set me straight on that. Sometimes you don’t need a complete mathematical proof of correctness of your entire system. Sometimes a one-shot prototype with an installer that only works on the one machine you’ll ever install it on suffices.) If you’re running short on one element, you have to adjust the other three dials appropriately. If you have an excess of one element, you can reduce the pressure on the other three dials.

It’s not a guarantee of success, nor a mathematically-precise model of reality, but it’s a good reminder that a Work requires a combination of factors. Someone must finish the work with a given amount of resources. This will require a specific period of time. It will have a measurable level of quality.

Constraints in Volunteer Projects

How does this relate to Perl 6?

Perl 5 has been a practical success. Try imagining the ’90s web without it. (Try imagining the 21st century Internet without it. A lot of projects millions of people rely on would disappear. Constraint #1 is that Perl 6 oughtn’t cannibalize on the success of Perl 5 without being substantially better.

Fortunately, the fanatical devotion to backwards compatibility exhibited by Perl 5 makes it a tempting target sometimes.

Perl 5 has lasted for nearly 13 years. That’s a long time. Fortunately, the language design has enabled evolution in a lot of areas (if imperfectly in some). CPAN is the best example. The internals are probably the worst example. Constraint #2 is that Perl 6 can succeed by enabling evolution in future directions. (Larry likes to say that the killer application for Perl 6 is something that no one has invented yet.)

Perl has always been quirky, even among programming languages. Plenty of languages used sigils before (some well and some very badly), but Perl hooked them into the type system. (With that, I’ve confused just about everyone in the world. Think about it.) Even if you throw out the pro-golf, pro-obfuscation coterie (and you really should–just smile and nod as you back away while keeping your hands in plain sight), there’s still a discernable sense of Perlishness in well-written Perl code. Constraint #3 is that Perl 6 should retain that sense of Perlishness, whatever other changes occur.

Some constraints are less philosophical.

By my calculations, Perl 6 has had just under three fully-funded person-years of development. That’s 6000 hours, split between Larry, Damian, Dan, and a handfull of microgrant recipients.

The 2000 developers on Microsoft’s Vista spent that much time reading e-mail in any given week.

Constraint #4 is a staggering reliance on volunteer labor. I have nothing but praise for the volunteers on various Perl 6 projects. I merely mean that practical matters like working at my day job to pay the mortgage, spending time with my family, and sleeping have sometimes taken priority over writing just one more line of code–and I’m not the only developer who’s made that choice occasionally.

It would be nice if there were a pool of Perl-friendly C developers ready to wade into a codebase like Parrot’s. Likewise, Pugs would benefit from the attention of a handful of brilliant Haskell hackers. Yet no matter how easy either project makes it to donate an hour a week, no matter how beautiful the code or clear the requirements, someone still has to sit down to the glow of a text editor and write one line of code at a time.

Plenty of quotes describe research and inspiration as long, dry desert treks punctuated occasionally by brilliant flashes of insight by cool oasis pools. Sometimes you have to slog down several wrong-way dark alleys before you realize that the correct path lies elsewhere. Constraint #5 is that, when inventing the future, sometimes you have to wait for the future to catch up to you. It took the widespread availability of certain features of the Haskell programming language (and some GHC enhancements) to make Pugs possible. The invention of PGE and TGE within Parrot made writing compilers much easier, much as the integration of IMCC made programming Parrot simpler as the development of NQP should make the Perl 6 compiler even that much easier to write.

Sometimes you have to stop to build your tools before you can build your machines.

Embracing Constraints

Kicking against the goads is a good way to end up with a limp. Sometimes the only way to make progress on an impossible task is to realize exactly how impossible it is and to sneak up on it from a different direction.

That’s why there are parallel implementations of Perl 6, for example.

That’s why Parrot generates a lot of C code that we could write (and worse, maintain) by hand.

That’s why the Parrot compiler toolkit is so important. Sure, lex and yacc are good tools. They’ve been good tools for decades, if you want to write another compiler in C.

Sometimes, solving difficult problems requires brilliant new ideas. Sometimes those new ideas are stepwise refinements of older, good ideas. Maybe they’re radical rethinkings in other ways.

Still, the best way I know of to get something done even in the face of constraints–artificial or otherwise–is to accept them as fait accompli, then ask “What if I really didn’t need a thousand words to tell that story? What if a hundred really would suffice?” and then just work toward that goal.

Maybe you’ll have to turn the Scope dial way down, or the Time goal way up. That’s the nice thing about software–and words. It’s almost infinitely malleable.