A lengthy thread on the Perl 5 development mailing list has asked if there’s self-generated FUD about the future of Perl worth debunking.
I’m not the person to ask about the viability of using Perl 5 for new development projects in your own environment (and I can’t wait for Perl 6), but I had an interesting thought about how to gauge the suitability and evolution of a language today….
How Does a Programming Language Stagnate?
A new question arose yesterday in the monster Perl 5 Porters thread (see my Come Back, Zinc!). Given the increasing length of time between stable major releases, do users think Perl 5 is stagnating?
The question is more complex than it sounds at first. For example, who is this mythical pluralized user? What kind of work does he or she do? Does this user follow p5p or Perl Monks or read Perl.com? What kinds of features does he or she use or want to use? What level of involvement does this user have in the system?
(The always-imminent Perl 6 has a further effect on the question. Will Perl 5 be a legacy system soon? For how long? Then again, given how much Apache 1.3.x and PHP 4 and MySQL 3.23.x and even Perl 5.004 and 5.005 are still around, perhaps that legacy question isn’t as important to the users we ought to be considering. Maybe this is the most important question, though.)
Which Part of the System Actually Matters?
I immediately objected to the idea. My brain immediately said, “You’re a smart man, person who posted this thesis, but there’s something really wrong about it.”
Why?
I had to think about it for a while. (My poor roommates heard the whole story though, but that’s what you get for walking around in my house.)
Something Sylvain Wallez said in On JavaScript libraries struck me today. He suggested considering libraries as levels of abstraction.
Of course, everything in programming is about creating, using, and managing levels of abstraction. That’s the platform, the core language (syntax and semantics, possibly even idioms — but nothing more), the libraries, and the tools.
Striating your language into these levels brings up several questions:
- Who develops at which level?
- At which level do most users concentrate their use?
- Which is the most appropriate level for a new feature?
- How well integrated with the entire system is a feature at each level?
I suspect that the level at which most developers write the most code also dictates at least some of the utility of the entire stack as well. (That is, if everyone’s building developer tools, who’s building end-user applications?) That argument might go the other way too — the level at which most users need to use the stack dictates the level at which they need to work. It’s difficult to discount the abstraction capabilities of the language, though. (Would Java have as much tool support if its syntax were more malleable?)
You can’t discount the purpose of the stack either. The GNOME and KDE and XFCE projects have different types of developers and different types of users than do programming languages. Yet I digress.
Where Does a Language Grow?
I’ve long argued that abstraction and extensibility are two of the most important characteristics of any whole programming language. (Often I’ve argued this in terms of maintainability, so the point there is indirect.)
Consider C. The language has a small syntax and a reasonably well-defined semantic model (measure K&R versus Java in a Nutshell part n). There’s little evolution in the language or its semantics (okay, C89 and C99, but how long until you can rely on working cross-platform and cross-compiler implementations of the latter?). The only reliable (macros don’t count) extension mechanism is creating shared libraries.
Fortunately, C is portable enough that well-written shared libraries are reasonably easy to rely on. Besides that, if your users are “people who care about programming in C”, you can assume that they have the ability at least to call C. (It gets better that C is easy enough to link to from almost any other language. That’s further growth.)
Languages such as Perl and Python and Ruby have it both a little easier and a little more difficult. These languages don’t need a specific compilation step in the sense that, once you have the runtime on your system, you have what you need to run most of the code written in those languages. There’s no distinction between having the Perl compiler and the Perl runtime, as there is with C and C++ and even Java.
That has two opposite design effects. First, it’s easier to push some of the complexity and extensibility of the whole language into libraries. Second, it’s tempting to pull that complexity into the core language layer to make further extensions easier or even possible to produce. It’s probably even the right approach, much of the time.
Distributing pure-language applications can be easier because, in theory, the runtime abstracts away platform-specific behavior behind a useful interface and, also in theory, pure-language code can run anywhere. There are obvious exceptions, but I suspect this is frustrating enough when it doesn’t work because it works most of the time.
Non-pure extensions are much worse. If users only need the language
runtime, most of the time, can you rely on them having the compiler for
another language available? It depends on the user’s strata. Perl 5 porters
have a working C compiler set up to compile Perl, or they can’t do much
useful. (Okay, I exaggerate slightly. They at least have a
make utility and sane shell settings. That’s the requirement
for being able to patch and test the core libraries.)
I wonder. Does a language grow despite the difficult parts? Even though distributing compiled extensions is difficult in comparison to pure-language extensions, are people using them?
That’s mostly rhetorical.
When and Where Do Users See Stagnation?
If I submit a patch to p5p to add a new feature, when can I rely on that in code I want to distribute to users? It depends on the users… but it will be a couple of years. Maybe the bleeding-edge Linux distributions will pick up Perl 5.10 within several months of its release. Maybe not. The BSDs will come along soon.
Yet will people running stable applications that need to stay up and supported for months and months upgrade any time soon?
My experience says no. (It’s a pity too, because I want to use that new feature now.) Again, I point to Apache 2, PHP 5 and PHP 6, and even Perl 5.8.8.
Could the idea behind the levels of abstraction work for measuring the level of involvement in the language? Some users always stay on the latest stable version, but my theory is that these levels form an inverted pyramid. At the base (or tip) are people such as the various Perl 5 pumpkings and Larry. As you move up the pyramid, each level has more users until you reach the wide-open Internet, with millions and millions of people using sites and applications built with Perl. They just don’t know it.
If the propagation of a new version of Perl throughout the world reflects the distance of a user from the tip of the pyramid. I expect someone still running Perl 5.004 to know and care very little about new features of Perl. If there has been no reason to upgrade in the past decade, why upgrade now?
It’s not just people tied to legacy platforms, though. I installed Wine on my desktop machine a couple of years ago and haven’t used it since. I don’t use it often, I don’t use it heavily, and I don’t have any real reason to upgrade. I suspect that applies to a lot of languages as well. (I’d have used Java as an example, but people might figure out it’s my whipping boy.)
Do new features in Wine matter to me? Not until they matter to my use of Wine. It’s a fine project and I’m glad it exists, but even if the pace of change did slow down for a while (and my impression is that it hasn’t), it wouldn’t affect me greatly.
More importantly, my impression of Wine isn’t that important, especially to the Wine developers. I’m not a target user.
The People Whose Perceptions Matter
Whose impressions matter? Target users.
Who are those target users? That’s a big question.
If my theory of layers is right, far fewer people will care about the core language than the libraries. In a well-designed system, you also need far fewer people to maintain the core language than the libraries. (That may be why we call such a system well-designed; it meets the reality.)
Attracting people to do that is useful and necessary. Yet for every one person you can attract to hack on and maintain the core language, you probably can attract a hundred or a thousand people interested only in doing useful things with the whole language and libraries and the platform and tools.
I strongly suspect the health of a language and its perceptions depends more on those thousand people than the one, though without the core hackers, a language must eventually converge to a stable implementation or just die. (There may even be a difference.)
When someone talks about a language stagnating, though, there’s an implied audience. Who thinks it’s stagnating? Why? Do their impressions matter? Is the impression accurate? Is it worth contesting?
I’m not sure how to answer those questions With regard to Perl 5. Most of the evolution of the whole language takes place on the CPAN, and the rates of development and invention and change keep increasing every year.
That doesn’t matter much if people at levels above the libraries can’t or won’t use them though.
There are still language changes and enhancements going into Perl 5.9.x (which will become Perl 5.10) and I’m anxious to use them… but I’m not sure they will make the people above the library layer rush out to upgrade. That’s not bad, they’re just not fundamentally different features. They’re mostly high-end enhancements, with a couple of notable exceptions.
These are all good things, though. Perl’s evolution shouldn’t have to wait for Perl 6 or the release of Perl 5.10 or any other single point of failure or control.
Yet there must be mechanisms for users who don’t want to have to understand how to set up a compilation environment on their platforms and to slog through several different but similar modules to find the right one for their purposes, mechanisms by which they can get the benefit of this evolution without having to manage it themselves.
The same goes for any other language.
Progress Happens
If there’s a perception problem, perhaps it’s because people are looking at the wrong thing and expecting the wrong answers. Perhaps Perl end-users need to concentrate on the work of distributors and framework and tool builders, rather than the comparatively raw, low-level output of the core builders.
If that’s so, maybe there should be a way for these groups to communicate. Maybe the core language is converging on its ultimate set of features and its ultimate steady state of stability. Maybe it’s temporarily reached a false maxima. I don’t know. I do know that evolution and growth is not slowing at the edges.
Is Perl dying? There are far more and often better tools and libraries available right now than there were a year ago. That seems like progress to me.
Now the trick is demonstrating that — and, just maybe, moving a few people to deeper layers, bit by bit.
One thing you haven't mentioned is perl jobs and projects..
I've been looking at www.jobstats.co.uk, and it shows a steady growth in perl both the number of perl jobs, and the share of perl jobs.
I've also had more freelance work than I have time for, and am working on a new mission critical project handling, scheduling and providing data feeds and briefings for pilots and airlines. Rather than moving from perl to another language, all this work is either new or replacing existing perl or PHP with new Perl.
One thing that I've noticed is the presence of Perl code that hasn't been updated at all for 2 to 4 years, and is often 6 to 8 years old - there has been so much progress in last 2,4,6 and 8 that it's possible to replace hundreds of lines of code at a time by updating to modern modules and ways of doing things.
I've also noticed more standardisation appearing than the perl community is given credit for. Particularly among developers I work with on a freelance and contract basis : those that are involved in the perl monger community are using unit testing, packaging their modules to pass Kwalitee test, using CPAN modules more consistently, and following better practices.
I think that the increasing number of workshops, conferences and technical talks, as well as the high quality of articles and books available together with projects to normalise and standardise some areas of CPAN like Date/Time, Email, etc are having a really healthy effect.
I don't see any stagnation here, I see that Perl is still the cinderella of languages, doing the hard work, while the ugly sisters get all the attention ;)
Good points, Teejay.
I wonder if the better development practices you mention are spreading throughout all of Perl development, or if the kinds of places that hire good freelance and contract developers have sufficient community connections that they learn and follow good community practices soon after their discovery.
Hi chromes,
You're right, the perception of Perl "languishing" is due to some shortcomings well outside the core language. In the web app arena, it's self-inflicted.
Rails is hitting a sweet spot with "convention over configuration" with regard to Java (and all its frameworks). Perl programmers naturally reach for Catalyst, hoping for the same "programming within restrictions" experience where convention has already sorted out the best/easy way to get things done.
Then we find that Catalyst is nothing like Rails -- too many choices. Do I really want to hear how flexible Catalyst is? No, I want to hear how simple it is -- that's what frameworks are for, right? Simplicity, not infinite choice.
Something "canned" like Rails just feels so smooth compared to the choice-heavy Catalyst which makes me stop, think, choose and question whether I made the right choices before I can get working. That's annoying.
It reminds me of the old office software debates about "best in breed" vs. "suite". Rails is an all-in-one suite, Catalyst is an "a la carte" menu of choices.
A lot of programmers don't get going on something until they thumb through an O'Reilly book on the subject. Unfortunately with Catalyst, the book will be 900 pages long, and the first half will cover configuration like choosing which templating system, object-relational mapper etc. Bleh.
What ever happened to the notion that Perl "just works" and "understands what you mean"? The language is great, we just need a more monolithic framework.
Paradoxically, much of the resurgent Perl 5 development can be viewed as evidence of stagnation, or obsolesence. (I tried to find a less inflamatory term there but failed; I'm really trying to imply something less drastic than "end of life" and more like "senior citizen".)
As a long-time Perl developer, I find the latest Perl6:: modules and the similar "convenience" modules very helpful, but what do they imply for new developers? When new people complain that certain tasks are onerous, and a Perl elder tells them, "Hey, just use SUPER, NEXT, Attribute::Handlers, IO::All, IO::Prompt, Perl6::Say, Getopt::Euclid, and Perl6::Rules and your code will be so much leaner," what do they think? Where's the coherent pattern in this mix of modules, or even the roadmap? You have to either get PBP or be plugged into the Perl community fairly well to be in on that news. It all looks great to us who are already there, but it forms a rising barrier to new entrants who look at, say, Ruby, and realize that they can have much of the same ease and simplicity they'd get with the spiffy (or Spiffy) Perl modules just by reading the one little Ruby book. Like Ptolemaic epicycles, these patches on Perl 5 could be viewed as attempts to patch up an inadequate foundation.
That's harsher than I'd like it to sound but I want the point to be visible. It's the reason why I've been waiting so eagerly for Perl 6 for so long, because I see it as the solution that brings more fresh blood into Perl. I have a little nightmare: We've probably all experienced some aging guru at the workplace who was married to some ancient language. People will be talking about how to write, say, linked lists in C, and this person will say, "Hey, that's a piece of cake in RPG/COBOL/JCL" and go off and do it in fifteen minutes to prove it. And it works, but no one else can understand it or be bothered to try to understand it. And my nightmare is that we end up like that. Which is why I so much want Perl 6 to succeed.
Stagnation is a tricky thing to wrap the brain around. How do empires fall? With all those resources, they should be able to last forever. But they never do. Stagnation isn't a uniform affliction; it starts in one place and works its way to the others like some sort of digestive process. Recognizing it at the outset is the hard part.
That's an interesting point, Peter. To me, it sounds like the recurring debate over core and modular features.
It's a shame, for example, that Perl 5's built-in metaprogramming facilities are so clunky. Ruby has a clear advantage there.
However, when it comes to finding and (often) installing extensions, Perl 5 has an even larger advantage with the CPAN. It's difficult to argue with over 10,000 available modules -- but if you don't know what you want or how to find something that takes away the pain, is that really an advantage?
Like you, I look forward to better defaults in Perl 6. That's clearly one necessary improvement.
I intended a more important point than revisiting the core/modular split (not that I wouldn't rather have io() and say() in the core than getgrent() and shmctl(), but not to get sidetracked...) Modules like SUPER and NEXT are pragmata designed to make Perl behave the way we (for large values of "we") think it should have behaved to begin with. Attribute::Handlers is the same; who's going to try and do much with attributes without it?
Then there's Class::Std. It is my understanding that aside from inventing something so powerful, a major reason for creating it was to provide a standardized method for writing O-O classes so we could end so much time wasting caused by the dark side of TMTOWTDI. I've been giving presentations on Class::Std and telling people to just use it so they don't have to spelunk around CPAN and read numerous articles to try and figure out what they should be using, but many people seem to want to keep the debate going, and I think that hurts us when people can go use languages like Ruby that haved moved past that.
So you've got to be clued into which "hip" modules to load just to get Perl behaving the way many people think it should. The standard documentation doesn't have a roadmap leading people in this direction; they pretty much have to dig around and keep up with the latest Perl news and books. I've got no problem with requiring people to be clueful to join in the game, but learning what amount to arbitrary incantations doesn't accomplish that.
Of course, Python and Ruby will go the same way after a few years if they're as successful as Perl, because this richness is a mark of maturity. (I'm looking forward to rubbing a few smug noses in it when it happens.) But it's also a mark of a legacy system. And the only cure is reinvention. Perl borrowed from many languages and improved upon them; now Python and Ruby borrow from Perl and add their own improvements. The only answer is to innovate. Jefferson thought there should be a revolution every generation; same here.
Yes, Perl is ahead - considerably - in areas like CPAN, but as I was alluding before, stagnation happens nonuniformly. The Persian campaign can be expanding the imperial frontier at the same time the citizenry is wobbling around from lead overdoses.
I would take a different tack if we didn't have Perl 6 coming up. I am convinced it will take the programming world by storm and I'm just frustrated it's not released yet.
I still use Perl for my sites (brooders.net, neoscholar.net), as well as tidbits of code, prototyping (even if the app will end up being in Java) and a whole host of things.
My work bought me Komodo from Activestate so I could do my Perl and Ruby coding when needed. I agree that more of the focus now is on modules like CGI::Application, Catalyst etc etc and not so much in the language itself. Yes .. there is a big gap between 5 and 6, but perl 5 is still useful.
I am also a little embaressed to admit it, but I probably still largely have a coding style like it's perl 5.005_003. I first learnt Perl 4 at university, then upgraded from 5.001 to 5.005_003 in my first job. Perl 5.6 came and I didn't change that much, even with Perl 5.8 .. same thing.
Perl still fills a gap. If need to grok a log file, do some web scanning (WWW::Mechanize is cool) or prototype something, Perl still helps me out. I do like Ruby as well, and Agree the whole Rails thing has made web dev easy in that environment compared to CGI::Application with Class::DBI, but you know what? I still maintain an Perl Web App I wrote 6 years ago .. with a homegrown templating system, and it still works.
I think the whole "perl is dying" thing comes from people seeing other languages and frameworks in the limelight. I think one of Rails biggest successes was a build a web application in 15 minutes tutorial in 15 minutes, and watch the video if you can't be bothered doing it .. and it looked pretty. C still has its place in microprocessor programming and OS level stuff. Perl also has it's place as well as PHP. The whole Sapir-Whorf hypothesis (which Ruby language creator Matz talks about as well) talks about language influencing thought, and perhaps our thoughts influence the programming language we want to "think" in.
I think the language grows through the modules, and it's still growing. Like you mentioned in the comments, CPANs modules for Perl5 are a huge strength for the language.
I think this pretty much sums up the problem with perl
http://www.feld.com/blog/archives/2006/10/php_vs_ruby.html
Dramatic decrease in new projects and new developers.