Ted Neward attempted to pull apart some of the silliness in the debate over scalability with Can Dynamic Languages Scale?. In particular, one of the most important insights is:
There’s an implicit problem with using the word “scale” here, in that we can think of a language scaling in one of two very orthogonal directions:
- Size of project, as in lines-of-code (LOC)
- Capacity handling, as in “it needs to scale to 100,000 requests per second”
While I felt the need to quibble with Ted on an unnecessary swipe at Perl at the very end of the piece, I agree that scale(1) and scale(2) are important but very different factors. They’re also not the only questions of scale.
If size-of-project is a good metric (and this should count both maximum and minimum size of project), then an important project management scalability question is “Can I hire or train enough good developers in this language or framework or platform to meet the project’s goals?” It’s a subtler question than it appears, though. Even setting aside questions of language productivity (which matters greatly), the expertise of the developers with the language, platform, libraries, and tools as well as in the domain of the problem affects your answer. It doesn’t matter if there are a million Blub programmers available if you want to do something in Blub that only three existing Blub programmers know how to do.
Downward scalability is likewise important. My head still hurts when I remember sitting in a meeting with a Sun system administrator who nearly demanded that Ward Cunningham find a wiki with an Oracle backend because, and I quote, “Flat files don’t scale”. (Ward had drawn a classic horizontal architecture with a cluster of read-only servers and a single write-then-sync-outward server.)
One of my favorite moments of tech schadenfreude was watching Java developers gradually react in horror as they saw exactly how little code and work was necessary to deploy the (admittedly stupidly simple) Ruby on Rails demo weblog application. Yes, it was stupid. Yes, it was simple. Yet it took a handful of seconds to make a change — you could build a very small project in an hour. Languages, libraries, platforms, and tools that scale down to small projects have advantages as well, particularly for encouraging energetic and dedicated novices to teach themselves how to use a technology.
(I spent a while installing a SAP server and development environment the past couple of weeks. The SAP server took a couple of hours to install and takes ten minutes to start, every time you want to start or re-start it. I realize it does a lot more than a simple Jifty or Catalyst or Maypole or CGI::Application project running on a SQLite backend, but guess what I’ll tell my nephew to use when he wants to build a web project?)
A final important type of scalability is from novice to experienced practitioner. How well does the ecosystem encourage people to become productive and to develop good taste in using the language’s idioms, patterns, and strengths effectively? This is not just learnability or the “intuitiveness” of the language’s syntax at first glance to complete novices.
I’ve rarely seen people address this question directly. In the Perl world, related questions often are “How can we overwhelm all of the lousy examples of Perl code in the world with good examples that use the strengths of Perl 5.10, not Perl 4?”, “How can we find the 90% of the million Perl programmers in the world who’ve never even heard of the Perl Mongers or PerlMonks or even Perl.com and encourage them to become part of the community for mutual benefit?”
This scale(5) is also subtle, however. Not all languages or ecosystems optimize for learnability and expertise in the same way. As a hoary example, Python deliberately eschews some advanced features in favor of learnability for programming novices. Idiomatic C++ favors pervasive template metaprogramming and genericity which isn’t exactly the first lesson I’d start in on after getting past syntax. Java recommends (requires?) the use of powerful tooling to promote a style of work that resembles directed code-generation.
These are all deliberate language choices or, at least, first-order results of deliberate language choices. Their effects on scale(5) are measurable, however.
One point often overlooked in academic language-bashing discussions of scale (and note the lack of a disambiguating number) is that scalability depends on what you need to do now and expect to do in the future. Disambiguating orthogonal axes of scale may help disentangle that discussion.
To summarize, then (re-ordered for clarity):
Unless I've misunderstood you, the last two seem awfully similar and can probably be combined.
Another excellent example of minimum scalability would be COBOL. COBOL was deliberately designed to require more work up front. This was done in the belief that by clearly specifying everything carefully in the "Identification Division", "Environment Division", "Data Division", and so on (and then duplicating a lot of that in the JCL), later maintenance would be made easier. Regardless of whether or not this is true, it does mean that one doesn't dash off a quick COBOL script.
Seems to be some overuse of the term "scale" here...
Each scalability dimension has multiple properties.
A. Maximum "scale up"
B. Minimum "scale down"
C. Linearity
If you take the two dimensions he calls Scale(1) and Scale(2) (complexity and volume) then you can apply those factors against each other.
Scale(1A) covers the ability of the codebase to grow, availability of developers, introspective analysis and refactoring toolsets, etc. Prime Example: Java
Scale(1B) covers the ease of creating and deploying simple applications, ability to be embedded in other systems, intuitiveness to someone without skills in that area etc. Prime Example: PHP (within it's specific niche at least)
Scale(1C) covers stuff like availability of tools that don't need to be used when a project is small but which can be added later (valgrind, perlcritic), training path from trainee to expert developers, support for modular development, etc. I can think of a great example.
Scalar(2A) is the ability to deliver vast amounts of $stuff. Covers things like determinism, manageability, monitoring capabilities and parallelism. Java and SAP are strong here.
Scalar(2B) probably covers embedding (low footprint) and customization (dealing with many different functions, each of which consumes no overhead). C and CGI.
Scalar(2C) is the domain of the O(x) algorithms and elegant designs. Believe of it not, Oracle is a standout example, since they sell on the basis of cost being related to demand. Amazon's cloud services are also big here.
Flat files don’t scale
Well, depending on what you do with them, they really don't scale.
Large text files (without any indexing assistance) must be sequentially read. If you're working with lots of small files must take into account the overhead of opening and closing the file. Also, a high number of deletions or re-writes is going to fragment your filesystem.
And the big one that everyone forgets is directory operations. A lot of filesystems will show degraded performance if a directory has over 2000 entries. Even simple stat operations would take longer.
You can't just say that a file is scalable (or not scalable) unless you detail the size of the files, the number of files (and layout), the operations performed against the file, and the expected response time.