Related link: http://blogs.tedneward.com/2006/03/12/My+Kingdom+For+A+Good+Macro+Language.aspx
First off let me say it’s great to be on the O’Reilly site as a blogger and I hope this first entry is thought provoking. Welcome and I hope you’ll subscribe as I plan on being prolific in this blog. I’ve been doing OO development for almost 20 years now and hopefully I can share an opinion that has value to you. And with introductions over, lets get on with our first entry!
Ted Neward yesterday mentioned in his blog entry My kingdom for a good macro language! some points around the idea of us needing more features for Java, specifically:
- A MOP
- A better language for generating byte-code for the platform
- Macros for Java - Not preprocessor macros, but fully language level aware macros
His points led me to these questions: What do you get if you add a MOP to Java? What will allowing Lisp style Macros do to Java? Is there a better language to generate byte-code?
What do you get if you add a MOP?
Today we have some level of meta capabilities thru the reflection abilities in Java and the several Aspect-Oriented tools out there. (AspectJ, JBoss AOP, Spring AOP and others) But those clearly aren’t a MOP. A MOP is about fully exposing the languages core concepts and implementation thus allowing the language to be extended at well defined places and allowing us to provide new capabilities and extensions previously not thought of. All of that while still maintaining the same basic syntax of the language. Meta-objects and Meta-object Protocols. Aspect-Oriented programming tools give us some capabilities to reposition our behavior and to cause it to be called in the correct situations, but fall short of a true MOP. Java itself has class objects (they don’t do much, but we have them), we have Dynamic Proxies and we have XDoclet and byte-code modifiers. All this to give us the ability to work with the code and meta-system.
What will allowing Lisp Style Macros allow?
Ted mentions OpenJava as a tool that allowed macros of similar power to Lisp macros to be added to Java. This is very interesting work and you should read the papers on that site for a good understanding of what they were working towards. There is a point in the paper OpenJava: A Class-Based Macro System for Java, that I disagree with concerning the power of the Lisp macros to be customized for specific types, but lets just say that you can get that same effect using a combination of macro system with and on the Lisp MOP. As for other macro systems for Java, I don’t think we’ll see one worth working with without a MOP of some kind. If you read this paper, they basically describes a simple MOP that is able to manipulate a program at compile time and then save that meta-data in a separate set of byte-code as the JVM doesn’t need that extra info to run. But notice, that they have to separate that description and also that they save some of that data in little strings in the classes. Ted doesn’t mention whether he wants the MOP data around at run-time. But for us suffice to say macros are about allowing us to have code that manipulates code that we have and either modifies it or creates new code.
Is there a better language to generate byte-code?
On the better language for generating byte-code he doesn’t really reference anything besides the work Microsoft is doing on the CLR and CLI stuff for .NET. He does mention them as more language friendly but does mention that it mostly has the same drawbacks as the Java platform. I conjecture that the byte-codes lack of full power is part of the issue. This is why SUN is going to add a new byte-code to support scripting languages.
There was a push both internally and externally early on in Java’s history before 1.0 was even out, to extend the byte-code in concert with Parc-Place Systems (an old Smalltalk vendor - some of who’s employees went to SUN to work on Java) to support more of this dynamic capability including MOPs, but choose not too.
So what do we get really if we add all that?
I conjecture one step “Back to the Future!”. Ted mentions one of the single most powerful languages ever invented right at the start. Lisp. He then mentions one of it’s most powerful features. The code as data - data as code idea. Lisp almost from the start had this idea that you can manipulate the program. It’s 40 years old in the Lisp incarnation. Lisp 1.5 was based on S-expressions (denoted in Lisp by those parentheses you may hate). The big idea being that the language itself was self describing. The compile parser and run-time parser deal with the same representation, as do you! Love the S-expression! That’s part of it’s power. The MOP of Lisp really is only for CLOS, not for the Lisp language, because the Lisp language already has a great meta-level capability in it’s READER and it’s macro system which Ted alludes too. But can we get as much power and simplicity and expressiveness by adding a MOP and macro system to Java?
I dare say we can not! But we may get closer…
Lets look at Ruby as he mentions it in his blog. He shows how Ruby using it’s meta-level features allows what appear to be new keywords to be added to the language. These are attr_reader and attr_writter. But he then calls them terse. You can’t have it both ways.
If you want full MOPs and macros, then calling the Ruby terse is negating your own request. All macros will be terse in that situation. This is akin to the argument we saw a few months ago about minimal API’s versus humane API’s. People have no problem creating all kinds of utility helper classes and methods, but doing it thru some meta-level operation is somehow terse. The point clearly defeats itself. I guess something that looks like a function call is “OK”, but something that looks like a language extension is not. Well, put your minds at ease because the Ruby that is mentioned is exactly that! The Ruby is just some method calls that are standard, but are implemented via the meta-level layer in Ruby and provided via that meta layer.
Ruby is not typed, making code generation easier. The thing to notice though is that in the Ruby code (you’ll have to follow the link to the Person class he mentions to see the full code), method calls can happen anywhere! Those attr_* methods are called right at the top. There’s really no special place for them. In Java things are not so simple! Lisp has the same thing going for it. Function calls can be done anywhere! This has to do with the fact that the language definitions for Ruby and Lisp are simpler than the one for Java. The fact is they have less syntax and less syntax actually provides more power!
While Ruby and Lisp have less syntax they can both appear to add new capabilities to the language itself, not just a new library. Today people are doing that with the tools mentioned above and are now starting to do it with the new Annotations added in Java 5. Aren’t those terse too? They certainly added new syntax and more special rules about when and how they work, etc. Compare that to just a simple method call. Seems the method call is easier to understand, debug, put in to the system in the first place, more pure, etc.
Why is that it seems more pure for Ruby and Lisp? Because the languages were built with the idea that a rich meta system was important. Java’s concerns were different and coming back and adding something as intrinsic as a MOP and macros is going to be difficult to do. And adding more syntax creates more special cases and rules etc. that makes your MOP much harder and the macro system more complicated. Not only that, but the code manipulator will me more complicated too! But does that matter to us if we get the MOP and the macros? I think it does and here’s why: If we expect to use that same MOP and macro system, then we want it to be nice and clean and easy to use and understand and without all kinds of exceptions and special cases. Think Java Generics. Lots os special syntax, special cases, etc. Java is now much harder to read. Terse if you will!
So the Java world is now going in lots of directions it seems to me. 1) We have new features being added to the language at a glacial speed. 2) We have new wants and desires being expressed for more power at the language level itself for programmers. 3) We have the want for simplicity, readability, etc. 4) We have the people who want only one way to do something and the people who want more than one way to do some things (minimal vs. humane).
Underlying all that is a language that was created for remote controls, set-top boxes and interactive television! A language with a syntax to appeal to C and C++ coders, networking and security being the prime concerns. A language whose least concern was a meta-level self description!
Instead of adding a MOP etc, we’ve seen Java add what the C++ community added. Java has taken a similar track to that community and it’s not surprising. Most of the converts were old C++ people. Not people who used and programmed highly meta-level systems. But now we as programmers have been influenced by the more dynamic meta-level systems community and the power of those systems that we lust for.
We’ve used all the patterns to try and get close to what we really want, we’ve added Aspects, we’ve used refection and dynamic proxies and byte-code manipulation….. but what do we really want….? No really, what is it we really want?
What do we want?
To me its as clear as the nose on my face… we want a totally new language that we don’t have to learn, that gives us all the power we already have, can use our current libraries and provides us all the power of yester-year and still is flexible enough to add new features at any time!!!
And no I’m not kidding! To add a MOP and macro system and a different code generator, we are basically talking about a new language. Maybe a familiar syntax will still be kept, but it will be a new language. Much as Java 5 really is a new language. Once you start using it’s features, you program differently, you solve your problems differently…. it is a new language. It’s changes are that significant. No set of Java features before I would say that about, but Java 5 was. And it’s all due to the Annotations and Generics. Terse is being kind here.
So if we’re going for a new language why don’t we start with the meta-level stuff being in from the beginning along with security, networking, multiple language friendliness, a better byte-code set, a MOP, macros, and all the other stuff we’re hearing people want now!
Or maybe we’re not going to have a new language…. because we don’t want to leave behind our frameworks!… How sad a choice that would be. A new language would empower us to make much better frameworks with out the limitations and the code bloat we see today to just so that we can simulate those powerful features we want.
I’d also suggest that we also add optional typing to the mix. We have a perfect example of how to do it. Again it’s Lisp. Lisp itself lets you specify the type of a variable or object optionally, it also lets you specify a safety level and a optimization level allowing you full control over both type safety, runtime safety as well as how much it will try and optimize around that variable to gain speed/safety. Lets use that so that we can have a language that is both a fully generalize language good for full systems development as well as for scripting. And let that safety be linked in to the secure environment, etc.
We have as programmers and developers a great example of the things we seem to be clamoring for Java to add and it is Lisp. Code is Data/Data is code, full macro system, fully controllable typing system, customizable parser (the reader), a MOP, self aware system, highly efficient code generation, able to operate in compiled or interpreted mode, simple syntax, etc. (You’ll find all these in Smalltalk as well)
Ruby is a nice partitioning of some of those features, and you can see how hot that community has gotten over it. Java caused the same stir in it’s day with garbage collection, networking libraries standard, threads standard. But no language right now has all that we seem to want. Not even the .NET CLR/CLI set of tools and languages.
We need a new language….. or we need to “Go Back to the Future”. I myself, think going back to the future by building on Lisp or Smalltalk is the shorter path, but I know that the syntax is a big issue. Lisp has always been a ball of mud in that it could take on anything and still look like Lisp even though it now had all kinds of new capabilities. CLOS being a perfect example. But because I’m enlightened and experienced and I’ve seen history over these almost 20 years I’ve been programming, I know that the majority is what makes a new language stick and for that we will need a “New Language”!
In closing, any time you want to dump on Lisp or Smalltalk or some of those “other” languages, look closely at them again. Then create “Hello, World” in Java and Swing (9 Meg just to print “hello”) and then the other language. Then look at what all you need to run a J2EE app server vs. something similar in one of those other languages, look at there web frameworks, etc. Then compare how humane there libraries are verses Java’s. Finally, think that some of them can do all the same stuff we do in Java in about 1/8th the RAM and code size and for the most part they are just as stable and scale just as well as the Java tools do. Then think why? Can we get to that point with a “New Language” You bet we can!
Having said all that, Java isn’t really bad… We are as programmers growing to want more power than ever before and it’s leading us the same place it lead those researchers all those years ago. Kinda nice to know that isn’t it? It’s also kinda nice to know that we have a community that is pushing the envelope and pushing the language vendor to add new power all the time.
Here’s hoping I’ve stirred the pot a bit….
Till next time - Thanks for reading….
Please let me know what you thought and your viewpoint on MOPS, macros and whether you think we need a new language.
