One of the things that I so enjoy about watching the Mozilla Firefox development process is that they are not shy in pushing forward with technologies that many would have thought solid and immutable. JavaScript is a case in point. JavaScript had largely stagnated as a language from about 1998 on, until AJAX (spurred largely by renewed interest in the Mozilla platform) suddenly came out of nowhere. Accessors (getters and setters), more sophisticated array handling, E4X and other innovations have come out of the development process.

With the release of Firefox 2.0 beta 1, there are several new features that have been dropped into the language, features which are likely to be used sparingly at first but which offer a significant set of capabilities that will likely become welcome tools in any AJAX developer’s toolbox.

Yield to Understanding

One of the most profound new changes (and admittedly one of the harder to understand) is the generator construct. Generators are key in solving the problem of enumerations, in which you have a sequence of objects, only one of which has relevance or context at any given time, but they have many other uses as well.

The most traditional way of enumerating through a set has traditionally been to use an index sequence of the form

for (var i=0;i<10;i++){ var result = someFunc(i); }

There are a few problems with this. For starters, the index variable i typically has comparatively little semantic relevance. It’s an index variable, it indicates position in a sequence, but its primary purpose is to find the next item in that sequence. JavaScript does have alternative means to perform iterations, though they are generally locked up with specific types. You can use the for() and foreach() keywords with objects to retrieve all the properties of (most) objects:

for (key in document){
    print(key+","+document[key]);
     }
            

but this approach tends to be an all or nothing one - you will get all properties that the object exposes, including all inherited properties and those properties which are likely not relevant to you as a developer. The Array object is a prime example of this. Iterating over the object properties of an array will retrieve the numbered entries as keys, but will also retrieve the length property and all of the assorted array methods, forcing you back into a mode where you have to iterate over indices, then assign the object to the index.

Firing up the Generator

This is where generators come in. A generator looks a lot like a function, but differs in one important respect - the presence of the yield keyword. Think, for a moment of driving - you’re driving along when you see a large yellow triangle - a yield sign, indicating that you should stop, wait for traffic coming the other way, then proceeding forward once traffic is clear.

In essence, the yield keyword indicates that the function is evaluated to the line holding the yield keyword, the expression to the other side of the yield keyword is evaluated, and the result is returned to the calling function. However, unlike the return keyword, once the result is evaluated the code immediately after the yield statement takes over and the function continues until it either completes, or it encounters another (or in a loop the same) yield statement.

Once the generator ends, it generates an exception. Now, this may seem to be counterproductive activity, but in point of fact most pre-defined enumeration handlers within JavaScript do the same thing, and the constructs such as for (… in …) specifically look for this exception before terminating. More on that in a bit…

First, it’s worth providing a couple of basic examples illustrating how such generators work. As a simple example, consider a range statement, which returns an iterator that starts at the first argument provided, and terminates with the last argument:

function range(a,b){
    var i=a;
    while (i !=b){
           yield i;
           i++;
           }
     }
var r = range(5,12);
print(r.next());
==> 5
print(r.next());
==> 6
... // repeat until the iterator reaches 11
print(r.next());
==> 11
print(r.next());
==> [Exc: Stop Iteration]            
            

In this particular case, the variable i is the “index” variable. It is initially set to the value of parameter “a”, here 5, is then passed into the while() loop testing as to whether it has hit b=12 or not. If it hasn’t, the generator parks on the yield statement with the value 5, until the generator’s next() method is called, whereupon the function returns the value after it (here, 5), then iterates, loops around, and parks again on the yield (a lot like waiting to pass through the Canadian border, to be honest). Once you hit the point where the conditional is no long true, an exception gets fired.

In essence, what you’re doing here is taking advantage of closure to handle the iterations for you - and while there is iteration going on, the iteration variable stays hidden.

As it turns out, the for(in) operation actually works by calling the internal next() method of an Iterator(), the only different here being that until now that functionality wasn’t explicitly exposed to JavaScript. Thus, you could create a for(in) version of the above sequence with trivial changes:

for (key in range(5,12)){
    print(key);
    }
==> 5
==> 6
==> 7
..
==>11
    

Note that the for (in) statement doesn’t throw an exception here, because the for(in) keyword actually uses such an exception itself - all you’re doing is providing the interfaces that it expects to be able to do the enumerations properly.

The iterator here is not privileged - the yield statement does not in fact need to return the iterator at all, but can instead return something dependent upon the iterator. For instance, you can create an array iterator that can work with any array, by retrieving the value that the iterator itself retrieves from the array:

function $a(arr) { 
    var i = 0; 
    var k = arr.length; 
    while (i != k) { 
        yield arr[i]; 
        i++; 
        } 
    }
            
var colors = ['red','green','blue'];
for (var color in $a(colors)){
    print(color);
    }
==> 'red'
==> 'green'
==> 'blue'            
            

(I’m using the notation $a() as a nod to the $A() object covered in the Prototype.js set - the iterator function may actually make many of the functions out of prototype obsolete fairly quickly.)

Similarly, you can pass functions into iterators and retrieve objects out of them. For instance, consider the following generator, which will take a function, lower and upperbounds for that function and an increment, and will return coordinates back as results:

function $f(fn,a,b,dt){
    var t = a;
    while (t<=b){
         yield {x:t, y:fn(t)};
         t += dt;
         }
    } 
    
var sqr = function(x){
   return x*x;
   }
    
for (var p in $f(sqr,0,10,2)){
    print('(' + p.x + ',' + p.y + ')');
    }    
==> (0,0)
==> (2,4)
==> (4,16)
==> (6,36)
==> (8,64)    
==> (10,100)    
    

Here, the generator $f takes a function, lower and upper (inclusive) bounds and the delta to be added to the variable, then evaluates the function at each of these points, returning an anonymous object p{x,y} at each step in the iteration. This is useful of course for evaluating a function one point in the range at a time, but what if you wanted to take a more declarative approach, returning an array containing all of the points. As it turns out, the new Javacript functions can handle this particular use case as well, with a slightly different application of the for keyword:

function P(xp,yp){
     this.x = xp;
     this.y = yp;
     this.toString = function(){return 'p(' + this.x + ',' + this.y + ')';}
     }

function $f(fn,a,b,dt){
    var t = a;
    while (t<=b){
         yield new P(t,fn(t));
         t += dt;
         }
    } 

var sqr = function(x){
   return x*x;
   }
        
var points = [p for (p in $f(sqr,0,10,20))]
==> [p(0,0), p(2,4), p(4,16), p(6,36), p(8,64), p(10,100)]
    

(I’ve slightly redefined the $f function to express the result as a point p(x,y), just to showcase how.}

More formally, this would be expressed as:

arr = [expr(x) for (x in iteration(x))]    
    

This is a very rapid way of populating a complex array. and showcases an increasing preference within JavaScript to avoid informal indexed iterations in favor of higher order formal enumerations.

Iterating Outside the Box

Note that such generators are not necessarily panaceas - they do come with considerable overhead. A generator obviously must remain in memory during the scope of its operation, which means that it is in general far more efficient to define a generator at a global (or near-global) scope, though you should also only instantiate generators in more localized scopes. If you instantiate an iterator as part of a for(in) loop, that iterator will remain in scope only while the for(in) loop is in effect, and will be garbage collected thereafter, but if you work outside of the for(in) loop, the iterator (not just the generator) will remain in memory until you explicitly wipe it out. This is why, if you’re not using a for(in) loop, you should generally wrap any use of a generator in try/catch expression, create a named iterator, and set it to null when you’re done:

try {
     var f = $f(sqr, 0, 10, 2); 
     while ((pt = f.next()) != null) { 
            print(pt); 
            } 
      } 
catch (e) { 
      f = null; 
      }                       
            

You can also use another construct in the JavaScript 1.7 library, the let statement. Now, for those of us who remember the very earliest days of Visual Basic, let doesn’t necessarily bring big positive connotations with it, but at least within this implementation, let actually serves a real purpose. The let keyword lets you define a variable within its own scope, then lets you work within that scope. Once the operations are completed, the variable defined by let is automatically garbage collected, and if any other variables had that name, these are automatically restored to value. For instance:

var f = 5;    
let (f = $f(sqr,0,10,2)){
    while ((pt = f.next())!=null){
             print(pt);
             }
    }
==> (0,0)
==> (2,4)
==> (4,16)
==> (6,36)
==> (8,64)    
==> (10,100)        
print(f);
==> 5    
>

Generators and Iterators will likely take a while to percolate into general usage, but its not hard to see the potential that these particular operators open up for JavaScript. Beyond immediate enumeration, such iterators could be tied into data input and retrieval (the yield in this case retrieving the ongoing load of data from an external device), fractal generation (I won’t even begin to go into this one, save that I’m already thinking of a new variant of Conway’s Game of Life, and multiple other uses. Regardless, I suspect that this will get people thinking about the nature of programming (and quiet little JavaScript) in a whole new way.

---

Kurt Cagle is an author, software developer, and system architect. He lives in Victoria, BC with his wife and daughters, and should probably be doing something besides messing around with JavaScript features, but he will get to it eventually..