This column provides a lightweight demo of Dojo’s increasingly popular data grid and demonstrates it serving up a million records. The intent of this little demo is to demonstrate the basic pattern for putting the grid to work and save you from spending so much time trying to grok the source code.

It’s a given that you can use the grid to display relatively small data sets effectively in the browser and get the niceties of sorting, column resizing, etc. that come along with it. That’s cool and all, but there’s a practical limit to the number of records you can deal with at any given time, which eventually leads to the concept of paginating results.

Well, go ahead and forget about pagination; those days are finally over. Dojo’s grid works by lazy loading data as the grid scrolls. For a relatively small data set consisting of hundreds of records, lazy loading amounts to building out the DOM for a pre-loaded data set when you scroll. For example, if you had 100 records but could only view 20 of them at a time, you wouldn’t want to build the nodes for any of the records in 21 through 100 until you scrolled to that particular section. Sorting by a column and related tasks work in memory as expected for small data sets since you can effectively get things done in JavaScript.

For very large data sets, however, it quickly becomes impractical or even impossible to use JavaScript for tasks such as sorting by a column; if the data set is gargantuan, it’s not even practical to try and maintain it in the browser through something like an ItemFileReadStore. Dojo’s approach to the problem of dealing with large data sets is simple, elegant and boils down to two things: a dojo.data implementation (we’ll use QueryReadStore) that can request data from the server in arbitrary page sizes, and a grid that is capable of scrolling such that it requests and loads a particular page on demand. Now, let’s see it in action.

To get started, save the following page as grid.html and spend a few minutes skimming it. (You may find Part 1 and Part 2 of a primer Bryan Forbes wrote on the grid to be useful context.) The basic pattern we use to put the grid to work is connecting it to a model intermediates between it and a dojo.data store; the model we use, a QueryReadStore, happens to be one that lends itself to fetching pages of data from the server. The basic idea is that the model consumes these data pages from the store and the view updates to reflect the model.


<html>
    <head>
        <title>Dojo Goodness, Part 6 (A Million Records in the Grid)</title>

        <link rel="stylesheet" href="http://o.aolcdn.com/dojo/1.1/dojox/grid/_grid/Grid.css">
        <link rel="stylesheet" href="http://o.aolcdn.com/dojo/1.1/dojox/grid/_grid/tundraGrid.css">
        <link rel="stylesheet" href="http://o.aolcdn.com/dojo/1.1/dijit/themes/tundra/tundra.css">
        <link rel="stylesheet" href="http://o.aolcdn.com/dojo/1.1/dojo/resources/dojo.css">

        <script
            type="text/javascript"
            src="http://o.aolcdn.com/dojo/1.1/dojo/dojo.xd.js"
            djConfig="parseOnLoad:true"
        ></script>

        <script type="text/javascript">
            dojo.require("dojox.data.QueryReadStore");
            dojo.require("dojox.grid.Grid");
            dojo.require("dojo.parser");

            // some functions to format grid cell data
            computeTotal = function(rowIndex) {
                row = model.getRow(rowIndex);
                return row ? "$"+row.cost*row.quantity : "";
            }
            formatCost = function(rowIndex) {
                row = model.getRow(rowIndex);
                return row ? "$"+row.cost : "";
            }

            // a simple grid layout that consists of 5 columns
            layout = [
                {cells:[[
                    {name:'ID',field:'id',width:'5'},
                    {name:'Item',field:'item',width:'auto'},
                    {name:'Quantity',field:'quantity', width:5},
                    {name:'Cost',field:'cost', width: 5, get: formatCost},
                    {name:'Total',field:'total', width: 5, get : computeTotal}
                ]]}
            ];

        </script>
    <head>
    <body class="tundra">

        <!-- The dojo.data API implementation that talks to the server -->
        <div dojoType="dojox.data.QueryReadStore" jsId="store", url="/data"></div>

        <!-- The intermediary between the dojo.data API and the grid -->
        <div dojoType="dojox.grid.data.DojoData" jsId="model" rowsPerPage="20" store="store">  

        <!-- The grid, which relies on its DojoData abstraction for info -->
        <div style="height:300px; width:400px;" dojoType="dojox.grid.Grid" model="model" structure="layout" delayScroll="true"></div>

    </body>
</html>

With the client-side paradigm in place, all we need now is a web server that can pass back whatever data the store requests. Fortunately, we just happen to have one on hand. If you save the following simple web server as server.py, it should run by simply running python server.py in a terminal. As a suggestion, you might use EasyInstall to get CherryPy and the demjson module that are used. The web server itself does little more than serve up a static HTML page and provide a URL that returns a data set that is customizable through the query string.


"""
An ultra-simple web server that provides slices of a very large (mock) data
source for a dojox.grid.Grid client that uses a dojox.data.QueryReadStore
to page the data on demand
"""

import cherrypy #do an "easy_install cherrypy" to get it
from cherrypy.lib.static import serve_file

import demjson #do an "easy_install demjson" to get it
import os
from random import randint #for building up mock data

json = demjson.JSON(compactly=False)
jsonify = json.encode

NUM_ITEMS = 1000000

class Content:
    def __init__(self):
        """
        maybe you would call out to a db with some sql to get some data
        based on the query string that comes into /data. for now, we'll
        build up some static data to use.
        """

        self.items = []

        possible_item_names = ["Foo", "Bar", "Baz", "Bop"]
        id=0
        for i in xrange(NUM_ITEMS):
            self.items.append({
                    'id' : id,
                    'item' : possible_item_names[randint(0,3)],
                    'quantity' : randint(0,10),
                    'cost' : randint(0,100)
            })
            id +=1

        #keep track of sort order b/c sorting is expensive...
        self.current_sort_order = ""

    @cherrypy.expose
    def data(self, **kw):
        """
        serve up the data via http://localhost:8000/data

        kw will contain whatever is in your store's query. by default
        the query string will come across as something like:
        ?name=*&start=0&count=20 to populate the table 

        note: you may get into trouble if you have multiple users
        trying to access this url and changing the sort order of items
        all at the same time (but relax, this is just a little demo.)
        """

        #sorting the items by values for a given dictionary key...
        if kw.get('sort') and self.current_sort_order != kw.get('sort'):
            if kw['sort'][0] == '-': #descending order, slice off the -
                self.items.sort(lambda m,n:cmp(m.get(kw['sort'][1:]), n.get(kw['sort'][1:])),reverse=True)
            else: #ascending order
                self.items.sort(lambda m,n:cmp(m.get(kw['sort']), n.get(kw['sort'])))
            self.current_sort_order = kw['sort']

        #slicing the data...
        start = int(kw['start'])
        end = start + int(kw['count'])

        #serving up the slice of interest as well as the total size
        return jsonify({'numRows':NUM_ITEMS, 'items':self.items[start:end]})

    @cherrypy.expose
    def grid(self, **kw):
        """
        Serve up the web page through http://localhost:8000/grid
        """
        return serve_file(os.path.join(os.getcwd(), 'grid.html'))

cherrypy.server.socket_port = 8000
cherrypy.quickstart(Content(),'/')

Assuming you fire up the web server and have the grid.html page stored alongside of it, you should be able to navigate to http://localhost:8000/grid and take it for a spin. It’s particularly interesting to inspect the individual page requests that the QueryReadStore fires off through the Firebug console. You’ll see that a typical request is something like http://localhost:8000/data?name=*&start=0&count=20, which passes in the appropriate offset so that the server returns the page in question. (Note that it may take a moment for the server to sort a million records, so be patient when you start sorting the columns.)

As always, feel free to leave comments below, and be sure to check out my upcoming book, Dojo: The Definitive Guide if you find yourself in need of a reference.