Meerkat: The XML-RPC Interface

Editor's note: Meerkat predated the popularity of syndication, feed services, and feed readers. Now that other groups are providing this service, we have removed Meerkat in favor of their better solutions. We maintain these articles for the sense of historical interest.

Meerkat, O'Reilly Network's Open Wire Service, extends its open API with XML-RPC, affording a more standardized XML-based interface to its aggregated RSS database.

Meerkat is an open service. Its API (Application Programming Interface) is open and well-documented. For HTML jockeys, adding syndicated stories to a web site is only one line of cut-and-paste JavaScript away. Application developers with a little know-how and a sense of adventure may use Meerkat's rich XML feeds and powerful "recipes" (read: queries) as a framework upon which to build their own applications.

Meerkat's API is "URL-line"; similar in concept to the Unix command-line, Meerkat's engine may be queried over the Web using combinations of various arguments.

The URL-line interface is rather common in the web-scripting world. A script's URL is postfixed with "?" and one or more "&"-separated "key=value" pairs representing variables and their associated values. With the emergence of CGI scripting, the URL-line became the Web's command-line interface. The URL-line is lightweight, bookmarkable, simple (hopefully ;-), and builds on the idea of a URL being a unique identifier for content on the Web, whether static or dynamically generated. But how does it fare in more complex applications with richer data requirements? And is there any consistency across URL-line APIs?

The RPC in XML-RPC stands for "Remote Procedure Call," a fancy way of saying that you're calling procedures (functions, subroutines, or methods) on another, remote, server. Just like a URL-line CGI script, arguments are passed and results returned; the difference is only in the serialization (the way it's written), which here, as the name suggests, is XML. XML allows for richer message-passing between client and server. For more information on XML-RPC, pointers to language implementations, and a quick read of the specification, visit XML-RPC.com.

Meerkat's XML-RPC API is a direct port of its straightforward URL-line recipes to an RPC environment. Written (as with the rest of Meerkat) in PHP, it uses Edd Dumbill's PHP XML-RPC modules. These provide simple, clean implementations of both client and server -- I highly recommend them!

Meerkat's XML-RPC server is at the URL: http://www.oreillynet.com/meerkat/xml-rpc/server.php

The server supports five methods: introspection, getCategories, getChannels, getChannelsByCategory, getItems. We'll go through each of these in detail. For demonstration and edification purposes, I've provided a link to Meerkat's live test suite for each of the methods.

Introspection

Signature: array system.listMethods()
This method lists all the methods (including itself) that Meerkat's XML-RPC server knows how to dispatch. It returns an array of structs of method names, their signatures, and documentation.
[Test system.listMethods]

getCategories

Signature: array meerkat.getCategories()
Returns an array of structs of available Meerkat categories, each with its associated category ID.
[Test system.getCategories]

getChannels

Signature: array meerkat.getChannels()
Returns an array of structs of available RSS channels, each with its associated channel ID.
[Test meerkat.getChannels]

getChannelsByCategory

Signature: array meerkat.getChannelsByCategory(int)
Returns an array of structs of RSS channels in a particular category (specified by integer category ID), each with its associated channel ID.
[Test meerkat.getChannelsByCategory]

getItems

Signature: array meerkat.getItems(struct)
Returns an array of structs of RSS items given a recipe struct. The getItems method makes full use of Meerkat's powerful recipes (read: query parameters). The following is a complete list of parameters from which to build a recipe:

• Search Criteria
  • channel - (int) a channel's numeric ID
  • category - (int) a category's numeric ID
  • item - (int) a particular item's numeric ID (retrieve a specific item)
  • search - (string) /MySQL regular expression/
  • search_what - (string) By default, Meerkat's searches meander through story titles and descriptions. This option instructs Meerkat instead to search another field in particular. Currently supported are the Dublin Core Metadata elements: {blank = title, description}, dc:title, dc:creator, dc:subject, dc:description, dc:publisher, dc:contributor, dc:date, dc:type, dc:format, dc:identifier, dc:source, dc:language, dc:relation, dc:coverage, dc:rights
  • time_period - (string) travel back in time; format is #(MINUTE|HOUR|DAY)
  • profile - (int) a particular profile's numeric ID
  • mob - (int) a particular mob's numeric ID
  • url - (int) a particular url (Yes, find a story by its url!)
• Display Recipes
  • ids - (0=off, 1=on) return story IDs
  • descriptions - (0=off, 1=full, >1=length) return item descriptions (0=off, 1=full, >1=length)
  • categories - (0=off, 1=on) return categories with which items are associated
  • channels - (0=off, 1=on) return channels in which items appeared
  • dates - (0=off, 1=on) return dates items were picked up by Meerkat
  • dc - (0=off, 1=on) return Dublin Core elements for items
  • num_items - (0 < n <= 50) number of items to be returned

Here's a simple PHP-based example, searching three days' worth of the xmlhack channel for any mention of XML or Java and returning IDs, descriptions, categories, channels, and dates. Only the latest five items are requested.

$f = new xmlrpcmsg("meerkat.getItems", 
  array(
    new xmlrpcval(
      array(
        "channel" => new xmlrpcval(724, "int"), 
        "search" => new xmlrpcval("/XML|[Jj]ava/", "string"),
        "time_period" => new xmlrpcval("3DAY", "string"),
        "ids" => new xmlrpcval(0, "int"),
        "descriptions" => new xmlrpcval(200, "int"),
        "categories" => new xmlrpcval(0, "int"),
        "channels" => new xmlrpcval(0, "int"),
        "dates" => new xmlrpcval(0, "int"),
        "num_items" => new xmlrpcval(5, "int"),
      ), 
      "struct"
    )
  )
);

[Test meerkat.getItems]

Here's a little sample code to get you on your way. There are implementations for various programming languages available at XML-RPC.com. I've chosen PHP, Perl, and Python for my examples.

Sample 1: PHP
A PHP code snippet for retrieving and printing all of Meerkat's categories.

<?php

  # Include the PHP XML-RPC library
  include("xmlrpc.inc");

  # The server (script) URL
  $server_url = '/meerkat/xml-rpc/server.php';

  # Create a new XML-RPC message instance, calling the getCategories method
  $msg = new xmlrpcmsg('meerkat.getCategories', array());

  # Create a new XML-RPC client instance
  $client = new xmlrpc_client($server_url, "www.oreillynet.com", 80);

  # Debugging is off by default; uncomment this line for all the nitty gritty
  # $client->setDebug(1);

  # Send our XML-RPC message to the server and receive a response in return
  $response = $client->send($msg);

  # Did we receive a response?
  if (!$response) { die("send failed"); }

  # Retrieve the response's value
  $value = $response->value();

  # And convert it to a PHP data structure
  $categories = xmlrpc_decode($value);

  # Iterate over the results, printing each category's title
  while( list($k, $v) = each( $categories ) ) {
    print $v['title'] . "<br />\n";
  }

?>

Sample 2: Perl
A Perl script using Ken MacLeod's Frontier::Client module, part of his Frontier-RPC XML-RPC toolbox. This script retrieves and prints the component channels of Meerkat's Perl category.

#!/usr/bin/perl

# Use the Frontier::Client Perl module
use Frontier::Client;

# The XML-RPC server (script) URL
my $server_url ='http://www.oreillynet.com/meerkat/xml-rpc/server.php';

# Create a new Frontier XML-RPC client instance
my $client = Frontier::Client->new( 
  'url' => $server_url,
  'debug' => 1, 
  'encoding' => 'iso-8859-1'
);

# Call the meerkat.getChannelsByCategory() method, passing the 
# Perl category's numeric id.
my $response = $client->call("meerkat.getChannelsByCategory", 2);

# Iterate over the results, printing each Perl channel's title
print map { $_->{'title'} . "<br />\n" } @{$response};

Sample 3: Python
An interactive Python session using PythonWare's xmlrpclib to search Meerkat for the latest five stories on Python. Descriptions, turned on by default, have been disabled for brevity's sake in this example.

Python 1.5.2 (#1, Feb  1 2000, 16:32:16)  
[GCC egcs-2.91.66 19990314/Linux (egcs- on linux-i386
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam

... # Import the xmlrpclib module
>>> from xmlrpclib import *
... # Instantiate a new XML-RPC Server object
>>> server = Server("http://www.oreillynet.com/meerkat/xml-rpc/server.php")
... # Simply dump the results of our query
>>> print server.meerkat.getItems(
    {'search':'/[Pp]ython/','num_items':5,'descriptions':0})

[{'title': 'Vote on the new name for Python Methods', 'description': '',
'link': 'http://www.zope.org/Members/hathawsh/MethodNamePoll'}, {'title':
'Vote on the new name for Python Methods!', 'description': '', 'link':
'http://www.zope.org/Members/hathawsh/MethodNamePoll'}, {'title': '(The
Eff-bot Guide To) The Standard Python Library', 'description': '', 'link':
'http://shop.barnesandnoble.com/booksearch/isbnInquiry.asp?isbn=1588955710'},
{'title': 'Python-URL!', 'description': '', 'link':
'http://linuxtoday.com/news_story.php3?ltsn=2000-11-06-007-04-OS-SW'},
{'title': 'Python MP3 Manager 0.9.1', 'description': '', 'link':
'http://freshmeat.net/projects/pymm/'}]
>>>

Hopefully I've succeeded in providing an API that will be useful to you in building your application -- or just to fiddle with or learn from. As usual, I welcome any constructive criticism you might offer. Please post your suggestions, bug reports, and other feedback (maybe praise?) to the O'Reilly Network RSS Forum.

And, if you're so inclined, spread the Meerkat word with a nifty "Meerkat Powered!" button. To do so, simply copy-and-paste the following HTML into your document:

<a href="http://meerkat.oreillynet.com"><img src='http://meerkat.oreillynet.com/icons/meerkat-powered.jpg'></a>

Rael Dornfest is Founder and CEO of Portland, Oregon-based Values of n. Rael leads the Values of n charge with passion, unearthly creativity, and a repertoire of puns and jokes — some of which are actually good. Prior to founding Values of n, he was O'Reilly's Chief Technical Officer, program chair for the O'Reilly Emerging Technology Conference (which he continues to chair), series editor of the bestselling Hacks book series, and instigator of O'Reilly's Rough Cuts early access program. He built Meerkat, the first web-based feed aggregator, was champion and co-author of the RSS 1.0 specification, and has written and contributed to six O'Reilly books. Rael's programmatic pride and joy is the nimble, open source blogging application Blosxom, the principles of which you'll find in the Values of n philosophy and embodied in Stikkit: Little yellow notes that think.

Discuss this article in the O'Reilly Network RSS Forum.

Return to the O'Reilly RSS DevCenter.

Copyright © 2009 O'Reilly Media, Inc.