Meerkat: An Open Service API

Editor's note: Meerkat predated the popularity of syndication, feed services, and feed readers. Now that other groups are providing this service, we have removed Meerkat in favor of their better solutions. We maintain these articles for the sense of historical interest.

You want to know where the future is? No, not plastics. The future, my friends, is Web APIs.

I'm not talking here about Web applications (or "Weblications") themselves, but their public interfaces, those parts of themselves they choose to share.

An API (application program interface) is the public interface provided by an application for communication with external applications and programmers. This interface can be as simple as a set of supported parameters/arguments or as advanced as specific services (also known as methods or functions) that may be called remotely.

Let's bring this definition home a little -- those of you who get the drift can skip on down to Open Service. Think of yourself as an application -- some of you might find this harder than others :) Your inner workings are hidden from the outside world; everything that goes on in your brain is private, known only to you. You do, however, provide interfaces to the outside world: your five (or six) senses. So what, then, would be your API? Perhaps you're a short-order chef. The cafe's menu provides a list of the dishes you can prepare -- your available "functions" or "methods." This menu serves as your basic API. Diners (external applications, if you will) request that you run certain methods to produce their breakfast. In addition, they may request that you hold the cheese or scramble those eggs -- these parameters or arguments provide further detail to the methods called.

The APIs du jour are focused on the following three acronyms: XML, HTTP, and RPC. The exchanged data is marked up in XML ("eXtensible Markup Language"). The mechanism used to transfer this data over the Internet is HTTP ("HyperText Transfer Protocol") -- yes, the same protocol used to transfer Web documents from servers to your Web browser. RPCs ("Remote Procedure Calls") are full-blown methods or functions that may be remotely requested of an application.

XML-RPC is a class of RPC utilizing XML to communicate between servers and applications. A new XML-RPC protocol you might have heard mentioned quite a bit in the technology news recently is SOAP, a joint proposal by Developmentor, Microsoft, and Userland.

There's been considerable attention of late given to the wonders and power of Open Source. Not much is said, however, about the array of Open Services out there that are every bit as valuable as source code. Brian Behlendorf, President of the Apache Software Foundation and Founder and CTO of Collab.Net, a company that provides tools and services for Open Source development, writes: "I think it's fine to have a world where open and closed source software speak - by far the most important thing is that the protocols and APIs be stable and open and free from encumberance, that the software source code is open is secondary." (Editor's note: Collab.Net is affiliated with O'Reilly & Associates.) Open APIs are the lifeblood of the Web developer. Whether such interfaces are simply a side effect of the way in which the application is written or intentionally left open for public use, you'd be amazed at how they're used.

Open services are the building blocks behind home-brewed scripts, modules, and full-blown Weblications. Here are a few examples:

• DeepLeap is an online assistant which, among its many talents, provides a quick search through Google and local movie information via Yahoo! Movies.
• The Finance-QuoteHist Perl module fetches historical quotes from such sources as The Motley Fool and FinancialWeb.
• YahooChart grabs stock charts from Yahoo! Finance.
• The DMOZ Open Directory Project (ODP) is a comprehensive searchable Web directory staffed entirely by a distributed group of volunteers. DMOZ's uniqueness and power is that it's open on both ends, providing an Open Service and the ability to loop back and contribute to the very service you're using. Netscape, Google, and AltaVista are a few of the more the 100 sites using ODP data. In addition to the RDF (Resource Description Framework) dump of DMOZ's database available for download, interfaces such as the phpODP PHP module are making the process even easier.

"OK, I'm inspired! How do I open my Weblication up?" Glad you asked.

1. Create a straightforward interface to your service.
2. Document! This is the step most folks, for one reason or another, forget to do. Don't rely upon discovery; make it obvious.
3. Accept and incorporate constructive criticism. Your audience is there to help you -- listen to and learn from them.

Don't be embarassed if your code's a little messy -- nobody's going to see it. If you think your application isn't of any particular use, think again! If you found it necessary to create it, chances are somebody else will find it invaluable. No API is too simple or too small; even a door slightly ajar is better than one triple-bolted with a "Beware of Pointy-Toothed Rabbit!" sign emblazoned across it.

So here's where I take my own advice and open the doors on Meerkat: An Open Wire Service.

Meerkat is a Web-based syndicated content reader based on RSS ("Rich Site Summary"). RSS is a fantastic, simple-yet-powerful syndication system rapidly gaining momentum. For discussion of the ins and outs of RSS, as well as more information on Meerkat, I direct you to my article, Meerkat: An Open Wire Service and the Resources section at the end of this article.

RSS is an XML-based rendering of stories on your site you'd like to make available for syndication. These stories are composed of a title, a link (back to your site), and an optional description or blurb. I and others come along and grab these stories for incorporation into our own sites -- with links back to the full stories on yours.

Meerkat takes this a few steps further. Rather than simply incorporating a few headlines into the O'Reilly Network's web site,

• Meerkat's back-end acts as an aggregator for technology/computer/geek/science-related content.
• Meerkat's database provides said content to the O'Reilly Network and affiliates.
• The front-end affords visitors a spiffy RSS reader, organized along category/channel/chronological lines and sporting the power of regular expression searches.

It came to my attention one day that a popular news site was visiting Meerkat on a regular basis and scraping the HTML that Meerkat produced to grab the stories and incorporate them into their site. Needless to say, I was outraged! This poor programmer not only has to suffer through the tedium of writing a program to parse Meerkat's HTML, but must constantly follow changes I make between Meerkat releases with code updates. So I set to work making it easier for him/her and others like her/him, resulting in what I've affectionately termed Meerkat Flavours...

Meerkat flavours are predefined, yet customizable, templates defining how Meerkat displays RSS stories. Specify your flavour of choice on the URL-line using the _fl={flavour} argument. The current menu of flavours consists of:

• Meerkat
• Tofeerkat
• Minimal
• RSS 0.91
• RSS 1.0
• XML
• JavaScript Source
• PHP Serialized String
• The Meerkat Sherlock Plugin
• Notation3 (experimental)

Meerkat's default flavour (click for larger view).

Meerkat
URL: http://meerkat.oreillynet.com/
Stories: 50 per screen
Attributes: Action, Title, Link, Description, Source, Category, Date
Meerkat's default flavour, providing a comprehensive interface for your RSS-reading convenience. Note: no _fl argument is needed for the default flavour.

Tofeerkat flavour (click for larger view).

Tofeerkat
URL: http://meerkat.oreillynet.com/?_fl=tofeerkat
Stories: 50 per screen
Attributes: Action, Title, Link, Description, Source, Category, Date
For those who prefer something a little lighter, the Tofeerkat interface is a compact version of the default, optimized for those who take full advantage of the power of Meerkat's Profiles.

Minimal flavour (click for larger view).

Minimal
URL: http://meerkat.oreillynet.com/?_fl=minimal
Stories: 15
Attributes: Title, Link, Description?, Source?, Category?, Date?
Designed for the minimalist, Meerkat's Minimal flavour is ideal for Lynx browsers, handhelds using AvantGo or the like, and wireless micro-browsers.

RSS 0.91 flavour (click for larger view)

RSS 1.0
URL: http://meerkat.oreillynet.com/?_fl=rss10
Stories: 15
Default Attributes: Title, Link, Description?, Dublin Core Metadata?
Coming full-circle, returns 15 stories as RSS 1.0, ready for incorporation into your web site or insertion into a database. Similar to the RSS 0.91 flavour below, Meerkat's RSS 1.0 includes Dublin Core Metadata via the RSS 1.0 Dublin Core module and will continue to evolve, supporting any other standard modules as they become available.

XML flavour (click for larger view).

XML
URL: http://meerkat.oreillynet.com/?_fl=xml
Stories: 15
Attributes: Title, Link, Description?, Source?, Category?, Date?
Modeled loosely on RSS/0.91 format, the XML flavour gives you more of the information Meerkat collects for each story: source, category, and date. The rather simplistic DTD (Document Type Description), meerkat_xml_flavour.dtd is available for your perusal.

JavaScript flavour (click for larger view).

JavaScript Source (.js)
URL: http://meerkat.oreillynet.com/?_fl=js
Stories: 15
Attributes: Title, Link, Description?, Source?, Category?, Date?
Meerkat's JavaScript Source (.js) flavour is probably the most exciting to Web designers with little or no programming experience. If you know basic HTML, you can insert Meerkat stories right into your web site with a copy and a paste.

Simply insert the following line into your HTML document where you'd like the stories to appear:

<script 
language="JavaScript" 
src="http://meerkat.oreillynet.com/?_fl=js">
</script>

JavaScript-enabled Web browsers, upon encountering this tag, will grab the stories from Meerkat and insert them right into the page.

But wait, there's more! Meerkat's JavaScript Source flavour even has stylesheet support, so you can even customize the appearance of the various story elements using standard CSS (cascading style sheets). The following CSS classes correspond to the elements of each story:

Story Element	CSS Class
Title	meerkatTitle
Description/Blurb	meerkatDescription
Category	meerkatCategory
Channel	meerkatChannel
Date	meerkatDate

For example, the following sample HTML code snippet:

<html>
<head>

 <style>
  .meerkatTitle       { font-family: sans-serif; 
                        font-size: 11pt; 
                        font-weight: bold; 
                        color: black;            }
  .meerkatDescription { font-family: sans-serif; 
                        font-size: 10pt; 
                        color: black;            }
  .meerkatCategory    { font-family: sans-serif; 
                        font-size: 9pt; 
                        font-weight: bold; 
                        font-style: italic; 
                        color: brown;            }
  .meerkatChannel     { font-family: sans-serif; 
                        font-size: 9pt; 
                        font-style: italic; 
                        color: brown;            }
  .meerkatDate        { font-family: sans-serif; 
                        font-size: 9pt; 
                        color: green;            }
 </style>
</head>

<body>
 <table border='0' bgcolor='#eeeeee' cellpadding='10'>
  <tr>
   <td>
    <p><center>
    <u><b>MEERKAT TODAY</b></u>
    </center></p>
    <script language="JavaScript" 
      src="http://meerkat.oreillynet.com/?_fl=js">
    </script>
  </td>
 </tr>
</table>
</body>

</html>

...produces a page similar to this:

JavaScript flavour using CSS (click for larger view).

PHP Serialized String
URL: http://meerkat.oreillynet.com/?_fl=php
Stories: 15
Attributes: Title, Link, Description?, Source?, Category?, Date?
PHP programmer? Not much in the way of XML/RSS experience? Never fear -- you're only a few lines of code away from a serialized string chock full of Meerkat stories.

<?php

  $url = "http://meerkat.oreillynet.com/?_fl=php";
  if ( $cereal = @implode( "", file($url) ) )
   $stories = unserialize( $cereal );
  else
   print "Couldn't make cereal!\n";

?>

This simple block of code yields an array (actually an array of arrays of arrays ;-) as if constructed thusly:

$stories = array(
  0 => array(
    'title' => "Meerkats not Mere Cats!",
    'link' => "http://meerkat.oreillynet.com/not.html",
    'description' => "Scientists prove ...",
    'category' => "Biology: Mammals",
    'channel' => "Meerkat Today!",
    'date' => "2000-05-01 20:00:17"
  ),
  1 => 
    ...
  ),
  ...
);

Printing the second story's title is as simple as: echo $stories[1]['title'];

Sherlock Plugin
URL: n/a Stories: 100 Maximum
Constraints: 7 Days Worth of Stories, Macintosh Only
Attributes: Title, Link, Description, Source, Category, Date
The Meerkat Sherlock Plugin affords Macintosh users the ability to search the stories picked up by Meerkat from the comfort of Macintosh's Sherlock interface. For more information and to download the plugin, click here.

Here are a few screenshots to whet your appetite (or maybe make you supremely jealous if you're not a Mac user ; ).

Screen shots of the Sherlock Plugin (click for larger view).

And as if flavours weren't enough, you can also pass Meerkat various parameters or arguments to hone in on just the stories you're interested in.

Parameter	Description
Query Properties
s	Search For Instructs Meerkat to search for something in the story title or description. The same effect as entering a search query into the search box in Meerkat's standard control panel. Settings: Either a + delimited list of keywords or a a // enclosed regular expression. Example: http://meerkat.oreillynet.com/?s=perl+apache "stories whose title or description contains either perl or apache"
sw	Search What By default, Meerkat's searches meander through story titles and descriptions. This option instructs Meerkat instead to search another field in particular. Currently supported are the Dublin Core Metadata elements: The same effect as choosing a field to search from Meerkat's standard control panel. Settings: {blank = title, description}, dc_title, dc_creator, dc_subject, dc_description, dc_publisher, dc_contributor, dc_date, dc_type, dc_format, dc_identifier, dc_source, dc_language, dc_relation, dc_coverage, dc_rights Example: http://meerkat.oreillynet.com/?s=perl+apache "stories whose title or description contains either perl or apache" Example: http://meerkat.oreillynet.com/?s=tim%20o'reilly&sw=dc_creator "stories whose written by tim o'reilly" (spaces in URLs are replaced by %20)
c	Channel Instructs Meerkat to display only a particular channel. The same effect as selecting a channel from the Categories/Channels menu in Meerkat's standard control panel. Settings: A numerical channel ID. Example: http://meerkat.oreillynet.com/?c=1243 "stories from the 'oreillynet.python newsgroup".
t	Time Period How far back Meerkat should look for stories. The same effect as choosing a time period from Meerkat's standard control panel. Settings: A number followed by one of: MINUTE, HOUR, DAY, ALL. The number is optional when choosing ALL. Example: http://meerkat.oreillynet.com/?t=7DAY "the past 7 days' worth of stories"
p	Profile Restores the settings from a particular Meerkat profile, whether global or yours personally. The same effect as choosing a profile to restore from Meerkat's standard control panel. Settings: The numerical ID of an existing profile. Example: http://meerkat.oreillynet.com/?p=563 "stories caught by profile number 563 (The O'Reilly Network)"
m	Mob Displays the stories associated with a particular Mob. Settings: The numerical ID of an existing Mob (see "Manage my Mobs..." under the Profiles/Mobs menu for your Mob's ID. Example: http://meerkat.oreillynet.com/?m=123 "stories grouped under Mob number 123"
i	Id Displays a particular story. Settings: The numerical ID of an existing story. Hold your mouse over the Mob (ring of dots) icon to see the story's id. Example: http://meerkat.oreillynet.com/?i=456 "story number 456"
Display Properties
_fl	Flavour Allows you to choose your preferred Meerkat Flavour or output format such as 'meerkat' (the standard interface), 'xml', and 'php,' to name but a few. Settings: A valid Meerkat flavour; see Meerkat Flavours for a comprehensive listing. Example: http://meerkat.oreillynet.com/?_fl=rss "choose RSS flavouring; display the output as an RSS channel"
_de	Descriptions Turn on or off story descriptions or blurbs. You lose some of the story detail, but gain a compact display for easy scanning. Settings: 0 = Off, 1 = On (default) Example: http://meerkat.oreillynet.com/?_de=0 "hold the descriptions"
_ca	Categories Meerkat's channels are cataloged into a category hierarchy; if these categories aren't useful to you, feel free to turn 'em off. Settings: 0 = Off, 1 = On (default) Note: Some Meerkat flavours (e.g. RSS) may exclude the display of categories, ignoring this setting. Example: http://meerkat.oreillynet.com/?_ca=0 "no categorization necessary, thank you"
_ch	Channels Meerkat's stories are picked up from hundreds of channels. Don't really care from which channel a particular story comes? Turn the channel display off. Settings: 0 = Off, 1 = On (default) Note: Some Meerkat flavours (e.g. RSS) may exclude the display of categories, ignoring this setting. Example: http://meerkat.oreillynet.com/?_ch=0 "turn off the display of channels"
_da	Dates When Meerkat first noticed and picked up a story. Settings: 0 = Off, 1 = On (default) Note: Some Meerkat flavours (e.g. RSS) may exclude the display of categories, ignoring this setting. Example: http://meerkat.oreillynet.com/?_da=0 "no thanks, I already have a date"
_dc	Dublin Core Metadata Meerkat supports the RSS 1.0 RSS 1.0 Dublin Core module, augmenting RSS's standard title, link, and description with such attributes as creator (author), subject, rights, language, publisher, format, and so on. Settings: 0 = Off, 1 = On (default) Note: Some Meerkat flavours (e.g. Minimal) may exclude the display of Dublin Core Metadata, ignoring this setting. Example: http://meerkat.oreillynet.com/?_dc=0 "no thanks, title and description are enough for me"

You may have noticed ? marks next to some of the attributes in the various Meerkat Flavours descriptions on page 3. These indicate that display settings affect whether or not these attributes are displayed. Unless specifically turned off, these attributes will be displayed by default. Those without ? (Title and Link) cannot be turned off.

How about a few more real-world examples to get you started?

Example 1: "Show me today's Macintosh-related stories as RSS 1.0 and go ahead and include the Dublin Core metadata."
	http://meerkat.oreillynet.com/?p=1065&_fl=rss10
	p=1065		This is a simple one since a Meerkat global profile (#1065) exists for Macintosh stories.
	_fl=xml		Use Meerkat's XML flavour.
Example 2: "I'd like to, as simply as possible, insert the latest Wireless stories into my home page."
	<script language="JavaScript" src="http://meerkat.oreillynet.com/?p=9&_fl=js"> </script>
	p=9		Let's make use of another of Meerkat's predefined profiles (#9).
	_fl=js		Return the stories as a .js file.
Example 3: "Let's build an RSS channel devoted to Apache modules."
	http://meerkat.oreillynet.com/?t=ALL&s=mod_&_fl=rss
	t=ALL		Use an unrestricted time window.
	s=mod_		Search for any occurrence of the substring "mod_" in the story title or description (e.g. "mod_perl").
	_fl=rss		Output as an RSS channel, please.
Example 4: "I want to use my Palm Pilot and AvantGo to grab just the latest stories and their descriptions."
	http://meerkat.oreillynet.com/? _ca=0&_ch=0&_da=0&_fl=minimal
	_ca=0		Turn off category display.
	_ch=0		Turn off channel display.
	_da=0		Turn off date display.
	_fl=minimal		Invoke the "minimal" Meerkat flavour.

A few notes on constructing URLs using the various flavours and other customization parameters. Be sure to prepend the list of parameters with a ? mark; this should come right after the basic Meerkat URL like so: http://meerkat.oreillynet.com/?. Also, you must separate each parameter with an &, as in s=linux&_ca=0&_fl=xml.

Build something wonderful ... and remember to keep it open and let people know about it! And, if you're so inclined, spread the Meerkat word with a nifty "Meerkat Powered!" button. To do so, simply copy-and-paste the following HTML into your document:

<a href="http://meerkat.oreillynet.com">
<img 
src='http://meerkat.oreillynet.com/icons/meerkat-powered.jpg'>
</a>

Hopefully I've succeded in providing a useful and well-documented Open Service and API. As usual, I welcome any constructive criticism you might offer. Please post your suggestions, bug reports, and other feedback (maybe praise?) to the O'Reilly Network RSS Forum.

The following is a list of starting points from which to explore further some of the topics covered (or not) in this article.

• Meerkat
  • Meerkat
  • "Meerkat: An Open Wire Service" by Rael Dornfest, The O'Reilly Network
  • "Meerkat Help" by Rael Dornfest, The O'Reilly Network
• Stylesheets (CSS)
  • WebReview: Style Sheets
• RSS
  • The O'Reilly Network RSS DevCenter
  • WebReference: Web/Authoring/Languages/XML/RSS
• XML
  • XML.com
• JavaScript
  • "Specifying a File of JavaScript Code", Netscape DevEdge
  • Client-Side JavaScript Guide, Netscape DevEdge
• PHP
  • PHP.net
  • Zend
  • PHP's implode() function
  • PHP's unserialize() function
• Sherlock
  • The O'Reilly Network RSS DevCenter
  • Apple: Mac OS: Sherlock
  • Apple-Donuts: Sherlock Internet Search Archives
  • Apple Developer Connection: Technical: Mac OS: Sherlock

Rael Dornfest is Founder and CEO of Portland, Oregon-based Values of n. Rael leads the Values of n charge with passion, unearthly creativity, and a repertoire of puns and jokes — some of which are actually good. Prior to founding Values of n, he was O'Reilly's Chief Technical Officer, program chair for the O'Reilly Emerging Technology Conference (which he continues to chair), series editor of the bestselling Hacks book series, and instigator of O'Reilly's Rough Cuts early access program. He built Meerkat, the first web-based feed aggregator, was champion and co-author of the RSS 1.0 specification, and has written and contributed to six O'Reilly books. Rael's programmatic pride and joy is the nimble, open source blogging application Blosxom, the principles of which you'll find in the Values of n philosophy and embodied in Stikkit: Little yellow notes that think.

Discuss this article in the O'Reilly Network RSS Forum.

Return to the RSS DevCenter.

Copyright © 2009 O'Reilly Media, Inc.