You’ve probably heard about S3. It’s Amazon’s new storage service that debuted just a couple of days ago. Developers can store data of any kind, up to 5GB per chunk, and retrieve it at will. (The 5GB limit is just for a single file. You can store as much data as you want.) The idea is to provide flexible and reliable data storage using the same scalable model already seen with other Amazon Web Services (AWS), such as the Simple Queue Service, Mechanical Turk and Alexa Web Search. Amazon, in short, is building the backbone for a new kind of Web.
Like the other AWS services, it’s pay as you go. Fifteen cents per gigabyte per month storage, twenty cents per gigabyte transferred. It’s not a super bargain, but it’s not exorbitant and you only pay for what you use. It’s pretty easy to use at that. And of course, you get to use funky amazon URLs like s3.amazonaws.com/erica/elvio, which by themselves are, you know, cool.
Each developer can create buckets to store data. You add stuff to the buckets in key/value pairs. For example, you might create a “New Years 06 Pictures” bucket. The you might add an image called “Champagne Toast” and another called “Counting Down”. These keys allow you to associate a name and a meaning with the data as well as to retrieve the data by name.
Of course, you’re not limited to pictures. You can store text, sounds, video, whatever. If you have a mime-type, you’ve got a place to store it.
Even better, you can defray some of the transfer costs by posting your data using the BitTorrent protocol. Using BitTorrent allows your audience to help shoulder the data transfer bandwidth so you’re not just paying through the nose to Amazon when you’re hosting a popular data item.
Although sufficient to the job, the API is pretty limited. You can create and delete buckets, and add and remove data from them. Which is, in the end, exactly what you need an online storage system to do. (Amazon also allows you to annotate your data with custom tags, but that’s mainly for your own bookkeeping.
From a Mac point of view, none of the original code samples I tried seemed to want to work. For example, my perl libraries and packages didn’t have all the right components and I couldn’t find the right ones over at cpan. So in the end, I wrote everything from scratch which took only a few hours because the Mac rocks for simple Web/XML development. I built a baby sockets package and a HMAC encryption utility in C and then wrote bunches of useful little scripts (all alike) in perl. I’ll probably shove these behind a AppleScript Studio user interface if I get the time
You do have to be very, very precise when working with the APIs. For example, I wanted to make the photo (linked above) publicly readable. The part of the PUT request for this is X-Amz-acl: public-read. However, the part that you have to add to the authentication signature is x-amz-acl:public-read. Notice the difference in capitalization and spacing? Argh. Of course, Amazon did document this idiot design decision, but it was buried deep, deep, deep in the documentation.
Do you have an interest in developing for S3 from the Mac? Or would you like to see me throw my scripts into a GUI? Let me know in the comments.