Here’s a potential media application. Can Flash or one of the Java media libraries handle it?

One of the weaknesses — and I’m surprised by the lack of critical talkbacks, c’mon people, this is a blog, where’s the hate? — of my little Java Media trilogy (viz., acts I, II, and III), is that after a point, I expect the reader to take on faith my claim that Web 2.0’s read/write nature will eventually express itself in media, and that we’ll need media frameworks that offer more than just playback. So, I’d like to propose an application and go about seeing how feasible it is.

The key to my current line of thinking is that I want to get in the mindset of the application developer who knows she’s doing a media application, but doesn’t yet know which languages, libraries, or platforms to choose. This is different than the usual mindset of “we’re all Java programmers, what do we want in a media library?” I think it’s important to understand what advantages, if any, a Java approach offers.

Starter deck

The idea for this application comes from the fact that I’m interested in learning more about trading card games, specifically the Fullmetal Alchemist TCG. It looks interesting, and the designers seem to really get the show.

Thing is, these games have a real network effect to them — popular games have more players to enjoy the game with, more places to buy cards, etc. If you like something nichey, say, the Battlestar Galactica game or the City of Heroes game (or FMA, for that matter), instead of popular war-horses like Yu-Gi-Oh or Magic: The Gathering, then you’re going to be hard up to find opponents.

Obvious solution: go online. The internet is all about niches. And this can work, but then again, why bother being a card game if it’s gonna be an online computer game? At that rate, you might as well play an online RPG.

But somehow, I got to thinking… I guess it was the idea of Delicious Monster turning my iSight into a bar-code reader.

oj-media-app-monster.png

So, imagine if every card manufactured had a universally unique ID number, represented as a bar code on the card itself. This would allow an online version of the game to be played with your physical deck, simply by scanning your cards with your webcam as they’re played. Something like this (but not as cheesily fake):

oj-media-app-card-scan.png

Since the online version would work with physical cards, it would encourage the continued sale of physical decks, cards, and accessories, and help titles achieve a network effect by connecting players to one another. You could probably sell some Google ads on the site too, just to make it even more appealing to management. Maybe we’ll throw in optional voice and video chat so the players can socialize.

There are lots of implementation details that can be discussed (how to prevent counterfeiting, whether tracking players by their cards is incompatible with real-world trading, etc.), but that’s not my point. My point, firstly, is to get this up as prior art just in case this Method of Associating Collectible Gaming Cards with Online Records by Means of a Video Capture Device isn’t patented yet. (^o^)

But my main point is this: if you are Mr. Trading Card Company, what technology do you use to implement this?

Discard Pile

My guess is that most developers would start by looking at Flash, because they’re probably already using it for their website (such as, say, the online tutorial for FMA). Flash has the advantage of being the only web-centric Win/Lin/Mac solution that supports video capture. Here’s a short step-by-step on setting up a basic camera preview as part of a Flash presentation.

The problem with the Flash approach is that there doesn’t seem to be a straightforward way to get at the pixels, which you would need in order to run an algorithm to find and interpret the barcode (I’m taking it on faith that such an algorithm can be developed from available resources on bar-codes, or licensed from a subject-matter expert). Neither the Camera nor the Video ActionScript objects seem to have calls to get the pixels from the camera.

This might be possible with Java Media Framework. I don’t remember all the details of JMF anymore, but I think you would need to construct a plug-in like a Codec (I think?) that would get an opportunity to have its process() method called with captured frames from the camera. It’s in this call that you’d have access to the camera’s pixels, and be able to run your barcode-finding logic. Thing is, with JMF, while your code might compile, nothing’s really promised at runtime: you might or might not be able to get a Processor to use with the discovered capture device. And the list of supported capture devices does not inspire confidence, considering that Windows compatibility is listed entirely in terms of deprecated version of the OS (98, NT, etc.). Plus, the JMF solution only works on Windows — if you’re going to write a one-platform application, you might just skip the webapp (and Java) and write a native desktop application.

QuickTime for Java would definitely work, something I can say with confidence after working extensively with its capture API in the last year (I wrote a minimal-hassle, compress-as-you-go, actually-has-a-preview-window video capture app for Veldhiezen Group, who are really into QuickTime and were great to work with). For the card game app’s scanner, you’d create an SGDataProc object that would get called back every time the “Sequence Grabber” has new data. The callback includes a pointer to a possibly-encoded block of memory, and an ImageDescription describing the format of that data. This is a really nice design, because it speaks to the likelihood that the data coming off the camera may already be compressed (always the case with DV, and often the case with other cameras — why burn bus bandwidth on uncompressed RGB data?) If you’re just saving the captured data to disk, it makes sense to just leave it compressed, but in this case, you’d want to decompress to a block of memory to run the barcode-search logic. It’s not easy code to write, but it’s very flexible and powerful. Three knocks against QTJ, though: there’s no onscreen Java component to render the capture data (although if you’re already decompressing to RGB, it’s a simple blit to get it into an AWT component), few Windows cameras include the needed QuickTime VDIG component (though you could presumably volume-license Abstract Plane’s software VDIG to deal with this), and of course QTJ doesn’t exist at all for Linux or other platforms.

I’m going to have to conclude at this point that there is no good cross-platform way to write this application today (although if you can think of something I’ve overlooked… sdljava maybe?… please post a comment). It’s a shame because the idea of using web-connected PC’s as barcode readers has myriad useful applications. Maybe the fact that Java, in all its bulk, can’t do it is a failure, but it’s also an opportunity, since Flash can’t do it either. See the “Rebooting Java Media” series for where I think things should go from here. And don’t expect Flash to sit still for long.

Booster Packs

What I set out to do in this blog was to propose a plausible, innovative application, not just a port of existing apps (”hey, let’s write iTunes in Java!”), and assess the abilities of Flash and the various Java media libraries to implement it as either an applet/presentation or a web app. There are surely lots of other valid business needs for media that have yet to be addressed by cross-platform executable environments like Flash and Java.

Again, a key to my thinking on this issue is trying to understand what application developers are thinking and why they do or don’t choose Java. Part of the reason this is important to me is that, as a Mac user, I keep seeing media devices come out that need a small amount of software support, and this software is invariably written as Windows executables. Like the “Lightning McQueen” digital cameras we got our kids for Christmas. Or the Sony Reader. The real functionality, and the value, is in the device, and surely the device makers aren’t wedded to Windows. And so I wish I could go to the relevant product managers and ask two questions:

  1. Did you consider Java for your software needs and find it lacking, or
  2. Did you never consider Java at all?

Honestly, the first answer would more reassuring, because something can be done about that. If Sony or Disney’s answer was “sure, we know Java’s cross-platform, but there’s no USB support, so how’s it going to work with devices?”, then at least there would be an obvious course of action to take. Things are worse if the answer were like “Java, isn’t that just for servers”, or “we’ve had bad experiences with Java in the past”, or the classic “no, it’s too slow”, then we have a much more serious problem. If Java has too little credibility after 10 years to even be considered for cross-platform development, then it’s done for, plain and simple.

I have a lurking suspicion in my dark, dark heart, that Sun has more people working on NetBeans than all other desktop Java technologies combined. If I knew that were the case, that this one IDE was the endgame of Desktop Java, I’d be pretty upset. And I’d probably be done with Desktop Java.