September 2004 Archives

William Grosso

AddThis Social Bookmark Button

Related link: http://www.simonstl.com/articles/xbrowse/xbrowse13.html

I was talking with George Feil the other day and he said something I thought was very interesting.

“You know,” he said, “when you’re dealing with Enterprise Apps, there’s no reason to generate HTML on the server. You should generate XML, and then use XSLT or CSS in the browser, right? Surely browsers are close enough to standards, anbd far enough along that, at this point, if you insist on a recent browser, it should work.”

“Huh?” I thought. “That might be true. It’s certainly not something I’ve thought about recently.”

From an architectural perspective, it’s interesting. In the Java universe, there have been a lot of new UI frameworks, like Tapestry which look very nice. But which, at the end of the day, are more about better ways to generate HTML and to design server-side apps, than they are about ways to generate XML.

And there’s been a lot of work done on richer client interfaces. From Flash to Avalon to General Interface and all points between, a lot of people have been noticing that the client machines are incredibly fast and trying to figure out what to do with all that memory and all those CPUs.

George’s point is an elegant one. Suppose instead of using those client CPUs to deliver enhanced functionality, you simply wanted to offload some of the server’s workload. Could you use XML to do that? Stay in XML and stay closer to logical layout on the server, and then generate the HTML in the client browser.

So here’s the data points:

Are people doing this? Am I missing something big or obvious?

Russell Miles

AddThis Social Bookmark Button

Related link: http://www.mac.com

.Mac got a much needed boost in terms of its email and general storage services today. Following on from Jason Deraleau’s blog on why it is worth shelling out more money for .Mac, the message from Apple appears to be that .Mac is alive, well supported and getting better all the time.

These improvements could be a seen as a bit of a case of “it’s about time Apple”. Being an author that is constantly emailing updates and edits to the O’Reilly team (one day my first draft will be production quality, but hell is still too warm at the moment), I would argue that the upgrade to 10 megs maximum email size alone is worth a sigh of relief. Combined with the 250 megs as standard on storage, the additional upgrade to 1 gig if I had the cash and the great suport and occasional extras from the Apple crew; I won’t be changing my subscription to another provider just yet.

Now I’m just waiting for free OS upgrades to complete the mix - might be waiting a while for those though!

Anyone have any more suggestions of what Apple could do to improve the .Mac Service?

Marc Hedlund

AddThis Social Bookmark Button

Related link: http://www.bloglines.com/about/pr_09282004

Later today, we’ll be posting an article on the new Bloglines Web Services. I think this announcement is a big deal for the RSS/Atom world — not only will it ease bandwidth consumption, I think it also opens up possibilities for great new applications. Check back later today to learn more, or head on over and start developing!

UPDATE: The article has been posted.

Tim O

AddThis Social Bookmark Button

I hate to say it, but I think Apache Xindice has entered a state of dormancy. There is one dedicated committer working towards the 5th beta release of 1.1, and there is very little activity on either the user or developer mailing lists. Because of this, I started working with dbXML, and I’m impressed. Here’s a screenshot of dbXML Administrator, a client application bundled with dbXML 2.0:


image

Screenshot of dbXML Administrator

This tool makes all the difference, from it you can create, read, update, or delete collections and documents.
You can view documents and query results as XML, in a grid format, or apply a stylesheet to documents from a particular collection. This is relief for anyone who had grown used to Xindice’s “Ugly Debug Tool”. If you have an application using the XML:DB API, you can switch over to dbXML 2.0 with a minimum of fuss, and the install process for dbXML is straightforward.

For more information about dbXML 2.0, take a look at the dbXML Group and the dbXML 2.0 Programmer’s Manual.

Using dbXML?

Tim O

AddThis Social Bookmark Button

So, I keep this blog as politically “neutral” as I can. You came here for technology, so I’m not going to sound off about deficit spending or ruminate about “strategery” on O’Reilly.com, but CSPAN was on in the background all day and I couldn’t help myself. I have a big CSPAN problem, and I have wasted many a day watching the minutae of governance. Some prefer sports; I opt for Senate hearings and BookTV. Read what the founder of CSPAN - Brian Lamb - said about TV in ‘99:

“I grew up with television being controlled by three men living in New York City. Everything trickled down from those three men, William Paley, Leonard Goldensen, and David Sarnoff. Now, they were just being good businessmen. They maneuvered to get the licenses first in radio, then in television. They set the standards for what television was going to be, and because there were only three, they tried to appeal to everyone all the time. But that is just not democracy.” - Brian Lamb, August 1999 Organization of American Historians

CSPAN makes our government more transparent. And, Brian Lamb’s experiment in transparency reminds me of the transparency of open source decision-making. What makes something like Linux successful isn’t the availability of code; most people never even see it. The important part of open-source is transparency and a decision-making process open to public criticism. Do I believe that transparency makes a better piece of software? Not at all; any project can fail. But, I do feel more comfortable trusting a decision-making process I can follow. In other words, open source isn’t about source at all, it is about transparency and the ability to watch and/or participate in an ongoing discussion. Open-source is a kind of digital democracy (or meritocracy).

Take Lamb’s quote and think of it in the context of technology (”s/licenses/patents/g” and “s/television/internet/g”). Although, not a direct analogy, CSPAN provides the same service as a developer mailing list at the ASF or a discussion list at the W3C. Why should only Microsoft have a say in defining the format of our office documents? or the architecture of our operating systems? So, keep Lamb’s quote in mind when you think about AOL, Debian, and Apache’s rejection of the Microsoft Sender-ID proposal. Letting Microsoft hold a patent to a pivotal piece of core internet infrastructure is “just not democracy”.

What do you think? Should we elevate Brian Lamb to his rightful position as “Pioneer of Open-source Government”?

Marc Hedlund

AddThis Social Bookmark Button

(For an overview of the G$D/Groovy series, see <http://www.oreillynet.com/pub/wlg/5789>.)

Finding Groovy help when this blog is not enough

On the “teach someone to fish” tip, here is a set of links that can help you find help on Groovy when you need it. These will go from the most general, non-expert and accessible methods to the more onerous methods. Please feel free to give additional links in the comments.

  • The Groovy Language Guide: this tutorial will get you up and running on the major features of Groovy, particularly if you are already familiar with another similar language like Python or Java. Many explanations are very brief, but the overview of the language is helpful.
  • The Groovy Reference Card (PDF): this excellent guide can be very handy in remembering Groovy operations when you are already familiar with the language. Some of the edges are a little rough but the information is very helpful.
  • The GDK Reference Page: be careful! Some of this information is confusing! The methods on this page are provided by default for Groovy objects, and they are very helpful in using Groovy to get ${stuff} done very quickly.
  • The Groovy User’s List Archive: this mailing list archive probably contains the answer to most early questions about using Groovy. You can also subscribe if you plan to have regular questions. The list volume is low and people are very helpful.
  • The Groovy Home Page: there’s a lot of information in here, just incomplete and organized in a very wiki-like fashion (since it is generated from a wiki!). Worth a browse if you’re stuck. Particularly see the articles page.
  • The JustGroovy Blog: this site contains news and articles about Groovy — add it to your RSS reader or search its archives for more help. (Thanks to juneau for the suggestion.)
  • The Groovy Tip of the Day: be careful! Some of these tips do not work in the released version of Groovy! However, there are some great ideas for getting comfortable with the Groovy language and using it more productively.
  • The Groovy Developer’s List Archives: while users should stick to the user list mentioned above, if you have a “deep” problem that requires a patch, it may have wound up on this list already. Of course you can also send patches here if you find a good fix.
  • The Groovy Bug Database: if you’re stuck, probably someone else has been stuck, too, and if you’re lucky they filed a bug. The developers monitor this and good info can be found in the bug notes — which sometimes doesn’t make it back out to the docs.
  • The Groovy Javadocs (for Java-comfortable folks only): the best documentation of Groovy is in the Javadocs. Get familiar with these and you’ll be able to locate most problems. Particuarly check out the default Groovy methods page. You’ll need to be comfortable with Javadoc information to get much value out of this. Note that this link goes to the live version of the docs, which reflect the current checked-in code, not the latest release. For the release docs, look in $GROOVY_HOME/docs/xref/index.html on your machine.
  • The Groovy Source Browser: come on, you can do it — dig in and find the real source of your problem. When you find it, fix it, and send it to the dev list. Everyone will thank you!

A long list — hopefully you won’t have to go that far. But it’s handy to have when you need it. Remember that Groovy is a young language and is changing quickly, and having a broad set of resources will make you happy.

Marc Hedlund

AddThis Social Bookmark Button

(For an overview of the G$D/Groovy series, see <http://www.oreillynet.com/pub/wlg/5789>.)

The each() method and collections

The each() method, which is a default method on every Groovy object, is a little bit magic. If you call each() on an integer, you get the integer back:


aNumber = 27;

aNumber.each() { println "${it}" }; // prints: "27"

If you call each on a string, you get one letter of the string (in order):


myName = "Marc";

myName.each() { println "${it}" };

// prints:

// M

// a

// r

// c

With collections, each() switches its behavior based on the type of collection. Lists hand each() one item of the list (in order), as you’d expect:


languageList = [ "java", "perl", "python", "ruby", "c#", "cobol",

"groovy", "jython", "smalltalk", "prolog", "m", "yacc" ];

languageList.each() { print " ${it}" };

// prints:

// java perl python ruby c# cobol groovy jython smalltalk prolog m yacc

Maps behave differently depending on how many arguments the closure expects. If you pass each() a closure that expects one argument (or you specify no arguments and use the default it parameter), the closure receives one Map.Entry per call. This isn’t very convenient — you have to, you know, make Java calls and stuff. If your closure specifies two arguments, though, the first argument will receive the key, and the second the value, for each entry in the Map. For instance:


weekMap = [ "Su" : "Sunday", "Mo" : "Monday", "Tu" : "Tuesday",

"We" : "Wedensday", "Th" : "Thursday", "Fr" : "Friday",

"Sa" : "Saturday" ];

weekMap.each() { key, value | println "${key} == ${value}" };

// prints:

// Su == Sunday

// We == Wedensday

// Mo == Monday

// Sa == Saturday

// Th == Thursday

// Tu == Tuesday

// Fr == Friday

The different behaviors of each() aren’t documented very clearly on the Groovy site, so it helps to keep in mind where the method varies.

William Grosso

AddThis Social Bookmark Button

Related link: http://sourceforge.net/projects/testrunner/

A friend, who’s used to the high-powered, high-priced, world of test-case management and execution recently asked me: what if Don’t have the budget? What can I use then?

I didn’t have an answer. So I’m putting this out to the general community: are there good open-source tools for test case management and execution.
At first glance, Bugzilla Test Runner looked like it might be a viable application. But development on it seems to have slowed, and it doesn’t look like a viable option.

Any thoughts?

What do you use to track test cases?

Marc Hedlund

AddThis Social Bookmark Button

(For an overview of the G$D/Groovy series, see <http://www.oreillynet.com/pub/wlg/5789>.)

Basic closures in Groovy

Closures are data types just like integers or lists, but the “data” they contain is itself a block of code. To define a closure, use curly braces to contain the closure code:


myClosure = { println "Hello, world!" };

Now you can run your closure using the call() method provided by
the href="http://groovy.codehaus.org/apidocs/groovy/lang/Closure.html">groovy.lang.Closure
class:


myClosure.call(); // prints "Hello, world!"

You can also run your closure as if it were a method or keyword:


myClosure(); // prints "Hello, world!"

Inside each closure, Groovy defines a default variable,
it, for one argument passed to the closure. This makes
it very easy to have a single parameter for your closures without
having to explicitly declare it. The it variable works
just like Perl’s $_ variable in subroutines:


namePrinter = { println "Hello, ${it}!" };
namePrinter("John"); // prints "Hello, John!"

What if one argument isn’t enough? If you want to use more than
one argument in your closure, you can define them on the first
line
of the closure, and end the line with a pipe character
(|), like this:


nameAndAgePrinter = { name, age | println "${name}, age ${age}" };
nameAndAgePrinter("Lisa", 54) // prints "Lisa, age 54"
nameAndAgePrinter("Henry", 27) // prints "Henry, age 27"

So far, we’ve been defining a closure by giving it a variable name
(”nameAndAgePrinter” in the example above). For any method that takes a
closure as an argument, we can use a shortcut to define the
closure directly after the method call. For instance, let’s say a
method named respond() takes a closure as an argument,
allowing you to tell the object to “respond” using code you define in
the closure. The respond method might have documentation like
this:


myclass.respond(groovy.lang.Closure closure)

This says that the respond() method takes one argument, a closure.
You could define the closure as we have above:


response = { // response code... };
myobject.respond(response);

Alternatively, we can define the response inline by placing it just
after the method call:


myobject.respond() { // response code... };

When Groovy sees a closure defined in this way, it will pass that
closure as an argument (the last argument, if there is more than one)
to the named method. You’ll see that this shortcut is very
useful for writing clear and readable code.

To learn more about using closures, check out Martin Fowler’s discussion of them, which is not specific to Groovy but which provides very helpful examples and discussion. (Thanks to Nelson for the pointer!)

Marc Hedlund

AddThis Social Bookmark Button

Related link: http://groovy.codehaus.org/Installing+Groovy

(For an overview of the G$D/Groovy series, see <http://www.oreillynet.com/pub/wlg/5789>.)

Installing Groovy with a working ~/.groovy/lib

The Installing Groovy page contains concise and accurate directions for getting Groovy installed — go ahead and follow them. (Note that you’ll have to have Java installed already.) If you make your way to the Running Groovy page, though, you’ll see a section called Adding Things to the Classpath. It reads (typos corrected):


When running command line scripts or interactive shells you might want to add things to your classpath such as JDBC drivers or JMS implementations, etc. To do this you have a few choices:

  • create a ~/.groovy/lib directory and add whatever jars you like there
  • add things to your CLASSPATH environment variable
  • pass -classpath (or -cp) into the command you used to create the shell or run the script

The second two options mentioned will work; by default, the first — adding jars to ~/.groovy/lib — will not (as of 1.0-beta6). You have to take one more step to complete your installation.

As documented in bug 429, a bug in a library Groovy depends on has led the Groovy developers to comment-out support for the ~/.groovy/lib directory. You can re-enable support for this library by:

  • Creating the ~/.groovy/lib directory;
  • Going to your GROOVY_HOME/conf directory;
  • Editing all of the *-classworlds.conf files so that
        # load user specific libraries
        #load ${user.home}/.groovy/lib/*
    

    is changed to

        # load user specific libraries
        load ${user.home}/.groovy/lib/*
    

After doing this, your ~/.groovy/lib directory will work.

I don’t personally believe the groovy team has made the right call on this bug — I’d rather have the interpreter create the ~/.groovy/lib directory on startup if it doesn’t exist, and I’d rather the documentation were correct! But, once you know about it, it’s an easy fix.

Marc Hedlund

AddThis Social Bookmark Button

Related link: http://groovy.codehaus.org/

I’ve been using and very much enjoying Groovy, a scripting language for the Java Virtual Machine (JVM). It’s a great scripting language, and it provides you with access to all of the class libraries available for Java. In other words, you get all the benefits of Java without having to write so much bloody code all the time. As someone who learned Perl and then Java and then a set of other languages, I’d also say Groovy doesn’t have the write-only, read-none properties of Perl, nor the Doctorine of Greatest Surprise of Python, nor the too-Perlish-but-why question of Ruby. These are my aesthetic opinions; yours will certainly vary. But I find Groovy to be a very comfortable, lightweight scripting tool without the drawbacks of these or other similar options.

Groovy is early in its life, and while it is on the path to standardization through the Java Community Process, it still has a lot of rough edges. On top of that, the documentation is very scant and in some cases downright speculative (some feature ideas are documented as features). As an O’Reilly person, I think that’s great — we will be able to sell lots of great books making that situation better. As a developer wanting to get work done right now, though, it leads to a bit of misery — not enough to abandon the language, but enough to have spent too much time reading Java source code to figure out the source of my Groovy bugs.

So, this is the first in a set of blog posts I’m calling the “Get ${stuff} Done With Groovy” series — I’ll just call it G$D/Groovy from now on. They will not be tutorials nor step-by-step introductions to the language — I’ll assume you have made a stab at reading the language guide. Instead, each one will take one particular task I’ve learned how to do — particularly one that took some amount of deciphering to figure out — and demonstrate it. A lot of the information will be specific to the current version (1.0-beta-6, as of this writing), but some will not. Be sure to check for notes on the current version to see if you can skip some of the workarounds suggested.

All of the G$D/Groovy posts are now indexed on a reference page.

I think you’ll like Groovy, and I’d like to make it more easily usable by a wider audience, so it grows and thrives.

Don Schwarz

AddThis Social Bookmark Button

Tired of dealing with checked exceptions?
These people are.

Compile-time checking of exceptions is a useful language feature, but
often this feature can become a nuisance. This is particularly true
when checked exceptions are used (in my opinion) inappropriately, for example to
signify an unavailable resource or a coding error.

When calling a method that throws a checked exception, you have a few
options. You can catch the exception and handle it yourself; however,
this is not always possible. You can declare that you also
throw the checked exception, but this can lead to a proliferation
of throws declarations as exceptions need to propagate
farther and farther up the stack. Many exceptions are best handled at the
top level of the stack, or even outside of the stack in an
UncaughtExceptionHandler, and it doesn’t take many checked
exceptions before this process can become untenable.

Another popular option is to convert checked exceptions into unchecked exceptions
by using a wrapper exception. For example:

public Object loadObject (int objId)
{
    try {
        Connection c = Database.getConnection();
        PreparedStatement query = conn.prepareStatement(OBJECT_QUERY);
        query.setInt(1, objId);

        ResultSet rs = query.executeQuery();
        ...
    } catch (SQLException ex) {
        throw new RuntimeException("Cannot query object " + objId, ex);
    }
}

Chaining exceptions in this manner has a few advantages.
Exception.printStackTrace() will print the entire chain of stack
traces, so no information is lost in the wrapping process. Each
exception can also annotate the stack trace with additional debugging
information (for example, by appending the object identifier in the
above example).

There is one major disadvantage, though. We’ve preserved the
debugging information contained in the original exception, but we’ve
lost the ability to easily trap exceptions based on this information.
For example, let’s assume that the following method is near the top of
the stack. This will no longer work, as the exception thrown is now
a RuntimeException rather than an SQLException.

public DataSet loadData ()
{
    for (Connection c : getAllConnections()) {
        Database.setConnection(c);

        try {
            return attemptToLoadData();
        } catch (SQLException ex) {
            logException(ex);
        }
    }
    throw new PersistenceException("cannot load data");
}

However, there is a fourth option. We can actually throw a checked
exception at run-time without either the compiler or the bytecode
verifier knowing about it. The public class sun.misc.Unsafe
provides a wide variety of “dangerous” utility functions, including
direct memory access (which can be used to simulate pointer arithmetic)
and object access which is much faster than reflection. In subsequent
posts I will discuss many of these other features, but in this post I am
focusing on the throwException method.

public void someMethod () // no "throws" clause!
{
    getUnsafe().throwException(new Exception("test"));
}

The Unsafe class and throwException method are both public, so
any security surrounding this class are based entirely on whether you can
obtain a instance of it.
Code that is distributed along with Java can simply call the static factory
method Unsafe.getUnsafe(), but we’re not quite so lucky. This
method verifies that the calling class was loaded through the bootstrap
ClassLoader, so we would have to modify the -Xbootclasspath
for this to work for our own code. However, there is another option — if
the current SecurityManager allows it, we can override the accessibility
of the static field that stores the Unsafe instance and retrieve it
directly.

public Unsafe getUnsafe ()
{
    try {
        Field field = Unsafe.class.getDeclaredField("theUnsafe");
        field.setAccessible(true);
        return (Unsafe)field.get(null);
    } catch (Exception ex) {
        throw new RuntimeException("can't get Unsafe instance", ex);
    }
}

“Why does such a method even exist?”, you may be asking yourself. Surely
Sun wouldn’t throw any checked exceptions from a method that doesn’t
declare those exceptions, right? That’s cheating!

Well, in fact, they do — in Class.newInstance(). If you get
a java.lang.reflect.Constructor and call newInstance()
on it, you will receive one
of three exceptions: IllegalArgumentException,
IllegalAccessException,
or InvocationTargetException. The first two exceptions are caused by
the act of invoking a method through reflection, but the third is used
to wrap any other exceptions which are thrown from within the constructor.

However, Class.newInstance() behaves differently; it will not throw an
InvocationTargetException. It may throw an IllegalArgumentException
or IllegalAccessException, but if
any exceptions are thrown from the constructor they will be thrown directly,
rather than being wrapped with an InvocationTargetException.

In fact, if you look at the code for Class.newInstance(), you’ll see this:

private Object newInstance0()
    throws InstantiationException, IllegalAccessException
{
    ...

    // Run constructor
    try {
        return tmpConstructor.newInstance(null);
    } catch (InvocationTargetException e) {
        Unsafe.getUnsafe().throwException(e.getTargetException());
        // Not reached
        return null;
    }
}

I’m not necessarily advocating the use of this method to avoid compile-time
checking of exceptions, but I think it’s important to know that you have
this option. It’s a bit of a hack, but hey — if it’s good enough for Sun
it must be good enough for us, right?

There is one place in particular where I can see this method being useful.
Some Aspect Oriented Programming (AOP) frameworks, such as AspectJ, have
support for automatically “softening” exceptions by wrapping them with an
unchecked exception. However, this suffers from the same problem that
we highlighted above — the identity of the exception changes after it
is wrapped, and therefore it is more difficult to catch the exception at
a higher level. It may be interesting to give these frameworks
support for using Unsafe.throwException instead.

Another interesting idea is to implement an annotation that softens
exceptions in this manner. Using either source-code instrumentation
(via apt) or bytecode instrumentation (as I described
in my
Peeking Inside the Box
articles), code could be inserted that would use
Unsafe.throwException to soften the exceptions. Unlike the AOP
approach, this would make it clear from the source code what is happening.

@Softens{SQLException.class}
public void loadData (int objId)
{
    // ...code that throws an SQLException...
}

Have checked exceptions done more harm than good in your development career? If so, how do you deal with it?

William Grosso

AddThis Social Bookmark Button

Related link: http://news.com.com/2100-1012-993226.html?tag=lh

About a year ago, Scott McNealy said something that struck me as subtly wrong. I didn’t think much about it at the time, but as I’ve been helping SDForum put together this year’s Distinguished Speaker Series, I’ve been thinking about it more and more.

Here’s what he said:

The world is down to two developer camps: One is .Net, the other is Sun ONE Java.

Ignoring the “Sun ONE” part as pure propaganda, it struck me as an interesting assertion. Where I live, there are two big camps and then a lot of little tents (it’s hard to say
PHP doesn’t matter. But it’s also hard to say it’s very important for most developers. It’s a tent, not in either of the big camps).

Part of me immediately wondered “Hmmm. Am I part of the Java camp?” Most of me responded
immediately. “Well, duh. See those lines of code on the screen? They’re in Java. So of
course I am.”

But, still, part of me found the description a little disquieting.

Then there was Tim Bray’s href="http://www.tbray.org/ongoing/When/200x/2004/01/15/OpenSourcePerson">description
of his recent job search. In which he wrote:


So I’m sitting there talking to this real smart guy who’s got a strategic job way up in
a Silly Valley titan and maybe we can do a deal, we’re winding up and I said “Anything
else by way of questions about me?” and he said “Nah, I got you pretty well
triangulated, you’re an Open Source Person.”

Which, I guess, more and more of us are.

And that resonated with me a little bit as well. It didn’t strike home, not totally, because I don’t often contribute to open source projects. The
companies I work for are mostly closed source, and the few times I’ve contributed to an
open source project, it’s more been out of I’ve got this bug fix sitting here and
it’d be nice if it was included in the next release
than it was out of any sort of
spirit of camaraderie.

But still. In another sense, in the Scott McNealy sense, I am an open-source person. While I work on closed source applications, there’s another question: in terms of mindshare, who sets the agenda in the technological part of my brain ?

By and large, it’s the open source community. When I think of myself as a developer, I think of myself as someone who writes to an open source stack, using a large set of frameworks and libraries that were developed in the open. That’s where my technological agenda is set. My code is, technically speaking, written in Java. But I pay far more attention to Apache than to Sun.

Scott McNealy was probably right in saying there are two large developer camps left. And he was probably very wrong when he identified one of them as “Sun ONE Java.”

And all of which leads me to think that the debate about open-sourcing Java is a non-issue (what it means, and whether or not it happens, just doesn’t matter). And that I need to understand open source, and how it impacts software development, a heck of a lot better than I do now.

Which leads to something I’m very excited to have helped put together. href="http://www.sdforum.org/dss">SDForum’s
Distinguished Speaker series for the 2004-2005 season is coming together around the idea of the “Software Commons.” The goal is to have 9 distinguished speakers talk about the common ground that’s being built in the open source community and to reflect on what it means and how it’s changed the way we all think of our craft. The series
starts with Larry Lessig on September 23 and continues on through a list of incredible speakers including
Jason Hunter, Steve Weber, Craig Newmark, Guido van Rossum,
and Howard Rheingold.

Sandy Rockowitz and Bebo White, the chairs of this series, have done a fantastic job in putting it together.

Are you an open source person?

James Elliott

AddThis Social Bookmark Button

On Friday afternoon I was handed a CD-ROM containing some movies created by tenth grade students who’d participated in the University of Wisconsin’s Information Technology Academy. My employer is sponsoring a new charity run that supports the Technology Education Foundation, and wanted to be able to show some of the students’ achievements that are being supported by the foundation. But the movies would need to be shrunk in order to become realistic downloads; the current files were much too big.

Since I’m known around work as a Mac guy who dabbles in media, I was a reasonable choice. If my own resources weren’t up to the task, I’d be able to lean on my friend and co-author Marc Loy who has a post-production studio in his attic, and recently finished a book about DVD Studio Pro. How hard could it be?

Well, I quickly discovered that QuickTime Pro wasn’t able to work with the files at all; they were in Windows Media format. So I schlepped them over to Marc’s, where we tried Discreet’s cleaner 6. Surely that would do the trick. Alas, no dice. None of Marc’s professional software was able to extract the movie content.

I went home again, with a determined set to my jaw. I’d already spent more time on this than my colleagues would have wanted, but I can be like a bulldog when technology isn’t working the way I think it should. Time to deploy Google and see what other people had come up with. After a bunch more dead ends, I located a blog posting that pointed me towards Forty-TwoDVD-VXPlus. This “AVI Video Transcoding System” looked promising. And there was a trial download!

I installed it and fussed with it for a bit, but it kept failing to extract anything from my movies. By this point, it was very late at night (well, early in the morning, actually) and I wasn’t thinking too clearly. The trialware nag message said the unregistered product would only extract the first two chapters of DVDs. I wondered if that meant that other capabilities, like converting WMV files, worked only after purchasing a license? Surely it was too late to ask for support, especially during a weekend, but perhaps the e-Commerce system was automated. It would cost $15 to register; I was willing to take that gamble.

Alas, even after rapidly receiving my registration code, I was still unable to convert the files. Looking at the error messages in the console more closely, I determined that the specific format of the impenetrable movie files was WMV9. At this point I was willing to concede defeat. I composed an email describing my hopes and failures to the support address at Ronin no Sakura Kai, thinking that perhaps someone would see it at some point during the weekend, and maybe get back to me with some advice.

Well, much to my surprise, before I could even step away to go to bed, I received a very friendly and apologetic reply. I’d misunderstood the nag notice (as, deep down, I knew all along) and not even this software could read WMV9 files.

Stunned (to have heard so immediately from a real person), I sent back a grateful reply and asked if they had any ideas about other approaches I might take. Once again, I quickly heard back, with a suggestion that I might look into Ambrosia’s Snapz Pro X, again with an apologetic note that it was somewhat expensive by comparison.

As it turns out, I already owned a license for the still capture capabilities of Snapz Pro (it’s incredibly useful when you write technical books and articles, since you always want good-looking screen shots, especially if you can get them easily). I’d never previously needed the movie capture capability. I’d fleetingly thought about it earlier during my Google searches, but put the idea aside in my search for a more plausible approach to extracting movie data. Now, however, no such plausible approaches seemed to exist, and I’d just received a report that other people had successfully used Snapz Pro to capture the content of WMV9 files.

OK, I was off again. Another $40 upgraded my Snapz Pro license to include the movie capture capability. So, to get the WMV9 movies into a form that I could work with, in order to re-encode them at smaller sizes and bit rates, I would have to play them in Microsoft’s Windows Media Player, while capturing a movie of the region of my screen in which the movie was playing.

Talk about a baroque solution! And I wasn’t there yet–for some reason, the option to record the audio from the Mac audio system was grayed out within the Snapz Pro interface. Well, Ambrosia’s an established company, they should be able to help me out… But no! Their FAQ and web forum were down! I sent them a pleading email asking for help, and have yet to hear a single thing back. It seems like a really bad idea to take down your online support forum during a long weekend when you don’t have people staffing your email support lines. I guess sometimes smaller developers still give you better service.

Happily, I eventually figured out the problem on my own. I had recently migrated to a new PowerMac, and used Apple’s new migration tool to move all my software and preferences, including Snapz Pro. It turns out that the Snapz installer adds a new driver to the OS X kernel, and (unsurprisingly) this had not been copied over by Apple’s tool. Re-running the Snapz installer enabled the “Mac audio track” check box, and I was able to capture both the audio and video from the movies.

In order to avoid re-compression artifacts as much as possible, I set Snapz Pro X to capture the screen movie with no compression of audio or video. It had to work hard, and ended up using over sixty gigabytes to store all the captured movies, but it did a perfect job. Snapz is a very useful tool, and works extremely well when it’s properly installed and understood. With more help from cleaner 6 at Marc’s, I was able to produce some nicely reduced versions of the films and burn them on a CD for our web designers.

But I still seethe over all the extra work, and silly hoops I had to jump through, because vendors feel compelled to create proprietary formats for storing information, and make it hard for other people’s software to simply read and write the information to achieve whatever goals their users might be pursuing.

Few people would have gotten as far as I did, and I’m still not sure this time was well spent. OK, it was for a good cause, and I’m glad I accomplished what I set out to do. But how many other worthwhile efforts have been dashed on the shoals of proprietary formats? And how much worse will it get before we demand better?

Have you had a similar experience you’d like to share? Or does anyone know a better way to convert WMV9 movies to portable, editable formats on a non-Windows platform?

Marc Hedlund

AddThis Social Bookmark Button

Related link: http://mh34ea.VOTEorNOT.org/

What a great idea — from the guys who brought us Hot or Not, a viral
contest to get people to register to vote. One person who signs up and
can prove they were registered to vote for this election wins $100,000;
if someone referred them, the referer also gets $100,000.

Here’s my referral link — follow it, sign up and get your own, and pass
it on.

http://mh34ea.VOTEorNOT.org/

Yay economics versus apathy!