Open standards are clearly a good thing. Hurrah for open standards, etc. Nail my hat to the ceiling!

But anyone who has been involved in community and consortium committees where there are commercial rivalries engaged knows that the thing that kills or corrupts a standard is when the spirit of mutual accommodation is overtaken by the spirit of competition. When I look over the standards that I have been to one extent involved with, at ISO, W3C and tangentially at IETF and OASIS, the golden rule is that the standards that come out of a nasty process have problems. The rancour during the Open XML debates does not auger well either for ODF and Open XML, in this respect, but I am an optimist.

The trouble with the ideal that people seem have of “open standards” is the extremely pragmatic one: how do we trust the committee? Who appoints these elders? Now this is something that MS have brought up about OASIS ODF, that ODF people have brought up about ECMA TC45, and which will undoubtedly be brought up about ISO (though ever more tenuously) by one side or the other no matter what the result is at ISO, sooner or later.

I think the problem is that rather than talking “open standards” we need to be talking as much of “verifiable vendor-neutrality”, if that is the goal for our public policy. It is nice that ODF and Open XML are open standards by the academic definitions, but it does not get us to where we need to be, and legislation based on mere “open standards” tickboxing will not succeed in getting vendor-neutral formats, if that is indeed the underlying aim. A standard may be as open as the grave yet not be good enough to the native format for an application, to bring up the current instance.

I have talked before about the need for profiles (to restrict extensible standards), and others are bringing up the natural progression from validation to test suites, but recently I am coming to believe that there has been a fundamental incorrect emphasis which self-defeats the open standards movements: the lack of scrupulous attention to the need for verifiable inclusiveness and fairness of process.

In other words (while not disagreeing that requiring “open standards” based on XML and ZIP is the best option now) the way forward for the EU and other governments is to direct and require that their application-suppliers participate in fair, mediated, format-harmonization standards processes (which is not the same thing as unification, and not the same thing as feature-leveling.) The boutique standards bodies, such as Ecma, OASIS, W3C are simply not constituted to be reliable here: they are democratic and two minor players will outvote one major one, which if done often enough will cause the major one to take off in a huff.

A company like Microsoft is famous for trying to keep effective control of its API. Some see Sun’s JCP as the same thing, it is a rational approach. So it is simply futile to imagine it is feasible that a company will give up control of an important asset to its business rivals: this is an issue that we have seen time and time again in the W3C, and is a tricky one in general, because it is not government’s role to solve this problem everytime. Some issues cannot be solved neatly or optimally or instantly, because of market forces (balanced markets or unbalanced ones!).

But the issue of public and archival formats for government and agency documents is clearly one where governments have a vital interest: the customer is always right. This is why I believe governments need to look beyond the current academic definitions of “open standards” and re-frame the issue as “How do we achieve verifiably vendor-neutral standards?”

Verifiable here means that there is a check in place that the committee proceedings did not discriminate against any player. Mere quorums and absolute votes are not enough. Vendor-neutral here means that the standard does not discriminate against any realistic players, either by making basic implementation too hard or by disallowing vendor-specific features or innovations or experiments, where appropriate. The only forum that I see that is set up for this kind of thing currently is ISO, where vendors can have committee input but only national bodies ultimately vote, but there may be some other approaches possible.

The way this approach would work would be for enough nations with purchasing power to get together and say something like “In 2012, we will require the use of verifiably vendor-neutral formats for office documents from ISO”: we will have observers at the meetings to assure us that companies are participating constructively and with regard to mutual advancement rather than competitive blocking behaviours.” Whether these formats are based on ODF with Open XML trimmings inside OPC packages, or whatever, it doesn’t matter. What matters is that the process does not turn into a competitive battleground, and does not discriminate against large or small players, and that there is sufficient compulsion and advantage to get vendor participation.

The problem is not “How can we get some group of vendors to agree on some format” but “How can we get any vendor to buy into a format unless all their needs and requirements are respected?” In that sense, without disparaging them in any way. the mooted attempt at reconciling DiTA and DOCBOOK and ODF cannot produce a universal result unless Open XML (and UOF and any other regional formats) are considered too. And though Patrick Durusau’s ideas on a meta-model for ODF and Open XML are good, it won’t work unless the procedural issue of a forum and a framework that allows for the appropriate kinds of plurality and profiling is in place. I don’t normally agree with Gary Edwards on much, it often seems, but these issues of fora and process are, in the absence of unilateral legislative action, the only practical way I can see of getting progress.

The issue of ODF versus Open XML will not go away, because they both have different drivers: the issue is not “open standards” but how to have vendor-neutral interchange formats that also allow good enough vendor-specific functionality to be used as native formats without embrace-and-extend, and most critically how to verify that the formats are not just open but also fair, participatory and non-discriminatory.