Recently I passed both my 44th birthday and my 15th wedding anniversary, just signed my daughter up for high school and was told by my doctor that my HDL was soundly thrashing my LDL. My beard, which I’ve worn since my early twenties, is now streaked with gray (a curse of red hair, I fear), and I notice that lately the stairs seem to have mysteriously begun to grow from one trip to the next. T.S. Elliot is beginning to become … relevant … to me. All signs, perhaps, that I am no longer the young spring chicken I once was.

As I was thinking about things to write for this particular column, this realization about age began to sink in about the standard that I’ve spent the last decade writing about. A decade is a long time in computer circles, especially when you figure that there’s only been five or six of them in the whole history of computing. XML has gone from being a “standard” that perhaps a couple dozen people worldwide knew about to a pervasive technology that is so well entrenched that many people don’t really even think much about it any more. We argue about the XMLification of word processing and spreadsheet programs, we debate whether Atom or RSS 2.0 will predominate, we shake our heads at the whole notion of web services and how the dominant web services protocol was designed largely by bloggers to let people know about their websites.

In short, while XML is not exactly doddering off to the rest home, its angle-bracket knees are no longer as flexible as they used to be. If it were a person you’d expect it to be muttering about those damn JSON punks and how property taxes and inflation are eating up its standard of living. It no longer is as flashy a technology as it used to be (even as Flash has been migrating to an XML format), and more than once I’ve run into twenty-something AJAX hot-shots who declare XML so yesterday (even as they write applications that bind AJAX objects to XML structures). It’s become the establishment, though in many respects I suspect that while its glory days are behind it, XML is becoming more integrated into the fabric of computing.

To that end, I wanted to offer up an assessment of where XML itself is going. As always, this is written by a guy in a coffee-shop, so take it with the usually assortments of saline condiments:

memcache.c:45:2: warning: #warning “Working around busted-ass Linux header include problems: use FreeBSD instead”

As seen in the libmemcahe 1.4.0.rc2 ./configure script. :D

I’d long ago told my Microsoft contact that I thought ultimately ANSI will abstain on the Open XML vote at ISO, due to an inability to achieve consensus, so I was quite surprised that they only missed out by maybe one vote in the end in the V1 technical committee. From emails from my friends on both sides of the table, it seems discussions at the V1 committee meeting became, if not acrimonious, then whatever the step beyond “robust” is. It will be interesting to see what INCITS/ANSI does to proceed, but it is delightful to see that the opponents of Open XML have started protesting that they really wanted Open XML all along!

(NOTE: I am withdrawing the rest of the blog, for now, while I think about whether it just perpetuates tit-for-tat sniping. Consequently, some of the comments no longer have their context. Apologies. The missing parts express surprise at Sun’s recent statement of support for Open XML, bring up what anti-trust means in the context of standards, points out the impracticality of adding thousands of pages of binary mapping documentation to the Open XML standard, looks at the bad logic used to justify it, brings up the notion of a “poison pill” which is a trick where an impossible-to-fulfill clause is added knowing it will cause refusal, and also thinks about voting procedures when there are multiple choices.)

This is a simple list of 10 general corrections to Open XML. There have been comments recently that pro-Open XML people are not contributing any fixes, so here are my big ticket items. They flow out of the principles that I mentioned in my blog before, and other discussions, and my distaste for non-verifiable specifications.

(If Australia decided to become a P-country and vote for ISO Open XML, these are the general corrections that I would submit to Standards Australia, apart from specific typos and unclear sentences. Whether they formed part of a ‘Yes with comments” or a “No with comments” wouldn’t bother me, since they all are fixable.)

It is here as PDF and some blog feeds will have it below in the extended entry. Download PDF file

Why would Open Source developers want to support Open XML becoming an ISO standard? Isn’t it from Microsoft, the great Satan? Isn’t is some kind of trick or trap to stop nice Open Standard ODF? Are we going to let this chance to overthrow the monopoly escape?

Now there have long been two different camps in the Open Source movement: those who think that it is important to have independent APIs and those who think that it is important to have Open Source clones of the most important proprietary APIs. This latter group is of course associated with Novel and the Mono effort is a good example: on their history, I don’t think they have much problem with Open XML going through ISO (Gnumeric’s Miguel de Icaza fits into this latter camp.) So this blog is more addressed at the first camp.

First, I would like to set the scene. I think the reasons for supporting Open XML at ISO become a lot clearer if we take a fairly hard-headed view of what is possible. Which is perhaps a nicer way of saying that I think some of the anti-Open XML case has been built on naively faulty assumptions about the miraculous power of ODF to disrupt Microsoft.

• No office suite or utility can afford to ignore any important format for long. So in a year’s time, every major office suite and utility (whether open source or not) will support ODF and Open XML, whether or not Open XML becomes an ISO standard.
• No vendor will adopt, as their default save file format, a format which does not support their particular feature set. So the only way that MS would make ODF its default file format would be if (when) it is improved to support Office’s feature set. (Support for Office’s feature set was not a design goal or activity of ODF’s development. See the second item in the minutes of the first ODF meeting for example. Or see ODF’s Gary Edwards "http://about.diigo.com/about/show?url=http%3A%2F%2Fwww.consortiuminfo.org%2Fstandardsblog%2Farticle.php%3Fstory%3D20070629070544217">comment that “There is no possible way anyone can claim that today’s OASIS ODF TC would welcome Microsoft and make accommodating changes to the specification!”)
• The poor state of ODF implementation by Open Source applications means that a too-fast adoption of ODF will backfire for Open Source developers. So paradoxically, supporting mandatory ODF too soon would be the kiss of death for open source ODF: bureaucrats will test applications and, finding them lacking, be forced to buy into a new generation of closed source tools (MS Office, IBM Lotus, Word Perfect, Sun Star Office.). If governments mandate ODF, it won’t exclude MS Office, in particular, and MS thinks their new GUI and features are competitive against other implementations, of course.
• No matter what format you use, the only way to get 100% page fidelity (apart from some good-as-read-only format like PDF) is to save in the native save format and re-open the same file using exactly the same application on exactly the same operating system configured exactly the same. ODF won’t give instant interoperability in the sense of full page fidelity; I’d expect the same would be true of Open XML on different platforms too.

Putting it all together, it means that there was never a chance that Microsoft Office would or could adopt ISO ODF 1.0 as its native and default format. So the real choice that faces us is whether we want Office to generate files in a format that MS controls with very few external checks (with all respect to Ecma) or to generate files in a format that MS instigated but which has the extra checks and balances that come from being an ISO standard. ISO standardization is not an Aladdin’s cave of democratic rights, and it is not a Pandora’s box for Microsoft, but it is way better than nothing. Because that is what the anti-Open XML people would achieve: no controls on Microsoft. Under the guise of supporting ODF.

If you, like me, are in the position where you don’t use MS products in your normal work lives, then you may not feel any urgency to support Open XML, but I think we at least should not oppose it. It is a good step forward.

We often read that Microsoft is doing this as some kind of sinister ODF spoiler. However, Open XML is a path they have largely been forced to take (though obviously they will try to make the path as beneficial to them as possible) in order to fend off continuing anti-trust problems in Europe: Microsoft went down this path after a very strong hint from the European Union: in the same report that recommends that ODF be submitted to ISO, the EU’s ‘Telematics between Administrations Committee’ recommended that Microsoft should consider the merits of submitting XML formats to an international standards body of their choice as well as improving the documentation, IPR issues and going all the way with XML.

I certainly support governments mandating that public documents should use standard formats: HTML and PDF being the primary two, and ODF after that, but also Open XML as a second source. However, having an ISO Open XML does not prevent any government from preferring ISO ODF. ISO standards are what are called “voluntary”, which means that they are not like laws where you have to adopt them. In my view, the drivers for ODF will continue unabated even after/if Open XML becomes a standard.

So, in my jaded view, ODF will not make Office go away, ISO ODF will not make Ecma Open XML go away, and ISO Open XML will not make ISO ODF go away. So I see no downside in Open XML becoming an ISO standard: it ropes Microsoft into a more open development process, it forces them to document their formats to a degree they have not been accustomed to (indeed, the most satisfactory aspect of the process at ISO has been the amount of attention and review that Open XML has been given), and it gives us in the standards movement the thing that we have been calling for for decades (see my blog last week that compared what Slashdotters were calling for in 2004 with the path that MS has taken).

In the 80s, there was a hilarious wrestler called George the Animal Steele. He was incredibly hairy, especially on his back, which was supposed to be emblematic of his sub-human state. His great flaw as a wrestler was that just as he was winning he would be distracted by the turnbuckle, often trying to eat its stuffing while his opponent recovered. I guess this is how I feel about the attempts to stymie Open XML at ISO: just as we have victory in our hands, with MS prepared to go XML and standards, along comes this distraction, ODF, which is great in its place but dumb, unworkable and counter-productive as a Microsoft buster.

Signs on the Sand: Saxon, NET and XInclude

Saxon, famous XSLT 2.0 and XQuery processor, supports XInclude since version 8.9. But in Java version only! When I first heard about it I thought “I have good XInclude implementation for .NET in Mvp.Xml library, let’s check out if Saxon on .NET works with XInclude.NET”. I did some testing only to find out that they didn’t play well together.

Turned out Saxon (or JAXP port to .NET, don’t remember) relies on somewhat rarely used in .NET XmlReader.GetAttribute(int) method (yes, accessing attribute by index), and XIncludingReader had a bug in this method.

Finally I fixed it and so XIncludingReader from recently released Mvp.Xml library v2.3 works fine with Saxon on .NET.

More goodness at the above linked post. Thanks, Oleg!

Some reference material and commentary for people interesting in getting to grips with the process. (This is a much revised version of the original page, which was confused. I think I have it right now, but corrections are welcome and I’ll edit them back into the page,)

Open XML Material

The text of ECMA 376 can be found at

• Ecma website
• Explanatory Report on Open XML Standard (ECMA 376) Submitted to JTC1 for Fast-Track

ISO Procedures

At the highest level, are the ISO/IEC Directives. These have two parts which can be downloaded from the ISO site:

• ISO/IEC Directives, Part 1 — Procedures for the technical work (2004, 5th ed.)
• ISO/IEC Directives, Part 2 — Rules for the structure and drafting of international standards (2004, 5th ed.)

These are the umbrella standards. ISO and IEC are two different organizations, but they share various common procedures. The branch of ISO/IEC that looks after Information Technology is called JTC1 (Joint Technical Committee 1). It has its own directives

• ISO/IEC JTC1 Directives (2007, 5th Ed. 3rd version)

which I found at the ISO/IEC JTC1 SC34 website. This version three from March this year has a few important changes since the previous version, so it is not enough just to find the 5th edition: I have been caught up on this. Similarly, you have to be careful that the general procedures given by the ISO/IEC Directives may be implemented or supplemented in the JTC1 Directives in a different way than you might expect.

There are two procedures for fast-tracking standards. ODF used the one called PAS (Publicly Available Specification), see JTC1 Directives Section 14. Open XML is using the one called Fast-Track Processing, see JTC1 Directives section 13. (The earlier version of this wass confused on this issue.)

PAS

The main place to find information on the “PAS” Fast-Track procedure is Appendix M of the JTC1 Directives The Transposition of Publicly Available Specifications into International Standards — A Management Guide. This gives procedures for an organization to be accredited as a submitter and the various procedures to be used. Because of the recent changes, there is still a little uncertainly as to how some of the material is to be interpreted in practice: in my opinion, the JTC1 directives cannot be interpreted in any way that goes against plain reading of the ISO/IEC Directives, which are the controlling documents: when there are disputes about the particulars of the JTC1 directives, the first place to look is at what the ISO/IEC Directives have to say.

When a PAS is submitted, it gets submitted with an Explanatory Report from the submitter.

Criteria for PAS Voting

The JTC1 Directives give detailed criteria, noting that it is not a matter of pass/fail for each. Some are mandatory:

• Organization-related criteria
• Co-operative stance
• Characteristics of the organization
• Intellectual property rights
• Document-related criteria
• Completeness
• Testability
• Suitability
• Availability
• Consensus

and the JTC1 directives give specific questions in each case. National bodies should ask themselves whether these are the questions they are in fact asking, one would presume. Then come lots of other possible questions, including market acceptance,

Fast-Track

Compared to PAS, there is little explicit help on how a Fast-Track draft should be evaluated.

• Before it is submitted, a draft can be submitted informally for comments to the appropriate committee (SC34) in JTC1 that looks after Document description and processing languages. Now Ecma did in fact approach several participants in SC34 individually for informal comments, asking “What do we need to have in the spec to make it acceptable?”, but not as far as I know through the formal “informal” channel spoken of here.
• During the one month “administrative review”, evident contradictions with ISO standards can be found. See my What is “Contradiction” of an ISO Standard and the followup No (showstopping) contradictions in Open XML?.
• National Bodies then get five months to prepare their votes and comments on the Draft of the specification. It is important to note, however, that section 13.1 twice describes the draft ballot as a vote on technical issues.
• If there was not a clear result, and yes and no votes with comments (see below for the possibility that a “no” vote must have comments) then a ballot resolution meeting is scheduled. The comments from different national bodies are sent to all bodies, who are supposed to form opinions about their acceptability in order to get the consensus at the ballot resolution meeting. After a few months study time, the meeting occurs, with no particular time schedule given. (I expect it may take two weeks, and the Convenor for the meeting is going to have to be fairly tough on time-wasting and long-windedness.)

Simple No Votes thrown away?

In the JTC1 Directives annex on PAS transposition there is an interesting note at at section M6.1.5 Ballot that says

Note to JTC 1: The ballot follows normal JTC 1 Fast Track rules in the case of an IS transposition, and normal ISP rules in the case of an ISP transposition. Negative votes by National Bodies have to be accompanied by comments giving the reasoning for the vote.

What is also interesting from the above note is the idea that negative comments have to be accompanied by comments. So simple “no” votes are actually not allowed. In fact, they must be disallowed! This aligns with ISO/IEC Directives Part 1 2.6.3 that for Enquiry Drafts, Abstentions are excluded when the votes are counted, as well as negative votes not accompanied by technical reasons. This is definitely an area where the people involved in the process need to get a good grip in which procedures apply!

I am not sure that there is any reason to expect that this does not also hold for the Fast-Track procedure. (Corrections welcome)

(Edits: Updated link to JTC1 Directives. Add paragraph on two different methods. Put PAS material in section. Remove or revise commentary on Open XML as a PAS. Added material on Fast-track. Re-order. )

Open standards are clearly a good thing. Hurrah for open standards, etc. Nail my hat to the ceiling!

But anyone who has been involved in community and consortium committees where there are commercial rivalries engaged knows that the thing that kills or corrupts a standard is when the spirit of mutual accommodation is overtaken by the spirit of competition. When I look over the standards that I have been to one extent involved with, at ISO, W3C and tangentially at IETF and OASIS, the golden rule is that the standards that come out of a nasty process have problems. The rancour during the Open XML debates does not auger well either for ODF and Open XML, in this respect, but I am an optimist.

The trouble with the ideal that people seem have of “open standards” is the extremely pragmatic one: how do we trust the committee? Who appoints these elders? Now this is something that MS have brought up about OASIS ODF, that ODF people have brought up about ECMA TC45, and which will undoubtedly be brought up about ISO (though ever more tenuously) by one side or the other no matter what the result is at ISO, sooner or later.

I think the problem is that rather than talking “open standards” we need to be talking as much of “verifiable vendor-neutrality”, if that is the goal for our public policy. It is nice that ODF and Open XML are open standards by the academic definitions, but it does not get us to where we need to be, and legislation based on mere “open standards” tickboxing will not succeed in getting vendor-neutral formats, if that is indeed the underlying aim. A standard may be as open as the grave yet not be good enough to the native format for an application, to bring up the current instance.

I have talked before about the need for profiles (to restrict extensible standards), and others are bringing up the natural progression from validation to test suites, but recently I am coming to believe that there has been a fundamental incorrect emphasis which self-defeats the open standards movements: the lack of scrupulous attention to the need for verifiable inclusiveness and fairness of process.

In other words (while not disagreeing that requiring “open standards” based on XML and ZIP is the best option now) the way forward for the EU and other governments is to direct and require that their application-suppliers participate in fair, mediated, format-harmonization standards processes (which is not the same thing as unification, and not the same thing as feature-leveling.) The boutique standards bodies, such as Ecma, OASIS, W3C are simply not constituted to be reliable here: they are democratic and two minor players will outvote one major one, which if done often enough will cause the major one to take off in a huff.

A company like Microsoft is famous for trying to keep effective control of its API. Some see Sun’s JCP as the same thing, it is a rational approach. So it is simply futile to imagine it is feasible that a company will give up control of an important asset to its business rivals: this is an issue that we have seen time and time again in the W3C, and is a tricky one in general, because it is not government’s role to solve this problem everytime. Some issues cannot be solved neatly or optimally or instantly, because of market forces (balanced markets or unbalanced ones!).

But the issue of public and archival formats for government and agency documents is clearly one where governments have a vital interest: the customer is always right. This is why I believe governments need to look beyond the current academic definitions of “open standards” and re-frame the issue as “How do we achieve verifiably vendor-neutral standards?”

Verifiable here means that there is a check in place that the committee proceedings did not discriminate against any player. Mere quorums and absolute votes are not enough. Vendor-neutral here means that the standard does not discriminate against any realistic players, either by making basic implementation too hard or by disallowing vendor-specific features or innovations or experiments, where appropriate. The only forum that I see that is set up for this kind of thing currently is ISO, where vendors can have committee input but only national bodies ultimately vote, but there may be some other approaches possible.

Lets imagine that we are transitioning into a “Document Engineering” style of architecture, so that we can model our entire business using old but not-as-outmoded-as-we-first-thought Data Flow Diagrams. At each data flow we need to ask the Exception Question: Does an exceptional document need human intervention or can it be dealt with automatically? Indeed, the expected answers to this question is probably what distinguishes the document community from the database community: the docheads would expect exceptions to be dealt with by humans who can monitor, fix and reset the production flow at all stages, the dataheads would expect exceptions to be dealt with by automated process, since humans involvement is at the input/output periphery of systems.

Obviously, the most “exceptional” kind of document is the invalid-against-a-schema document. However, Schematron allows a much milder (or tougher, depending on how one looks at it) bar: the presence or absence of any arbitrary pattern in the document can allow it to be marked as exceptional. (Schematron not only define valid/invalid, but it also allows complex dynamic diagnostic messages, and it also allows various flags to be set by assertions that fail.)

So the Exception Question then becomes a criterion for evaluating schema or constraint languages: when exceptional documents are to be sent to humans for intervention, does schema language A provide clear enough information to be usable by those humans. Similarly, when exceptional documents are to be sent to software (services) for intervention, does schema language A provide clear enough information to be usable by that software. Looked at in those terms, grammar-based systems do not shine. Grammar-based systems excel in all-or-nothing Great-Wall-of-China exclusion uses, but then throw the users (systems and humans) at the mercy of the validator-developer for the kinds of feedback and information possible, who has of course absolutely no idea of the problem domain of the schema. XSD is perhaps a little more organized in this regard compared to the other schema languages, because it defines a specific list of outcomes that can be found in the notional Post Schema Validation Infoset after validation.

But, the trouble is that, whether for humans or systems, the more that problems are diagnosed in generic terms (i.e. in terms of the markup) rather than in domain terms (i.e. in terms of the intended patterns, or dare I say semantics), the less chance that the diagnostic can serve any practical purpose for downstream systems. Notoriously this is true for system which “hide the markup” from the user: the grammatical errors are unavailable and incomprehensible to the users. Grammars have shown themselves over the last 20 years to make programmers more productive but to stupify end-users: the traceablity issue I raised this week on XML-DEV in response to one of Roger Costello’s excellent fishing expeditions is another head of the same Hydra.

I find there is little awareness of the health dangers of XML Conferences in our community. As indeed I was until it happened to me. It had me in hospital near death within a few days of returning from the XML Europe 2005 conference, and it took more than a year to recover, and I just heard of a friend hospitalized after a trip to another XML-related seminar.

So I am not being flippant here, but literally deadly serious. The problem is not dull speakers or conference food or sponsored talks on products or the body’s natural spasms reacting against the evils of WS-*. Open *, or * 2.0 as you might expect. The trouble is prolonged airflights, in particular intercontinental and interhemispherical flghts.

Many people are now aware of deep vein thrombosis and take good measures against it. (It is a real problem outside flight too: a friend of mine died of it last year during radiation treatment for cancer because he was too debilitated to move his limbs.) Dick Cheney recently had one for example.

But there is a more general risk in air travel: you are sitting in a box full of people at all sorts of stages of health, and you are making yourself fatigued. The excellently named Dr Dement from Standford says that fatigue reduces mucus production in the throat (and dry airplane air will accentuate this), which makes you more susceptible to coughs and colds and other airborn viruses. For example, I developed pericarditis which does not seem to be directly contagious however, many of the organisms that cause infections that can lead to … pericarditis are spread from person to person by coughing or sneezing.

Then there is of course the danger (or likelihood for some places) of food or water problems. Just because we know about it doesn’t mean it won’t happen. For example, on my last trip I got sick before India in Europe or Canada, while a friend became sick immediately after.

So here’s my travel tips:

• Never eat seafood when traveling. Actually, this advice comes from the aging queen, Elizabeth II, but it seems to have served her well. The trouble is that seafood poisoning is fairly catestrophic.
• Never travel for more than one intercontinental leg. My recent trip required me to go from Sydney to Toronto to Delhi to Sydney. So I scheduled 24 hour stopovers in San Francisico, Frankfurt and Bangkok. These cut down the flying times to more reasonable levels (Sydney to San Francisco is only about 14 hours fllight, the others are between 4 to 9 hours. Now, of course, that blows out the total time of a trip, and adds expenses for accomodation,
  etc. But knowing what I know now, you are crazy not to do something like this.
• Never travel overnight when there are day trips available. Sleeping on a plane is a complete gamble, even with the unpredictable delights of Stilnox (Ambien), plus you may be sleeping in a cramped position. And if you arrive too early, you then may have to wait till check-in time. Or, if you in somewhere dangerous or with an extreme climate you may have to book extra half-day hotel time at each end anyway (if they allow such a thing), so you may not actually save money either.
• Never travel on a Friday. Homecoming businessmen saturate the skies on Fridays. The more people on a plane, the more chance of contact with a vector, and the less chance of space to stretch.
• Avoid North American carriers. This goes without saying. North America is seemingly populated entirely by dwarves. Asian airlines are the best for legroom: Thai, Singapore, QANTAS.
• If I do have to take an overnight flight, I take a quarter of a sleeping pill, just enough to push me in the right direction but not enough to knock me out. I find traveling without my contact lenses also helps rest, because otherwise the infloght movies are too attention-grabbing.
• At the stopover legs, don’t do anything taxing. Pick low-hanging fruit and concentrate on rest. and relaxation.
• Take extra care on West-to-East trips. It sounds odd, but it is very real. Traveling West-to-East makes you lose time: you only get a shorter sleep period. Traveling East-to-West stretches things out, so you get more time to rest. (This is the same effect as with daylight saving, of course.) If I have to do an around-the-world trip, I’d always try to go Australia-Asia-Europe-America-Australia, not the other way around.
• Sit by the window for snooze room or in the emergency row, for more legroom.

Now it is difficult to get agents to understand all these kinds of criteria. My last trip probably took about 20 phone calls to get right. Am I obsessed? Well, so should you be! If you have to take prolonged trips, be careful.

I am interested in any other tips people have.

Bob is a really clear-thinking and enthusiastic guy, and one of most interesting to wine and dine with. His book Document Engineering is important for anyone who wants a better vision of where XML is leading us. I’ve just discovered the IT Conversations website, which has podcasts of various people of interest to me: Miguel de Icaza for example.

Bob’s podcast has much of interest. An idea that hadn’t registered with me before is that one of the drivers for (larger) business to adopt a document-engineering approach is because they need to componentize their business functions: a document doesn’t care whether it goes to Florence, Bangalore or Kinshasa. Globalization as a driver for XML: that’s a pretty strong driver.

Bob also has a blog with co-author Tim McGrath Doc or Die

Update: Problem seems to be fixed. Thanks to whomever@O’Reilly did the fixing!

Update: So this is *DEFINITELY NOT* an issue with Google Reader, and instead an internal issue with the feed generation,

Atom 1.0:

Parse error: syntax error, unexpected '{', expecting ')' in /title/oreillynet/htdocs/blogs/xml/templates_c/%%0A^0A4^0A404241%%mt%3A225.php on line 6

RSS “2.0″:

Parse error: syntax error, unexpected '{', expecting ')' in /title/oreillynet/htdocs/blogs/xml/templates_c/%%0A^0A4^0A404241%%mt%3A225.php on line 6

My sincere apologies to the Google Reader team for suggesting in the title that this was a problem on their end. Quite obviously that is not the case. Title has been adjusted accordingly.

Also, I should emphasize the point that this is a problem I see *ALL TOO OFTEN* with MovableType. One *REALLY* strange issue I kept running into when working on Lawrence Lessig’s new blog was random characters (the %, &, and { characters were the most common) being saved in the template files for no obvious or apparent reason.

Any other MT users notice this as of late?

[Original Post]
via a recent tip from W^L+ (thanks, W^L+!),

Google Reader hasn’t shown any XML Blog posts since June 26th. I thought you were all on vacation.

Anyone else experiencing similar problems with Google Reader? Or is the problem (potentially) with one of the feeds generated by MovableType (the blogging software we use on O’ReillyNet Blogs)?

Thanks in advance for anyones and everyones help in tracking down the potential problem!

Update: It’s official,

Safari on Mac
Safari on Windows

Thanks again (everyone@)Apple!

[Original Post]
So a few weeks back Todd Ditchendorf brought to the surface, and I followed up with a report, the fact that Apple had added scripted transformations to the WebKit mix. At the bottom of that same follow-up report,

** Though I wonder if Safari has migrated any of the EXSLT functionality from libxslt, in particular the node-set() function? Anyone know off hand? If no, then Opera still has one leg up on Safari. Of course they still have one leg down on Safari as well. ;-)

Extending from a “request for support” from Romain Brestac, the above question then led to an open discussion in the WebKit Bugzilla interface regarding adding support for exslt:node-set(). Less than an hour later, Dave Hyatt (yes, *THAT* Dave Hyatt), followed-up with,

There is a running gag in the Simpsons where, in a flashback, Homer or someone will boldly predict something wildly wrong: 8 track tapes will never die,, that kind of thing. I was reminded of this type of gag today when reading an Slashdot thread from 2004 entitled “Is the new Microsoft Office really open?” It relates to Office 11, but it is interesting to note what rabidly anti-Microsoft people were demanding at that time, and their expectations of getting it.

Some highlights:

• In summary ‘Microsoft says it’s opening its Office desktop software by adding support for XML–a move that should help companies free up access to shared information. But there’s a catch: It has yet to disclose the underlying XML dialect.’ Could this be grounds for another anti-trust suit against Microsoft?”
• Of course it isn’t open. It’s a silly question. Open is EVIL. Actually open would eliminate advantages. People would be able to create their own tools to interact with documents, instead of with MS tools. Where’s the money in that?
• The move to XML has the potential to eliminate that sort of brain damage once and for all provided they actually open their file formats.
• But they can make it so massively complex that it is very difficult to implement interoperability with foreign tools, but that it is somehow much easier to implement with MS-centric tools. … So maybe the XML format will be like that. If you’re Linux-centric, for instance, the threshold of pain for accessing Word XML docs will be fairly high, but if you’re Microsoft-centric, with all of their tools, code-snippets, documents, etc., then it won’t be nearly as painful.
• …even if they use absurdly complex element names,…
• As long as MS is effectively a monopoly XML will be whatever they say it is, for the majority of people.
• Of course, most people won’t use the XML format at all, since it won’t be the default.
• Even if a language is in XML, you still need to *document it* to be able to *understand* it.
• Well if the way Microsoft Word saves out as HTML is anything to go by, then concise it most definitely will not be.
• I suppose they could put some weird binary or encrypted data in the files, but that would defeat the purpose of XML.

If the XML files office produce are not made the default save types or if the XML merely encapsulates large portions of binary code, it will not matter one lick that office can save these xml documents because the majority of people will be stuck on the default, unreadable formats.

• That being said, it’s hard to see what business the government has engineering document formats. They could, on the other hand, specify disclosure of formats as a remedy in an anti-trust case, but they generally fall into one of two categories which precludes this: stupid or bought.
• It’s not “obfuscated” so much as it’s “optimized.” The whole idea seems to be for Word to save as quickly as possible–which the doc file is best at for Word for some reason, probably becuase it’s derived from how the program structures documents, and not how some document spec says documents should be handled.
  In an era of 2+ GHz computers with 7200+ rpm hard drives, it seems odd that Microsoft would be unable to write an application than can quickly save and open text files that, on average, run well under 50 kilobytes.
• The big question (to me) is whether Microsoft can put a legal encumbrance on the XML schema they use for a new file format. Could you publish a schema but have it so wrapped in legalese that (for example) open source projects could not be allowed to use it ?
• So, is this going to be XML like the rest of the world knows it, or is it going to be an embrace and extend XML? Or, could it be a mutant XML? How about an XML that makes reference to Windows specific resources IDs?
• Adoption of a “standard” is no guarantee of interoperability. Understanding the conceptual underpinnings of the standard is just as important. The question is, when Microsoft says they are using XML as a document format, are they doing it because they believe in the principles underlying it, or solely for the cynical “this is what is selling now” aspect?
• The Office file formats will be open if M$ decides to:
  
  * Document them, and
  * Not change them with every update.
  
  I doubt they will do either of those things.
• They simply have no reason to play nice in an industry consortium to agree on a DTD/Schema when they have 90% market share. But as long as they publish the details of their Schema and don’t leave chunks of encoded COM schwag lying all over the place it doesn’t matter. Of course, we all know the likelihood of that happening.
• Something in my gut tells me that beyond all the extraneous tags, attributes and data types, the XML is going to have a hash code built into it.
• For an XML language to be open, you need a full description of what each possible construct in that language means.
• It would be quite easy to make the M$ document xml format propriatry. Make all default generated documents have linked in components like some ActiveX HTML pages. You might be able to read the base parts of the document but that won’t make it very userful without M$.

And so on…ratbags and survivalists: a bitter cup but with a few drops of sweeetness from such as Liam Quinn plonked in. So lets suppose the Masters of the Four-Paned Beast capitulated and gave these Knights of the Slashed Dot pretty much everything they were calling for: we would get a requirements list along the lines of:

• Open up: use XML with no extensions
• Open up: provide copious documentation
• Open up: don’t require MS-specific tools for processing; allow people to make their own tools
• Open up: don’t use long element names
• Open up: make it the default save format for Office
• Open up: use real XML, don’t just wrap binary or encrypted sections in XML tags
• Open up: don’t have legal encumbrances on use
• open up: don’t have defaults that use (or require) COM or embedded objects or anything MS platform-proprietary.
• Do have some stability to the format: don’t have a lot of updates

however

• optimization is OK
• integration with the deeper “conceptual underpinnings” of XML is important

It strikes me that this is almost exactly a description of the direction that MS then took with Open XML. The Slashdotters don’t know they won, (or at least that in 2007 they would get what they were asking for in 2004.) …Perhaps in 2010 they may get what they are demanding in 2007! “Doh!”

The more I think about it, the more that I think that the reason why Schematron (or something like it) will ultimately win (i.e. consolidate into a mainstream place as a schema language with broad vendor support) is that it fundamentally asks a different, and more important, question about documents than the grammar-based validators, such as DTDs and XSD. The question it asks is “What information does a schema need to provide to the user and application?”. And a related question “How do we group that information so that it can be sequenced to the user and application in some useful order?”, which is just as important when there is a large amount of information.

Contrast this with, say XSD, where the question gets morphed into “What canned outcomes can validation have, independent of the schema’s actual semantics and domain?” XSD is crippled in this regard because there is not enough guidance about the appinfo and documentation annotation elements: are they for human end-users, for schema management or what? They are wasted elements.

Now I guess underlying this is the idea that the central issue with building computer systems is ultimately how to relate information to humans in ways they will understand. (I want to use the dreaded word “empower” here, to my shame.) That includes developers, but developers are only the initial target group, not the only ones.

When you see a data field with text like 2007-07-05 you are probably looking at a date in ISO 8601 date format. Year, month, day: YYYY-MM-DD

IS 8601 is in an international standard which gives several standard syntaxes for representing Gregorian dates and times. The full English title is ISO 8601:2004 Data elements and interchange formats — Information interchange — Representation of dates and times. It is only about 33 pages long; you can purchase it from your local standards body or from ISO, and as is common practice for ISO standards, there are final drafts available for free on the Internet. It is maintained by ISO TC 154 who have the Dr Who-ish name of Time Task Force.

Before IS 8601 there were multiple other standards for dates and times. For example, IS 2711 which allowed formats like 5 Jan 2000 and dates using ordinals. 2711 was withdrawn as an ISO standard in 1988, superceded by 8601. However, other nations and bodies have continued to use many of the other ex-standard formats, because it is convenient to have time written according to local conventions. The difficulty, you see, with IS 8601 is that they managed to get a nice unambiguous format for dates by adopting a format that no-one non-technical used: year, month, day. (I won’t be dealing with times in this blog.)

The second rub with IS 8601 is that it defines multiple formats. So as well as YYYY-MM-DD for years, such as 2000-01-01 you can also have the basic form 20000101. And the same date could also be represented as YYYY-DDD as 2000-001 with a basic form version of 2000001, where the DDD is an ordinal counting the number of days into the year. And it could be specified as a relative date too: you could specify it as a duration (from a base of, say, 1999-01-01) using the syntax PdddD where P means period and D means days: so P365D or even P12M where M is month. You can do the same with an explicit base and get the notations 19990101/P365D for example. The more exotic you get, the more chance that you have a need for what the standard calls “a mutual agreement” where the exchanging parties agree on what the notation means, because it could have several different meanings under the standard.

The third rub with IS 8601 is that it is based on the Gregorian calendar. This is unsurprising, in view of the dominance of the West and its ex-colonies in trade and standards adoption. However it imposes a conceptual, processing and formatting burden in places where other dates formats are used. It is not as uncommon as people think: I have lived in Taiwan and Japan, where non-Gregorian calendars are used for example. And obviously the Islamic calendar is in wide use.

XML Schema Standards and Dates

XML Schemas (XSD) is a Technical Recommendation made by an industry consortium W3C, which allows direct participation by representatives of fee-paying members. XML Schemas supports a wide range of ISO 8601-ish date formats. For most formats, there is no difference, and there is even an explicit appendix ISO 8601 Date and Time Formats which gives clear information.

XSD provides many different datatypes for dates, times and durations. However, it does not allow all ISO8601 syntaxes, and it does alter others. For example, ISO 8601 allow a year 0000. This is not allowed under the AC/DC calendar system, where you go straight from 1BC to 1AD. XSD’s date types disallow the year 0. Most importantly, the date notation used is the extended one with the minuses, so 2000-01-01 not 20000101. XSD allows you to “derive types” to restrict dates to certain values or ranges.

When we were discussing date formats in the W3C XML Schemas Working Group, I tried to get localized date formats allowed: I think it is the same principle as IRIs: it is good if a human can author directly (or generate directly) in the form or notation that they use to think about the data. However, my brilliant idea was rejected by the XML Schema Working Group (with the MS representative taking quite a strong stand that neutral/standard formats should be used, not localized ones) probably because I did not have a proof of concept. Since then, Jenni Tennison’s DTTL data typing language has come along, and is being adopted as part of the ISO DSDL multi-part standard. It is, I believe, exactly the right way to go: allow notations in the format that makes most sense to the stakeholders and application requirements, but provide a mapping to neutral/fixed-syntax formats.

In that sense, my personal belief is that ISO 8601 is a relic of a pre-markup and pre-schema mentality. That does not mean it is not valuable nor that it should not be maintained, nor indeed that it shouldn’t be the first port of call when looking at date formats. But it pushes localization to be an application consideration whereas I think it is just as legitimate and feasible to make it a markup/parsing (i.e. schema) level issue. This is not only because localized formats (rigorously described with an appropriate declarative schema language) make it easier for humans to read and write, but also because where the consumers and generators of data are computers and humans are relatively unimportant in the pipeline or critical path, then data field notations localized (again, rigorously described) for optimal computer performance is entirely appropriate and smart.

Office Document Formats

ODF and Open XML both use ISO 8601 dates in the YYYY-MM-DD form throughout for all dates. (The ODF spec uses US date MM/DD/YYY formatting in palces in its text, but don’t let that confuse you.)

ODF has quite a nice, basic and consistent approach to dates in spreadsheets: read and store them in a kind of ISO 8601 format but also allow a “null date” (such as 1899-12-31) to be specified to allow conversions of date into numbers. Spreadsheets very often actually store, manipulate or transfer dates as ordinal values from an index point: this makes calculations with dates very straightforwards. Representing dates as ordinals is also used in other ISO standards: for example, the SQL_DATE data type gives the number of days since January 1, 1841. (It gives this count as a simple integer.) See section 8.5.2 Calculation settings in ISO ODF for more information.

The draft specification for ISO Open XML, from Ecma, does have one oddity, which has attracted much controversy. In SpreadsheetML table cells only, dates are actually saved as durations, as ordinals. The base is set by an attribute on the workbook, and reflects the supported ranges of Excel on different platforms (on Windows, Excel does not support dates before 1900; on the Macintosh, Excel does not support dates before 1904; putting in such a date will be serialized out as 0 into SpreadsheetML.)

The reasons for saving as as duration rather than a date are obvious: it reflects the internal format directly, allows faster loading and save times to the XML, and allows faster loading and saving times when interfacing with an SQL system that uses SQL_DATE etc. The economic value of load/store times for Office documents is enormous, and it would be quite inappropriate to apply the criteria that one might use, say, for DOCBOOK documents, to standard office formats: I actually think that ODF gets it quite wrong here, and that best practice should dictate that optimized formats should be available. However, by the same token, I think that SpreadsheetML gets it wrong, and that it also should allow reading of data in ISO 8601 format as well as in its optimized notation.

The logical question that comes up is Should SpreadsheetML use ISO 8601 duration format rather than just raw ordinal integers. If the ISO 8601 standard notation was used, SpreadsheetML should use <v>P1D</v> to mark up the first day in the range, rather than <v>1</v>. However, the P and D are redundant, because the notation is clearly marked up by attributes (and documentation). This is the old issue of where the barriers should be between information in markup and information in embedded formats. I don’t see that <v>P1D</v> has any benefits over <v>1</v> frankly: it would seem to be an exercise in nominalism and pointless compliance.

<digression>The additional difficulty here is that we are let down by XSD here, again: XSD doesn’t allow the type of an element to be selected in part or whole by an attribute value on an ancestor, unlike ISO Schematron and ISO RELAX NG. XSD is completely deficient in support for these kinds of idioms, because the database mindset of its developers let them to conceive of attributes as merely funny kinds of elements rather than as metadata on an element, of the same importance and character as the element name. So XSD doesn’t allow attributes to select type; therefore Open XML would have to compromise its design, where elements are highly generic (i.e. data values in spreadsheets are in a “v” for value element) in order to allow values to be typed; however then Open XML could declare the value to be an xsd:duration which would then require the P1D notation. Another approach in XSD would be to use xsi:type where the v element is a union of integers, durations, string etc. However, then we would need to consider how to fit shared string references into the datatyping framework. Too much work! </digression>

The second reason why the ordinal values for dates in SpreadsheetML are controversial is because of an out-by-one adjustment that is needed for some functions for the first two months in 1900. To me, this is just a silly edge case: remembering that spreadsheets from Mac Excel don’t even get back to 1900, and on Windows they don’t go before 1900: it is hardly the wholesale subversion of the Gregorian calendar that you might suspect from various comments on the Web. ODF perhaps punts the issue, by allowing date indexes to start on 1899-12-31 or on 1900-01-01 (examples they give) and so they leave it up to the application developer or document generator to figure out which one is appropriate.

In my blog last month on Principles for reviewing standards, I took the position, which I think is the most reasonable one, that for embedded data fields the standard forms should be provided and optimized forms may be provided. From that POV, Open XML should also allow ISO 8601 durations and/or dates as well as the simple duration ordinal. And ODF should allow duration ordinals as a matter of best practice,

Here’s a quick tip for the interested. When someone says “Standard A violates standard B” ask “Which clause?” No clause, no violation. (A cynic might think that if no clause from standard B is mentioned, it may suggest that standard B has not even been sighted.)

Tim Bray recently announced his publication of a new Apache module, mod_atom, which will make it possible to use the Atom Publishing Protocol (APP) directly with the Apache HTTPD server. This is a pivotal achievement, and one that will rocket APP into daily use. APP uses Atom feed content and HTTP headers to build a publishing “blog” system, though its uses extend considerably beyond the normal scope for blogging and could very well be a staple of most data publishing systems within the next few years.

You know, there was a time in the not-too-distant past where an effort to standardize something Microsoft created was seen as a *GOOD* thing,

ECMA to create standard out of Microsoft rival to PDF

July 01, 2007 (IDG News Service) Standards body ECMA International has formed a technical committee to develop a standard built on Microsoft Corp.’s XML Paper Specification (XPS), a rival file format to Adobe Systems Inc.’s Portable Document Format (PDF).

According to ECMA’s Web site, the goal of the TC46-XPS Technical Committee is to create “a formal standard for an XML-based electronic paper format and XML-based page description language which is consistent with existing implementations of the format called the XML Paper Specification.”

When and why did that change?

Well, either way, good on ya Microsoft! I’m one of your *many* supporters in regards to keeping the transparency and openness rolling forward. Please do just that.

Oh, and thanks!

The central idea of the Semantic Web is to extend the current human-readable web by encoding some of the semantics of resources in a machine-processable form. Moving beyond syntax opens the door to more advanced applications and functionality on the Web. The Semantic Web Challenge offers participants the chance to show the best of the Semantic Web.

This year, I am one of the co-Chairs of the 2007 Semantic Web Challenge. Do you have a cool application you’ve developed? Enter it! We’d love to see what you’ve been doing.

http://challenge.semanticweb.org/