I did my prognostication schtick at the beginning of the year, but I thought given the resuscitation aspect of this particular blog, that I would go back and look at things in a little more depth, as I’m seeing a number of interesting, and in some cases disturbing trends that are unfolding right now, things that will likely have a fairly significant impact upon the XML development community.
The first issue is just the fact that there IS an XML development community. I’m fielding a fairly significant number of inquiries right now concerning my availability from recruiters nationally, though unfortunately at this point family issues and a fairly full plate are keeping me personally pretty close to home here in Victoria. Nonetheless, what I see is the rise of core XML technologies - XSLT being the biggest by far, but even XQuery and XForms are beginning to show up - in the job listings. There’s a rising demand for XSLT developers right, and I suspect that as people begin to start incorporating XSLT 2.0 that this will increase still further (and should - XSLT 2 is just … better). XHTML is becoming “standard”, at least at the corporate level, and this in turn is feeding back into the rise of the manipulation tools that XHTML opens up. I’ve even seen a rise in demand for ontologists (now THERE’S an obscure field) and RDF specialists, as people begin to harness the ability to create metadata structures. It should be interesting to see what happens as RDFa begins to become more regularly incorporated into XHTML generation tools.
What this means is a little uncertain, though as a guess I think it signals a major transitional shift towards the declarative web architecture. Certain aspects of it (such as XForms) are still a couple of years out from wide adoption, but they are gaining adherents and customers nonetheless. I think Firefox has had a lot to do with this, but there are other factors, the biggest just being the normal open standards development cycle, which seems to take longer to unfold than commercial development but that nonetheless tends to become far more pervasive when it finally does penetrate.
At the same time, I see the rise of JSON, E4X, Linq and other “XML” languages as a fairly stinging rebuke to the DOM oriented focus of the W3C. I do not think that JSON is going to “replace” XML; what I do see though is perhaps the dawning realization that the XML Infoset does not in fact have to be represented in angle-bracket notation, and manipulation of XML does not and should not require getElementByTagNameNS() or other such absurdly long method invocations. If you mentally replace the outer braces of a JSON expression {} with
We’re seeing such “compactification” going on in other arenas of XML - take a look at RNG compact notation (which bears a rather disturbing similarity to JSON, truth be told) to verify this, and compact notations of RDF are gaining considerable traction over the fairly unwieldy FULL XML notation (I expect RDFa to continue to push this principle).
E4X on the other hand works the other way in establishing the relationship between the JSON dot notation and the representation of subordinate XML elements, to the extent that such a long term convergence between XML and JavaScript is not only likely but pretty much inevitable. Declarative architectures are useful, but they are usually not sufficient; you need to have enough glue code in place - either explicitly with embedded scripting or increasingly implicit with behavioral XML bindings - in order to make the BEness DO something.
XML Binding languages will be the next arena of development (and contention) - the ability to assign a behavior to an XML tag is profoundly useful, and provides both the bones of the declarative structures and the muscles of the imperative one, while keeping the presentation layer safely off to one side. It’s not yet significantly utilized because browser support at this stage is still inconsistent - I expect that with the movement of XBL2 into the W3C this will likely change, though this change is unlikely to make its way into Internet Explorer without a fight. Time frame for this is about eighteen to twenty four months - far enough ahead that you won’t yet get a job writing bindings (though this is, in fact, exactly what all those AJAX developers are currently doing, even if they don’t necessarily recognize that fact) but close enough that if you are looking to make your mark in software world this is the place to be studying very, very carefully.
XQuery is out. I’m not sure how I feel about this. I’ve written two XQuery books over the years, both considerably prior to the actual release, and it is still a language that I have very mixed feelings about. The eXist database very nicely illustrates what you can do, however, if you mix XQuery with server based APIs and tie in XSLT 2 via Saxon 8.9. That is a potent mix, because you are essentially “inside” the XML at that point - there’s remarkably little impedance where you have to deal with XML processors or parsers or serializing in or out. I think we’re just beginning to understand what this means in terms of server-side development, but I suspect it will transform the way we think about deploying server code, especially once you start working with XHTML+XForms (which is so incredibly well-suited for XQuery).
Speaking of XSLT 2.0, Microsoft has greenlighted a version of XSLT 2.0 to be deployed with the next version of Visual Studio. My suspicion here is that if Red Hat hasn’t also raised the possibility of pushing libXSLT to the 2.0 level then it should do so - XSLT2 leaves XSLT1 in the dust in terms of both usability and performance, and I suspect that we’ll see more commercial level XSLT2 transformations appear over the course of the next year. The complexity, verbosity and need to resort to recursive programming has signficantly limited XSLT 1.0 adoption over the years … by largely eliminating the need for unnecessarily recursion and cutting down the amount of code by a factor of three or more to accomplish most transformations, XSLT 2.0 saves development cycles, more readily integrates with third party extensions, and can handle a lot of the more esoteric “routing” functions that currently are the province of large applications such as Microsoft’s BizTalk server.
For all that, the role of the W3C is shifting away from being the developer of forward looking specifications towards being the shepherd for smoothing the rough edges of existing “ad-hoc” specifications. SVG and XSL-FO both represent the move up into the application space and for the most part have been less than totally successful - for instance, XSL-FO at this point seems to be more a mid-point for transformation to PDF than a fully realized and widely adopted standard, and the recent debates on the Open Document Format vs. Microsoft’s OOXML formats showcase the fact that document specifications form more of a spectrum than a definitive single standard, because the requirements of what we do with those documents vary so dramatically. I think the jury is still out on SVG - it’s gaining some penetration into both the browser and mobile space, but interest in SVG waxes and wanes depending upon who is incorporating it at the time.
To me, the interesting things being done in XML are increasingly occurring in the application and vertical markets: HL7 in the health care field, GML in the mapping and geographical location space, XBRL or UBL in the business space and so forth represent sophisticated ontologies rather than formal schemas, the understanding being that getting complete agreement on the structure of any field of endeavor may be doomed to failure between differing participants, but you can (for the most part) agree upon the general terminology and language for those core concepts. This approach actually makes a great deal of sense - a dictionary is a self-descriptive ontology, but the province of dissecting the meaning of Tolstoy’s War and Peace should belong more readily to lit majors rather than linguists. We’re beginning to realize that meaning is a remarkably fuzzy concept, almost quantum mechanical when peered at too closely, and as such should not be all that surprised to find that there are few universal absolutes when dealing with language, whether natural or rationalistic.
Similarly, I suspect that while RDFa may have a fairly major hill to climb in terms of adoption, it will likely end up becoming integral to the semantic web fairly soon. Folk ontologies (or folksonomies, as some have referred to them) are not in fact really ontologies at all - they are instead simply property associations. If you can articulate a consisten property relationship using attributes outside of the normal XHTML ones, then you can do more than simply tag a document - you can in fact create relationships between entities in an XHTML document without having to leave the context of that document. That’s what RDFa does. These can then be interpreted by RDF enabled tools, making it possible to achieve something of the holy grail of the semantic web - provide a simple way of nonetheless encoding metadata into a document. I’ve argued for years that RDF as it exists right now is too complex for your average web developer, and what’s more it perforce requires duplication of content between the RDF and XHTML (or whatever document format you’re using). Eliminate this need for duplication by embedded the descriptive relational characteristics directly in the element’s attribute set, and all of a sudden the Semantic Web begins to move away from being unachievable to being doable.
I’d ask for your feedback on this as well - what areas do you see XML technologies expanding into, and where do you see it fading?
Kurt Cagle is an author and web technologist specializing in XML-based technologies. He writes understandingXML.com and is the webmaster of XForms.org. He lives in Victoria, British Columbia with his wife and two daughters, and is coming to terms (barely) with having a teenager in the house.
"I do not think that JSON is going to "replace" XML; what I do see though is perhaps the dawning realization that the XML Infoset does not in fact have to be represented in angle-bracket notation". I very strongly agree with that. 'XML' will come to mean the Infoset (or the XQuery data model, or tree views and XPath-like axes over object graphs) more than the bits on the wire format. That liberates XML tools to support JSON, various binary XML formats, HTML tag soup, etc. without insisting that everyone play by the XML syntax rules.
Michael,
I'm always happy when we manage to find points of commonality :-)
I think for me the realization on JSON occurred when I heard people talking about creating JSON schemas and JSON transformations, and watching the fairly convoluted discussions going on over at xml-dev about whether JSON was in fact "superior to" or "inferior to" XML. I also remember a few years back working with a language called Curl that was intended to provide an alternative client representation to XML + JavaScript, and the irony didn't escape me that the Curl notation was in the mainin fact XML represented as Scheme or Lisp.
I hadn't thought of the XQuery data model as another XML infoset, but of course you're right, it is. Makes me think that an XSLT2 transform could take in JSON via the unparsed-text method (or passed as a parameter) then run through templated regexes to generate the appropriate XML to run through a pipeline ... hmmm.
Thanks for the comment, though.
You have mentioned eXist in numerous posts, so I finally downloaded it, created a collection of documents (business expense reports), made an XForm ala Firefox to feed information into the collection via PHP, and then fooled around with PHP to XQuery the collection and generate reports.
Wow!
For any application that is document-centric this is simply smooth. I've been playing with XForms for about a year and eXist was the missing piece to bring everything together. Now just for SVG trend graphs in those reports!
And that brings me to the future of XML...walking around at the office I notice people use their computers in two basic ways: they are accessing the all-powerful central office application (a database backed application) that controls the core production and financial aspects of the company, or they are creating Excel spreadsheets to try to manage data that is not easily handled by the centralized software. I have seen some spreadsheets that are so loaded with programming, macros and cross-references they cause my eyebrows to receed right back past my hairline.
Its the same story everywhere I go... the users prefer the spreadsheets because they like to work with documents, they understand documents. They don't have to be connected to the server to work with a document. They can fill out a spreadsheet expense report on the airplane, for example. However, its not long before they realize the limitations of their spreadsheets - they don't scale, its hard to share the info inside them, its difficult to group multiple documents together and do analysis on them. Those are the jobs that the big corporate software does well. But, again, the corporate software system is annoying because it lacks all the benefits of documents.
The users want documents but the business applications want groups of documents. Both want to access the data in the documents, but they want to work with it in different ways. For so long it seems we have been made to choose between one or the other. For the past six years I have seen XML as the solution to this problem, but just now we are getting the tools to bring the possiblities to life: an XForm lets me, the human, manipulate the data in the XML document the way I would expect to do so, and an XML database such as eXist lets the central business application work with the data in the XML document in the way it needs to - in the context of a group of similar documents.
For example, if I have the company expense report XForm on my computer, I can be sitting on the airplane, open the form in firefox, fill it out, and save it to may local disk, as a single file. Then I can open it again when I get back to the office, make a few changes, then submit a copy to the centralized buisiness application. I get to keep a copy on my computer, as a file for easy reference in the future and the head office gets all the data they want, the way they want it.
From what I understand, this is essentially what Microsoft has done with InfoPath, and I can see why. What better way is there to bridge the spreadsheet--central database divide? XForms and XQuery/XML databases are the core, and I see SVG and XSLT and XSL-FO as the supporting tools used server-side to generate reports once the documents are in the XML database. Its funny, it seems that these supporting tools were created before the core pieces of the puzzle, but maybe its like my grandma used to teach me, "when you are starting a puzzle, always look for the edge pieces first and then work inward."
Anyway, if/when XForms and XQuery start gaining more traction, I can see a lot of these "supporting" XML technologies coming along for the ride.
Yeah, eXist is proving to be one of my favorite "best-kept-secrets". It's actually made me realize what the XQuery paradigm CAN do, and the irony is that querying is in fact only a fairly small portion of it.
Your comments about the office users is also well taken. I haven't been as heavily involved in spreadsheets as I have with "word-processing" document formats, but I know that a significant amount of the documents I personally have dealt with are contracts. Typically the workflow of a contract looks something like this:
Get sent a Word or PDF document - if I'm lucky they've put in form fields, but most of these generally are read-only.
Print the document out.
Fill out the form with a ball-point pen.
Fax the contract, or worse, send the contract in a Fed-ex package.
Now, at home (and my home office) I do not have a LAN line - we just have cell phones. No LAN line means no fax. Moreover, I normally have to wait until I can get home to print the damn contract out, fill it out, and scan it back in with a scanner, no doubt losing a certain degree of fidelity and wasting a good hour or more of my time on mechanics; this can be a real pain since I'm typically also trying to cook and get kids to bed in the evening. I've lost a few contracts this way simply because the workflow was such a pain that I never got the contracts submitted.
Now, combine something like XHTML+XForms and an eXist XQuery database, and the generation of the contracts becomes reasonably simple while still making it easy to fill the contract fields online. Security can easily be maintained via SSH to provide a fairly high degree of authentication (certainly more than you can get with a pen and paper and a fax machine ...) and you can perform pre-validation to insure that the information entered is complete and consistent, and the final "signed" contract can be emailed back as a secure PDF (through XSL-FO) for my files. I don't use any paper, don't need to have a fax machine or a phone line, don't need to worry about getting contracts lost in the mail or endless rounds of corrections because someone messed up or dropped out or were added to the contract at hand. The interfaces don't require that I pony up $500 for the latest Adobe Acrobat or MS Office, and the whole transaction can be completed in the time it takes me to fill in a few form fields.
So, yes, I have a personal motive in seeing XForms succeed - I don't feel I should have to pay for a fax machine or specialized software just to sign a contract.
3)
@mawrya - I also experienced the "wow" factor with eXist and xquery a while back, in a situation where a conventional database would be too big, and conventional documents too small for the demands of the project, amd I wholeheartedly agree with your comments.
With IBMForms pushing it along, I think XForms should gain a great deal of traction at the enterprise level, given enough time, but it seems that, as a scaling down of the active server model, xquery via eXist has so much to offer... to the long tail immediately, perhaps, where an enterprise solution is too grandiose. This may in turn lead to widespread adoption at the corporate level, as folks become more familiar with the idea of a database collection. Maybe if google adopted eXist into its ever expanding suite of software-as-service applications?
That would be something.
Interesting overview, but I dearly missed an analysis of the state of the art in the Semantic Web domain. Maybe in a next issue?
I'm still trying to get a sense of where the Semantic Web itself is going. There's a lot of very interesting stuff occurring around Sparql, RDFa has some interesting potential, and I see signs that OWL is finally penetrating Business Intelligence systems, but I'd like to hold off on covering that until I can get a chance to research it in greater detail.
RE: JSON is just curly XML.
I guess we should have kept the SGML Declaration file too. :-)
Len,
I need to clarify my position here. I do not think that the Infoset by itself will be the only factor in determining the nature of XML, and in reviewing my own comments it occurs to me that I need to be more specific concerning three factors - the general use of HTML as an infoset, the role of JSON, and "alternative" forms.
XML by itself consists of two distinct concepts - the containment model infoset with the ability to define attributes (in the larger programmatic sense) upon containers or leaves, and the specific pointy angle bracket notation that acts as the serialization of the first.
There is also a third notion at work, though one that's not always made explicit, the notion that containment should not be inferred. XML and JSON both have explicit containment models. HTML (and SGML in general) assume that containment can be implicit.
Unfortunately, once you start down that slope, things get slippery quickly - validating the well-formedness of XML is easy, in great part because you can insure that a structure follows an explicit containment model without having to know an underlying semantic. HTML validation requires that you do know the schema, because only specific elements can be "improperly" terminated (such as the <li> element).
JSON lacks a number of fairly critical components to make it the full equivalent to XML - namespaces, attributes, comments and PIs, just to name a few. However, while XML cannot always be mapped to JSON without introducing some form of conventional semantic, JSON can always be mapped to XML, with the one convention that there be an explicit name for the anonymous overall container of a JSON object.
My recommendation on that front is that the W3C should recognize this fact and define more clearly what in fact constitutes the conventions for valid mapping of a JSON-like language to XML and vice versa - a compact notation, if you will. If they don't do it now then a given market leader will do it, and the W3C will have to address the issue later with considerably less control over the final product.
This is the same issue that is facing the W3C with regard to binary XML ... the realization that there are formats that will end up needing to be supported that don't fit into the convenient happenstance of TBL's original HTML notation. I think rather than spin their wheels wrt "clarifying" XML with an XML 1.1 spec that no one is taking seriously, it would be worth the W3C's efforts to understand that the map is not the territory here.
Good article, but can you make it more readable?
>>JSON lacks a number of fairly critical components to make it the full equivalent to XML - namespaces, attributes, comments and PIs, just to name a few.
>>
But JSON was not designed to be a general "markup" language, but a language for exchanging data structures.
I'm not claiming (nor do I think any but the most die hard AJAX fanatic) that JSON is a replacement for XML in all circumstances - as you point out, a point I agree with, JSON is a way of encoding data structures. Even given this, it has some limitations - you cannot encode multiple items to the same hash key:
var obj = {a:"first",b:"second",a:"third"}
implies that obj.a = "third"
while
var obj = <root><a>first</a><b>second</b><a>third</a></root>
implies that obj.a is a nodelist of two <a/> nodes - and in fact there is no clean way that you can in fact render that structure in JSON without losing document order or relying upon some additional constraint semantics.
That's why f:json(obj)=>xml:obj is generally feasible, while f:xml(obj) => json:obj is valid only for a subset of highly constrained schemas.
So where is XML going? In search of its parent?
I enjoy watching evermore complicated explanations given to justify the centrality of a language that was awarded its central status by its claims to simplifying the complications of its progenitor.
Very karmic.
I feel fortunate to have been around the track enough times to understand a few simple things:
1. No language is fit for all applications.
2. Any syntax can be made to fit all applications with enough exceptional language in the ammendments to that '27 page spec' thrown so triumphantly to the floor.
3. Verbosity does matter.
4. Syntax barely matters but without it, you have to create an abstract object model that is more a ball and chain than an SGML Declaration ever was.
As the twig is bent, Kurt.
len
I am very interested in hearing about the XML SVG Standard (Scalable Vector Graphics). If you look at the potential for this market and the quality of the web playback the opportunity for SVG is really fantastic. We need an open web standard and not a closed standard like Flash or Microsoft. I also have been playing with a neat tool called SnapKast. The Snapkast software converts a powerpoint to the SVG standard and then delivers the content as a mpeg4 file or podcast. I hope we can start to see better animation from this standard. The possibilities are really great but we need more companies like SnapKast taking the standards to video applications.
I think your points are true of most languages. To me, the argument tends to come down to which domains is XML appropriate for, and which domains its not. We know that its not a terribly efficient language for representing highly indexed data, even though the serialization of that data in XML works reasonably well. It's a hideously inefficient imperative language, though as a wrapper to such languages it provides a nice metadata layer. It's not the most ideal language for expressing relational mappings by hand (I personally hate RDF+XML for its complexity) but it turns out that machine encodings of RDF as XML make a great deal of sense. It's a fairly inefficient mechanism for encoding data structures, but it fulfills the minimal requirements that you can express most relationships with it (you can only have one hash tag active at a given level in a JSON document, for instance, so you cannot express non-adjacent tag containers in JSON).
XML is, by definition, a compromise, and always has been. Like most such compromises, this means that there are almost inevitably better specialist tools to perform the same solution. In many cases, the benefits of those tools will almost certainly outweigh the benefits that the more generalist XML approach takes, but in other cases it may not.
I think a lot of people look back to SGML with fondness, like it represented a nirvana standard and the move to XML represents some kind of fall from grace. I came to SGML fairly late in the process, and for me, it was a complex, unwieldy, difficult language to work with - yes, it managed to embody a great number of concepts that have been "reinvented" under XML, but overall I think the reason for creating XML (warts and all) from SGML remain as sound today as they did ten years ago. Personally, I suspect that if SGML was in fact such a perfect language, then you'd be seeing a significant uptick in its usage, and I've seen no such thing. I had one commentator on another blog say the same thing about LISP's superiority over XML (which I'll readily grant), but notice again that LISP remains very much a niche market.
XML is hideous for some applications, marginally acceptable for more, and "good enough for government work" for a whole lot of others. I suspect that combination of necessary and sufficient is what keeps the XML engine going.
I never said it was nirvana and my thinking isn't that fuzzy. I look at the human reasons for technologies and find the overlap often quite contradictory. Yet much is made of the separation of validation, well-formedness, death to DTDs, yadda yadda yadda, Kurt and then I see this:
"Valid markup has become equated with two things nobody wants: impracticality and implausibility."
quoted at alistapart where an author who obviously starts his career with the web and continually has to justify the costs of working with broken code:
PRECISELY. And it only took the wisdom of the crowds here a decade to catch up to 1993 on that point.
It's broken, Kurt. The people who broke it keep defending that position with a lot of self-covering justification based on technical reasons that given the human uses are becoming highly suspicious because technical means did exist, were abandoned, are being replaced with even more complex means, and all of this in the name of 'simplicity'.
Sorry, fella, but at some point you have to ask yourself where we went wrong and what we should learn from that. I don't expect Bosak or Bray to ever do that because they can't, but the rest of us might want to. Given that quote, something has obviously gone very wrong. If we can't learn from that, then our blogs will only get longer because it takes a very long coat to cover asses hanging out that far.
So where is the solution? Go back to a pure SGML, where we end up replacing flawed but largely interoperable XML with flawed and largely non-interoperable SGML syntaxes? Recognize the validity of HTML as a "human-readible" format when most content nowadays is being generated by machines for machine consumption? Resuscitate CORBA from the dead?
It's easy to criticize the W3C - I've got enough personal reasons to fill the walls of my house in microtype - but overall I see the reason for the problems that occur on the XML well-formedness side (let's ignore validity for a moment) really come down to questions of poor programming practices, venality and ignorance ... and no standard, no matter how well crafted, is going to be able to overcome those three factors. Yes, JSON is simple, but its also woefully inadequate for representing most document structures. If you want to see confusion, try getting the average JavaScript monkey to learn LISP, even though its what XML (and JavaScript, for that matter) keeps trying to be. We can ignore the declarative side completely, stay only on the imperative fence, and we end up performing a lobotomy on the gestalt programming brain.
It's easy to criticize XML, but its a lot harder to see what's out there that has the potential to replace it. I see hopeful signs that this something IS emerging - I think that XML + JavaScript + XMLHttpRequest together make up a fairly potent mix, especially as we begin to discard a lot of the things that have been baggage - the use of the DOM as the only means of manipulating XML objects, the appearance of a more coherent XSLT, the rise of objectified XML objects. Already, DTDs and PIs are receding from usage, entities are frankly disappearing except when absolutely necessary. The language that's appearing out of the XML duck's nest may be more of a swan (to the bewilderment of the ducks) but I personally see that as a good sign.
As the first comment in this thread pointed out, ultimately the question becomes "What is the Infoset?". Well-formedness is another way of saying "unambiguous", not necessarily a measure of syntactical correctness. HTML 4.0 is ambiguous because it makes significant assumptions about completeness and containment that, I believe, require some semantics to leak into the syntactical model, which is bad design. Ideally you want to be able to avoid instantiating the infoset in order to determine its viability.
Thus, I need to know what you're questioning here as being flawed? If it is the syntax, then yes, frankly I'd agree with you, as I think many others do. To me, the best way to push that is to raise the question of what exactly we mean by Infoset (I'll likely do so shortly in a blog anyway) and try to push that envelope as much as possible. If you're arguing that the Infoset model itself is flawed, I'd have to disagree there. For what it needs to do, the infoset is pretty nicely balanced between flexibility and efficiency. Perhaps I'm being dense here, but I just cannot see the benefit that scrapping the infoset would have, compared to the gain that its already managed to bring.
Any ideas on this?
On the subject of SGML, JSON, etc, I think we should recognize that a standard can be a success merely by being ready at the time it is needed and bowing out after it has allowed technology to leap to its next stable state. ISO standards recognize this by having a five year review process; every five years, every living standard has to be verified that it is still in operation, otherwise it gets taken off the list.
We can get too caught up with the permanence of standards, as if they were holy scripture. Consider SGML: took years to develop and for Charles and everyone to gather the requirements, harmonize them, and get the standard out, then after 10 years it is superceded by something much simpler, and after 15 years people start doing the things that it originally standardized though without the benefit of an automatic mapping to SGML. It is frustrating, and dumb or ignorant in another sense; but it is perfectly OK, because the reason for a standard is to meet a market requirement at that time, not to be the answer for all times.