What we want to do is to have a Schematron pattern that just checks a very specific thing: when the use in a document of one element requires that another element immediate follows it.

Actually, I am skipping over a stage here, because this code is quite small, fun and instructive. Which is perhaps another way of saying and the code we are skipping over (for now) is quite complex. The stage we are skipping over for now has assertions to test partial order (like Topologi’s and James Clark’s RELAX NG validator JIng’s feasible validation mode: it passes any element which could go after the current element (in its parents) not just the element that can immediately follow it. Having the test for partial order is useful for progressive validation (for example for feasible validation where we have a document that we know is incomplete, but we just want to know if it is OK as far as it goes) but more importantly it lets us divide and conquer our task.

Back to our simple case… The XML Schemas schema for this is when there is a xs:sequence element, which contains two consecutive xs:element particles, with occurrence constraints set so that the first cannot repeat while the second is required.

First here is the kind of code we will have in our Schematron schema:

   <sch:pattern id="Required_Immediate_Followers">
      <sch:title>Required Immediate Followers (Simple)

      <sch:rule context="Address/StreetOrPOBox">
         <sch:assert test="following-sibling::*[1][self::Suburb]">
		When in a "Address" element, the element "StreetOrPOBox" should be immediately followed by
		 the element "Suburb". </sch:assert>
      </sch:rule>
     ...
   </pattern>

And here is the beta XSLT code to generate it from our (expanded and munged) XML schema:

	<xsl:template name="generate-immediate-following-elements-checking-rule">

		
    	<xsl:for-each select="//xs:element
		    	[not(@maxOccurs='unbounded') and not(@maxOccurs > 1) and not(@maxOccurs=0)]
		    	[@minOccurs='unbounded' or not(@minOccurs=0)]
    			[parent::xs:sequence]
    			[following-sibling::*
    				[self::xs:element
    					[@maxOccurs='unbounded' or not(@maxOccurs=0)]
                                        [@minOccurs='unbounded' or not(@minOccurs=0)]]]">
    			 	<!--  Store the name of the parent element -->
		<xsl:variable name="parent-element-name" select="ancestor::xs:element[1]/@name"/>
		<xsl:variable name="parent-element" select="ancestor::xs:element[1]"/>
		<!--  Store the context path -->
		<xsl:variable name="path-to-parent">
			<xsl:for-each select="ancestor::xs:element"><xsl:value-of select="@name"/>/</xsl:for-each>
		</xsl:variable>

		  	<sch:rule context="{concat($path-to-parent, (@name | @ref))}">
		  		<sch:assert diagnostics="unexpected-immediate-follower">
		  			<xsl:attribute name="test">following-sibling::*[1][self::<xsl:value-of
		  			select="concat(following-sibling::*[1]/@name, following-sibling::*[1]/@ref)"/>]
                                       </xsl:attribute>
		  			When in a "<xsl:value-of select=" $parent-element-name" />" element,
                                       the element "<xsl:value-of select="concat(@ref, @name)"/>" should be
                                       immediately followed by  the element  "<xsl:value-of
                                      select="concat(following-sibling::*[1]/@name, following-sibling::*[1]/@ref)"/>".
		  		</sch:assert>
		  	</sch:rule>

    	</xsl:for-each>

   </xsl:template>

One thing to note is the variable path-to-parent: we will see this used again later. It allows us to have local declarations as deep as we need. Another thing to note is that whenever we test the XML Schemas attribute maxOccurs and minOccurs we first have to do a string test for “unbounded” (or a test using number()) because they have a union data type allowing numbers and “unbounded”.

Looking at this code I see an immediate potential flaw: in XPath 1.0 you would only need to check the maxOccurs and minOccurs attributes for numeric values: the tests would gracefully fail if “unbounded” was used in the original schema. However, XPath 2.0 will generate a type error, so we put the test for string first (the attribute value will be first tested as a string, then as a number). This relies on shortcircuiting: the success of the first test means the second test is not evaluated. But, oh dear, shortcircuiting is not guaranteed in XPath 2.0 (it is XPath 1.0 behaviour.) So I will have to make these tests into little if ... then... expressions. This is one place XLST 2.0 really gets it wrong, it should add the short-circuiting constraint because it makes life sooo much easier for programmers. I am enjoying exploring XSLT 2, but this is thing is just dumb and un-idiomatic. If it ain’t broke don’t fix it, and so on. (Having said all that, SAXON acts the way I want here, and short-circuits or at least does not freak out. Keen readers: please let me know if my understanding it wrong here!)

This simple test actually handles a lot of the required constraints in content models, and obviously it can be improved on: for example, when the first element can repeat, the assertion needs to be broadened to allow it to follow itself. Or what the second particle is another sequence, or a choice? Or what if the second particle is optional? And what if the same particle appears several times in the content model? (See my initial article on this Converting Content Models to Schematron for some ideas.)

However, it does not generate false negatives, which is what we want as we create our finer sieve.

We can improve on the diagnostics given by the rules in the previous article in this series, Progressive validation for complex content models.

Diagnosing Similar Names

One of the most common typos is simply to make a mistake in upper-case/lower-case. We can generate Schematron code to check this:

<sch:rule context="*[upper-case(local-name())=upper-case('Address')]">
         <sch:report test="true()">The unexpected element "<sch:name/>" has been used,
            which is close to an element in the schema: the element "Address".
	</sch:report>
 </sch:rule>

And here is the XSLT for generating those Schematron rules:

	<xsl:for-each select="//xs:element[@name]">
		<xsl:sort select="@name"/>
		<xsl:variable name="theLocalName" select="replace( @name, '^(.*):(.*)', '$2' )" />
		<xsl:if test="string-length( $theLocalName ) > 0">
			<sch:rule context=
                             "{concat(
                                  '*[upper-case(local-name())=upper-case(&quot;',
                                  $theLocalName,
                                  '&quot;)]')}">
				<sch:report test="true()" role="note"
                                >The unexpected element "<sch:name/>" has been used, which is close to an
				element in the schema: the element "<xsl:value-of select="@name"/>"
                                <xsl:if test="contains(@name, ':')"> in the
				{<xsl:value-of select="ancestor::xs:schema/@targetNamespace"/>} namespace</xsl:if>.
				</sch:report>
			</sch:rule>
		<xsl:if>
	</xsl:for-each>

This code actually catches two problems: have you made an upper-/lower-case typo or have you used an element with a name in the current namespace but using a different namespace.

Actually, the code as it is will generate a false positive if the same element name is used in multiple namespaces. So I will give it a role attribute of “Note” (as in Note, Caution, Warning). The role attribute lets you know what function a particular assertion plays in its rule or pattern.

These generated rules get put in the pattern that checks for typos, after the checks for defined names, but before the wildcard catch-all entry at the end: this way elements that have correct names and namespaces are dealt with before these rules, and any names that have other problems get dealt with by the default. In Schematron, a schema is made from patterns: each pattern contains rules, and each rules contains assertions (assert or report elements): every assertion in a rule is tested in the context (an XPath that may match nodes of interest from the document) provided by the rule; the rules however form a case statement, so that if some node matches one rule they won’t be tested by a subsequent rule in the same pattern.

Towards terser, more declarative schemas

It is almost axiomatic that automatically generated code is ugly and unfriendly. Look at compiler generators for example. Of course, getting consistent code that does the same thing many times is why you use a code generator like Schematron in the first place rather than writing the XSLT yourself, in many cases.

But it is certainly possible to make the code more friendly and more declarative. In Converting Schematron to XML Schemas I showed how to use abstract rules to provide extra declarative information so that there is enough information to convert back to a kind of W3C XML Schema. It doesn’t go so far, but the idea is that abstract rules (and abstract patterns, together with the role attribute) provide the abstraction for grouping assertions and representing types.

I won’t go into the code, it is trivial, but the idea is that there are quite a few rules or assertions that don’t have any dynamic content (sometimes it is handled by the diagnostic element, other times we don’t expect the rule to ever generate messages, see Expressing untested and untestable constaints in Schematron) and we can use abstract patterns to make things much more declative, readable and terse.

Here is an example, for the rules that swallow elements names that are defined in the current namespace

      <sch:rule id="DefinedElement" abstract="true">
         <sch:assert test="true()">The element name "<sch:name/>" is defined.</sch:assert>
      </sch:rule>

      <sch:rule context="Address">
         <sch:extends rule="DefinedElement"/>
      </sch:rule>

      <sch:rule context="AgeNextBirthday">
         <sch:extends rule="DefinedElement"/>
      </sch:rule>

And here is an example for detecting various kinds of text content:

	<sch:rule abstract="true" id="NoDataContent-ns1">
         <sch:assert test="string-length(normalize-space(string-join(text(), ''))) = 0"
                     diagnostics="d1">Element "<sch:name/>" should have no text content.</sch:assert>
      </sch:rule> 

      <sch:rule abstract="true" id="NoElementContent-ns1">
         <sch:assert test="count(*|processing-instruction()|comment()) = 0" diagnostics="d1
         ">Element "<sch:name/>" should be completely empty (no XML comments, PIs, or elements).</sch:assert>
      </sch:rule>

      <sch:rule abstract="true" id="NoContents-ns1">
         <sch:extends rule="NoDataContent-ns1"/>
         <sch:extends rule="NoDataContent-ns1"/>
         <sch:assert test="count(processing-instruction()|comment()) = 0" diagnostics="d1"
                >Element "<sch:name/>" should be completely empty (no XML comments, PIs).</sch:assert>
      <sch:rule>

      <sch:rule context="BestTime">
         <sch:extends rule="NoElementContent-ns1"/>
      </sch:rule>

      <sch:rule context="Gender">
         <sch:extends rule="NoDataContent-ns1"/>
      <sch:rule>

      <sch:rule context="Female">
         <sch:extends rule="NoContents-ns1"/>
      </sch:rule>

      <sch:rule context="Male">
         <sch:extends rule="NoContents-ns1"/>
      </sch:rule>

Much easier to read than having all those assertions expanded!

Karen Deane has an article in The Australian Battle on Microsoft standard push which includes some quotes from a background interview she asked me for, to give her the gossip ahead of the big MS journalist fly-in last week.

There has been a fair amount of chatter lately about defending the value of SOA projects or justifying such projects to the “C-level”. Many of these discussions will point at the business value of doing more with less and achieving IT cost reduction by reducing redundant systems and reuse of services in a SOA. Also streamlining processes in order to run the business more efficiently is a popular opinion,

As per the description in the newly created Facebook group of the same titled-name,

It’s been 3 1/4 years too many since Chris Sells hosted the last SellsCon @ the beautiful Skamania Lodge in Stevenson, WA. It’s about time we SellsGeeks band together and demand our rights to have our brains properly nourished with the type of brain nourishment that only a SellsCon can provide.

If you haven’t experienced a SellsCon before, then here’s your chance to get in on the ground floor. If you have, then you don’t need any encouragement from me: You know exactly what I mean when I state that there’s no tech conference like a SellsCon. None of this fluffy-puffy corporate sponsored mumjo-jumbo. Just the best and the brightest converging together into the same place for a couple days to make sense of all the crap that get’s blasted in your face at all the other conferences (Okay, except for OSCON (and, of course, any other O’Reilly hosted/sponsored conference ;-)).

Now, before I get myself in any (more!) trouble: This isn’t something that is absolutely, without a doubt going to happen. This is just an attempt to get enough people to come together to ensure Chris is made fully aware just how badly we want to attend/participate in another SellsCon, and what better way to do that than to stick our names on an easy-to-locate list that Chris can then look at and go: “Okay, I’m convinced.”

That said: I invited Chris to join the group. And he did. So if nothing else, at least he’s interested in the idea. :D

Are you in?

NOTE: ILMOTD == IRC Leaving Message Of The Day

As seen in the scroll back on #xslt,

MikeSmith left the chat room. (”Less talk, more pimp walk.”)

Hi all,

A video recording of a SOA Grid presentation that I did at the BeJUG Enterprise SOA Conference has just become available at
http://parleys.com/display/PARLEYS/Next-Generation+Grid+Enabled+SOA?showComments=true

Topics include -

- A new grid based service bus infrastructure concept that combines process flow, horizontally scalable service state caching, and ESB mediation.

- Fault tolerant in memory data grid

- A controversial subject that I call “Not Your MOM’Bus”

- Patterns for transparent state management of load balanced services

- Patterns for transparent fault tolerance of stateful services.

- Optimal server resource allocation that is complementary to virtualization strategies.

Dave

Push Button Paradise | Blog Archive | WebPath wants to be free (BSD licensed, specifically)

The focus of WebPath was rapid development and providing an experimental platform. There remains tons of potential work left to do on it…watch this space for continued discussion. I’d like to call out special thanks to the Yahoo! management for supporting me on this, and to Douglas Crockford for turning me on to Top Down Operator Precedence parsers. Have a look at the code. You might be pleasantly surprised at how small and simple a basic XPath 2 engine can be.

Nice! I wonder if it runs via IronPython? That would *ROCK*! And if no,

So, who’s up for some XPath hacking?

ME! :D

Actually, there are plenty of reasons why F# ((F == Functional) == True)) *ROCKS*. Here’s a few from the previously linked F# site on Microsoft Research,

Combining the efficiency, scripting, strong typing and productivity of ML with the stability, libraries, cross-language working and tools of .NET.

F# is a programming language that provides the much sought-after combination of type safety, performance and scripting, with all the advantages of running on a high-quality, well-supported modern runtime system. F# gives you a combination of

* interactive scripting like Python,

* the foundations for an interactive data visualization environment like MATLAB,

* the strong type inference and safety of ML,

* a cross-compiling compatible core shared with the popular OCaml language,

* a performance profile like that of C#,

* easy access to the entire range of powerful .NET libraries and database tools,

* a foundational simplicity with similar roots to Scheme,

* the option of a top-rate Visual Studio integration,

* the experience of a first-class team of language researchers with a track record of delivering high-quality implementations,

* the speed of native code execution on the concurrent, portable, and distributed .NET Framework.

The only language to provide a combination like this is F# (pronounced FSharp) - a scripted/functional/imperative/object-oriented programming language that is a fantastic basis for many practical scientific, engineering and web-based programming tasks.

F# is a pragmatically-oriented variant of ML that shares a core language with OCaml. F# programs run on top of the .NET Framework. Unlike other scripting languages it executes at or near the speed of C# and C++, making use of the performance that comes through strong typing. Unlike many statically-typed languages it also supports many dynamic language techniques, such as property discovery and reflection where needed. F# includes extensions for working across languages and for object-oriented programming, and it works seamlessly with other .NET programming languages and tools.

For those of you unaware, F# is now a first class MSFT language, or in other words, this is no longer a “Hey, here’s an idea. Let’s research it.”-type project and instead a true-blue MSFT product backed by mean-green MSFT money, led by some of the very best and brightest minds @ MSFT.

If you were to ask me “What’s the future language foundation of the .NET platform?” I would first state “More than likely, XSLT 2.0++.” And then when you stopped laughing and slapped me upside my head to awake me from my dream I’d say, “What the F#!? was that for?” and you’d say “F#??,” followed by “Isn’t that for programming the way God intended for people to program on the .NET platform?”, and then I’d say “Okay, you got me on that.” at which point we’d move on…

So here’s the thing: While there are *TONS* and *TONS* of reasons why F# *ROCKS* (did I mention that F# is distributed as both an MSI and a ZIP, the latter designed to make it easy for folks using Mono to take full advantage of what F# has to offer?), the biggest reason it *ROCKS* is this,

It’s been a while since I’ve written a “non-directed” blog for xml.com, so while I will be covering a few XML topics here if you’re not interested in economic systems theory, then you might as well skip this.

As I write this, it’s about eight hours before the financial markets open in New York. The markets were closed today, Monday the 21st of January, for Martin Luther King, Jr. day, which may prove to be a bad thing in the morning. Today the average market loss globally was about 7% - here in Canada, the drop translated to a 605 point loss, or about 4.75%, on the TSX. The India Sensex fell 11% in two minutes before trading was halted. I can pull out other figures, but they say much the same thing.

It’s hard to say what will happen in the morning - I’m not even going to try, though I have my suspicions. Enough fire control may have been put into place to keep the US markets from getting too badly singed (though I have NO doubt that few people at any brokerage firm in the country were allowed to stay at home today), but what you’re seeing here is something that we’ve not seen in a long time … the start of a worldwide stock crash.

I’m happy to report that Ruby on Rails not only offers a comfortable way to develop web applications, but that a little-noticed feature makes some formerly theoretical open approaches to XML much more immediately practical.

The Editor’s Disposition of Comments is quite an important document in the standards development process at ISO. After National Bodies submit their initial positions and comments on a late draft standard, the editor of the standard puts together a document to try to satisfy the various comments. Even though the Disposition of Comments document is not official, in the sense that anything in it is automatically accepted, it is usually the starting point for comment resolution, and, given that most comments are uncontroversial, is often the end-point too.

Monday 14th Jan was the self-imposed deadline for the circulation of the IDS 29500 Editor’s Disposition of comments. (The comments and disposition documents have been leaked to the web, with no tears from anyone.) Here is my rough characterization of them:

The Editor (Rex Jaeschke on behalf of ECMA TC45) has accepted the lion’s share. There is a small chunk of comments that are out of scope (typically concerning IPR or procedural comments.) There is a small chunk which the Editor has decided are issues for the maintenance phase, not the fast-track process: these are typically how comments like “ODF has feature X, why doesn’t OOXML support it?” There is another chunk of issues where the Editor disagrees with the substance of the comment, but wants to address the issue by adding clarifying or helpful text to the specification: for example, the issue of bitmasks is handled by giving examplars of how to handle them in XSD, RELAX NG, Schematron, DTLL and XSLT.. And finally, another chunk where the Editor disagrees, and gives the rationale for the disagreement. These are typically where the comments cross ECMA’s line in the sand: that no currently valid OOXML document should become invalid.

Of course, even in the comments where the Editor agrees with the comment, there may be some cases where the Ballot Resolution Meetinig next month decides to do something different from the Editor’s recommendation.

So how does it compare with the touchstone issues I isuggested in Your Country’s Comments Rated!?

The particular touchstone issues I see are that spreadsheet dates need to be able to go before 1900, that DEVMODE issues need to be worked through more, that the retirement of VML needs to be handled now, and that there needs to be a better story for MathML.

Lets see the suggested resolutions for each of them

• Spreadsheet dates to go back before 1900 (and can use ISO 8601 date format),
• DEVMODE concerns printer-dependent data which may be binary: the editor suggests some minimal changes to say “information” rather than “data structure” and to show how the system would work with some future XML-based print structure, but leaves the issue of a standard format to maintenance and justifies the need for these printer-dependent data chunks on the need to package information in legacy documents:
• VML is being withdrawn from the places it is used in the specification, which now use DrawingML (e.g. for backgrounds in WordpocessingML); (furthermore, this provides a level of modularity that theoretically allows some kind of use of SVG for drawing, though I don’t expect this would be a popular option unless Office supports it.)
• For Maths, the Editor recommends allowing alternative formats in particular recommending MathML: this is not to replace the OOXML Maths, but in the context of “rehydration” which is where you want to round-trip through systems that don’t support your full language, so they use some lesser one (such as a graphic) as a fallback, but the systems maintain the text of a higher-level format. This is probably good for MathML adoption, but also for professional maths systems developers.

Finally, what about that issue I have been tracking that I think is a crazy edge-case blown out of proportion: the AutospaceLikeWord95? Well, now we have a few pages of documentation about a tiddly bit of extra space between digits and full-width characters (as used by Japanese); in fact we have much more complete documentation of typesetting behaviour that should not be implemented compared to what should be implemented! It doesn’t do any harm to have this documented, except that it is a distraction from more substantive issues and has notoriously been used as evidence that DIS29500 cannot be implemented.

… and then what will /. have to write about? *NOTHING*, I tell you! *NOTHING*!!!

Wait, that would be a good thing, huh?! *SWEET*! Please carry on…

Brian Jones: Open XML Formats : Mapping documents in the binary format (.doc; .xls; .ppt) to the Open XML format

I wanted to call everyone’s attention to a few interesting developments in Ecma’s proposed disposition document related to the Office binary formats. There were a few comments from national bodies that asked about the documentation of the Office binary formats and the availability of those documents. We had already been talking about these issues in TC45 where there were a number of existing experts in the binary formats (including Apple, Novell, and Microsoft). Based on the feedback from the national bodies, Microsoft decided last week to take some additional steps in this area.

Brian goes on to describe the fact that MSFT will be making it even easier to gain access to the documentation for the legacy Office binary formats, promises not to sue you if — you know — you actually read the documentation and then apply this knowledge to implementing support for these formats, and then takes it one step further by not only providing a mapping from the binary formats to Office Open XML, but promises to start a Binary Format-to-ISO/IEC JTC 1 DIS 29500 Translator Project on SourceForge.net in collaboration with ISV’s.

Hey Micah (Dubinko),

Why don’t they just ISO standardize their binary formats? That’s “backwards compatibility” for ya. -m

Micah Dubinko | January 24, 2007 10:13 AM

Okay, so maybe this isn’t standardization, but as it relates to legacy file formats, isn’t this good enough? And either way, nice work, MSFT!

A friend of mine here in the Salt Lake City area is in need of the best and brightest C# and Flex developers who either live in the Salt Lake City area or who are willing to relocate. From what I understand these are full time positions. If you have the proper skillset and have interest please contact me directly and I will forward your contact info on appropriately.

Thanks!

It just makes them more burnt…

Java and doneness

Just because a standards body can’t recogonize done-ness and declare victory, that doesn’t mean the fruits of that body have any bearing on reality.

Over the last couple of years, I’ve worked extensively with Firefox, and while it still has its warts (and while I believe that its days of double digit rises in adoption are probably coming to a close) overall, I’ve found that it has become, for me anyway, my de facto browser into the web and the focus of most of the web applications (and extensions) that I’ve built in the last year. For that reason alone, if nothing else, I’ve been watching closely as Firefox 3.0 approaches its final release.

The second beta version of FF3 is now out, and I have to say that overall I’m feeling quite pleased with what I’m seeing, with a few caveats. Since I do generally dig into the application daily, my focus in trying it out (and in writing this review) is less on the immediate UI and functionality changes for the typical user and more on how its going to affect web software development. Thus, I ask that you forgive me for not talking about the new theme (okay, though not a radical departure from the old) or other user improvements or give you a lot of screenshots … I want to look a little more deeply under the hood.

I don’t agree with everything Sean McGrath writes in his latest post as I think there are a lot of really smart people who have developed some really smart ways to handle the variable width nature of XML w/o turning to malloc() every time the length of an element or attribute name reaches past any given preset constraints. That said, I can’t help but agree with,

Memory-based caches of “cooked” data structures are your friend.

Absolutely!

For you .NET developers here’s a pre-written recipe that handles all of the dirty work of determining whether to create a new XmlReader or return the in-memory cached version based on the generated ETag for the source file (see Extended Overview below for a deeper understanding of how this works.) To use this recipe you need to do nothing more than create a new XmlServiceOperationManager when your application starts up like so,

XmlServiceOperationManager myXmlServiceOperationManager =  new XmlServiceOperationManager(new Dictionary<int, XmlReader>());

and then use the GetXmlReader method of the XmlServiceOperationManager, passing in the Uri (an actual System.Uri object, not the string value of the URI, though I guess it would be easy enough to create an overload that takes the string value of the URI. Another task for another day. ;-)) of the desired XML file to get an XmlReader in return like so,

XmlReader reader = myXmlServiceOperationManager.GetXmlReader(requestUri);

That’s it! Now you can use your “new” XmlReader however you might need and the next time that file is requested for processing if it hasn’t changed you save all of the time it would normally take to read the source file and convert it into an XmlReader which is fairly significant.

Source code and extended explanation inline below. Enjoy!

Oh, and stay tuned for the next installment of this recipe where we learn how adding,

1 Part memcached
1 Part ETag's

and

1 Part GZip encoding

… can turn your lame a$$ performance sucking web application into a lean, mean, kick a$$ performing machine. For a precursor, see Joe Gregorio’s AtomPub presentation slides from this past OSCON