Using Python and XML with Microsoft .NET My Servicesby Christopher A. Jones, coauthor of Python & XML
Microsoft piqued developer interest when it announced a new project code-named “HailStorm” in the spring of 2000. The idea of a distributed, XML-based data store that would allow your information to follow you from device to device, and from Web site to Web site, sounded attractive. The idea of Microsoft securing this data on your behalf, however, raised a few concerns. Well, Microsoft has been busily working on .NET My Services (the new name for HailStorm), and they recently released a beta version of the services at this year’s Professional Developer’s Conference.
What many developers realized after viewing the beta code, presentations, and samples is that Microsoft has indeed been focusing on the nature of these XML-based Web services and their utility, and they've created a series of services with an infrastructure fabric underlying them. In this article, you’ll learn a bit about .NET My Services, specifically the .NET Contacts service, and how you can program for it using Python and XML.
Microsoft’s .NET My Services is focused squarely at end users. The services allow you to manage all sorts of things, ranging from documents and email to address book entries and your calendar. One simple goal is to allow you to view this information conveniently from any device or platform without worry of synchronization. With an online device talking to .NET My Services, the days of waiting to cradle your PDA to get your latest email would be over.
Perhaps a more interesting goal is the ability to allow others to access specific parts of your data. For example, while making an airline reservation online, you could allow the airline to update your calendar with the flight information. If that information changed due to a weather delay while you were in a taxi to the airport, your calendar could be automatically updated from the airline’s scheduling system without you lifting a finger or making a harried cell phone call. Business associates or friends could be automatically alerted to the change of plans, instead of sitting around waiting for you to arrive at baggage claim. The scope of the services that Microsoft is developing is robust and forward-looking, and it includes support for such things as specifying the particular device you would like to receive an email or alert on at a particular time, or even tying your geo-location information into your .NET Location service.
In this article, we’ll create a request for the simple .NET Contacts service. This service, in the simplest case, is a list of all your contacts, their phone numbers, email addresses, and any other information you would like to provide. This list is analogous to your address book in your desktop email program, PDA, or cell phone. In fact, the goal of the .NET Contacts service is to centralize your contacts list so that you can easily retrieve your contacts from your phone, PDA, or desktop email (or even a trusted Web site) without having to worry about synchronization or duplication.
But perhaps the best thing about .NET My Services is you can create this request using Python and XML on Linux.
The number of tools available for working with Python and XML is growing all the time. Python’s popularity is steadily increasing, and developers are discovering its utility in working with XML. For the purposes of this article, I’ve chosen to work with Python 2.1 and the PyXML package. The PyXML package is the most comprehensive suite of XML tools for Python, and it's maintained on SourceForge.
For a list of O'Reilly's XML books, visit xml.oreilly.com, and to see our Python offerings, check out python.oreilly.com.
The PyXML package gives you a wide variety of tools for working with Python, including the Document Object Model (DOM) interface that allows for the parsing and creation of XML documents programmatically. While SOAP support is still emerging for Python, SOAP is simply XML over HTTP and can be performed using many different tools. For this article, I relied on Python’s httplib to send and receive XML packets accordingly. If you are a SOAP die-hard, it's easy enough to use one of the emerging packages or one of Python’s many bridges into COM, Java, or C/C++, and utilize a package intended for another language.
To create a client of .NET My Services with Python, you need to start with an operation or query in mind--after all, it’s a data store. Microsoft’s .NET My Services is a series of distributed services accessed via SOAP and XML. The data within .NET My Services is maintained as XML and the goal is to make this data readily available to any device or platform, mobile or otherwise. The focus of the data within .NET My Services, as noted earlier, is personal information, calendar and time management data, lists, contacts, and communications information, as well as information about the particular device you would like to receive information on at any particular time.
The client in this article was written in Python on Linux and communicated with an instance of .NET My Services beta software. The code featured in this article should also work with O’Reilly’s forthcoming .NET My Services book. Microsoft is planning significant security enhancements to .NET My Services, and the code here will likely need to be tweaked in future releases when .NET My Services begins to rely on .NET Passport for authentication.
The steps you’ll need to complete the request are straightforward:
As mentioned earlier, instead of using a SOAP tool to create and send the SOAP XML to .NET My Services, I chose to use the DOM to create the XML, and httplib for transport. The main reason is that the focus of this article is on using Python’s XML tools, and not necessarily a SOAP API.
Microsoft's .NET My Services uses XPath to target particular portions of the XML service document; various queries and inserts you perform against the services will always carry an XPath expression targeting a particular piece of the XML for focus.
The most interesting things about the SOAP packet created here are the payload and the headers. The headers contain a wealth of information (and will only contain more as .NET My Services evolves), while the payload contains a carefully constructed .NET My Services query. Microsoft's .NET My Services uses XPath to target particular portions of the XML service document; various queries and inserts you perform against the services will always carry an XPath expression targeting a particular piece of the XML for focus. In this article, we will simply extract all of the contacts within your .NET Contacts service, so the XPath is simple:
Have you used Python and XML with .NET? Share your experiences, and read what others have to say.
The ‘mc:’ is the prefix for the qName (qualified name, i.e., a namespace). The ‘contact’ is the actual contact element for each of the contacts within the service document. The ‘//’ implies that you wish to extract all occurrences of this particular element.
The XPath is wrapped within an xpQuery element, which in turn is placed within a queryRequest element:
<queryRequest xmlns='http://schemas.microsoft.com/hs/2001/10/core' xmlns:mc='http://schemas.microsoft.com/hs/2001/10/myContacts'> <xpQuery select='//mc:contact'/> </queryRequest>
In addition to queryRequest, .NET My Services has five basic commands represented as XML elements, including query (shown here), insert, replace, update, and delete.
Namespaces are used extensively within .NET My Services. Each service has an associated namespace, and the XML elements that make up the different services are sometimes reused in other services. Thus, it’s entirely possible that you use givenName and surname elements within the .NET Profile namespace within contact entries in the .NET Contacts namespace.
For a complete list of O'Reilly's .NET books, visit dotnet.oreilly.com.
The remainder of the SOAP request is standard SOAP fare, with the exception of the header. The SOAP header element is used by .NET My Services and contains routing information, user identity information, as well as application identity information. The entire SOAP packet generated by Python is shown below:
<?xml version='1.0' encoding='UTF-8'?> <Envelope xmlns='http://schemas.xmlsoap.org/soap/envelope/'> <Header> <path xmlns='http://schemas.xmlsoap.org/rp/'> <action> http://schemas.microsoft.com/hs/2001/10/core#request </action> <rev> <via/> </rev> <to>http://skweetis</to> <id>C6B9D29A-D4A9-11D5-AD8F-00B0D0E9071D</id> </path> <licenses xmlns='http://schemas.xmlsoap.org/soap/security/2000-12'> <identity xmlns='http://schemas.microsoft.com/hs/2001/10/core' mustUnderstand='1'> <kerberos>3066</kerberos> </identity> </licenses> <request xmlns='http://schemas.microsoft.com/hs/2001/10/core' document='content' service='myContacts' method='query' genResponse='always' mustUnderstand='1'> <key puid='3066' cluster='1' instance='1'/> </request> </Header> <Body> <queryRequest xmlns='http://schemas.microsoft.com/hs/2001/10/core' xmlns:mc='http://schemas.microsoft.com/hs/2001/10/myContacts'> <xpQuery select='//mc:contact'/> </queryRequest> </Body> </Envelope>
So how do you create this SOAP packet in Python? Fortunately, it’s pretty easy. There are several ways, but I chose to use the DOM support featured in PyXML. There are some imports and namespace definitions you need to create up-front, along with establishing your user identity and the target machine hosting .NET My Services. The need for a user ID will evolve with the use of .NET Passport for authentication. If you have a copy of .NET My Services beta, look in the introductory documentation to discover how to retrieve your own user ID.
""" hsclient.py A Python/XML client for .NET My Services Beta """ from xml.dom import implementation from xml.dom.ext import PrettyPrint import httplib import StringIO puid = "3066" # Your beta PUID server = "skweetis" # Server running .NET My Services # define namespaces URIs envns = "http://schemas.xmlsoap.org/soap/envelope/" routex = "http://schemas.xmlsoap.org/rp/" corehs = "http://schemas.microsoft.com/hs/2001/10/core" secss = "http://schemas.xmlsoap.org/soap/security/2000-12"
Now that you’ve established some important definitions, you can go about creating the XML with the DOM. The DOM features methods for creating both simple elements and qualified namespace elements. For .NET My Services, you’ll use namespace qualified elements. Using PyXML, you create a document object using xml.dom.implementation’s createDocument method. You can then “stack” the other envelope components onto your new document by creating elements and appending them to the document root:
# create XML DOM document doc = implementation.createDocument(None, '', None) # create soap envelope element with namespaces soapenv = doc.createElementNS(envns, "Envelope") # add soap envelope element doc.appendChild(soapenv) # create header element header = doc.createElementNS(envns, "Header") soapenv.appendChild(header) path = doc.createElementNS(routex, "path") header.appendChild(path)
After adding the path element to the header, you can go about adding the plethora of header information that was shown in the SOAP packet earlier. This includes much of the user and application identity information:
# add path elements action = doc.createElementNS(routex, "action") action.appendChild( doc.createTextNode( "http://schemas.microsoft.com/hs/2001/10/core#request")) path.appendChild(action) rev = doc.createElementNS(routex, "rev") rev.appendChild(doc.createElementNS(routex, "via")) path.appendChild(rev) to = doc.createElementNS(routex, "to") to.appendChild(doc.createTextNode("http://" + server)) path.appendChild(to) id = doc.createElementNS(routex, "id") id.appendChild( doc.createTextNode("C6B9D29A-D4A9-11D5-AD8F-00B0D0E9071D")) path.appendChild(id) # add license element licenses = doc.createElementNS(secss, "licenses") identity = doc.createElementNS(corehs, "identity") identity.setAttribute("mustUnderstand", "1") kerberos = doc.createElementNS(corehs, "kerberos") kerberos.appendChild(doc.createTextNode("3066")) identity.appendChild(kerberos) licenses.appendChild(identity) header.appendChild(licenses)
As you can see, when creating an XML document programmatically with the DOM, there is a lot of “stacking” involved. However, once completed, this way it’s easy to modify your packet dynamically and to add or remove elements and attributes. Working with a text blob of XML may seem easy at first, but when you need to start dynamically creating bits and pieces of it, you’ll find yourself juggling a mess of text strings together to try to create your XML.
The SOAP body contains the query request, and you again use the DOM to build this part of the SOAP packet:
# add request element request = doc.createElementNS(corehs, "request") request.setAttribute("mustUnderstand", "1") request.setAttribute("service", "myContacts") request.setAttribute("document", "content") request.setAttribute("method", "query") request.setAttribute("genResponse", "always") key = doc.createElementNS(corehs, "key") key.setAttribute("instance", "1") key.setAttribute("cluster", "1") key.setAttribute("puid", puid) request.appendChild(key) header.appendChild(request) # create body and queryRequest body = doc.createElementNS(envns, "Body") queryRequest = doc.createElementNS(corehs, "queryRequest") # add a namespace attribute for the qname in our XPath queryRequest.setAttribute("xmlns:mc", "http://schemas.microsoft.com/hs/2001/10/myContacts") xpQuery = doc.createElementNS(corehs, "xpQuery") xpQuery.setAttribute("select", "//mc:contact") # finish the envelope queryRequest.appendChild(xpQuery) body.appendChild(queryRequest) soapenv.appendChild(body)
At this point, your SOAP packet is finished. You could use PrettyPrint (from xml.dom.ext) to view your SOAP. In the next section, we’ll pour the SOAP into a string so that you can transport it using httplib.
I chose to use httplib to send the SOAP packet, rather than a SOAP API, mainly to give a more detailed illustration of working with the DOM in Python (instead of using packet support of the API) and to more carefully illustrate how the XML is delivered to .NET My Services.
The HTTP headers used for .NET My Services are standard, and you use a POST method to deliver the XML:
soapString = StringIO.StringIO() PrettyPrint(doc, soapString) # send request to .NET My Services http = httplib.HTTP(server) http.putrequest("POST", "/myContacts") http.putheader("User-Agent", "Simple") http.putheader("Host", server) http.putheader("Content-Length", "%d" % len(soapString.getvalue())) http.putheader("Pragma", "no-cache") http.endheaders() http.send(soapString.getvalue()) # get the HTTP response reply, message, headers = http.getreply() print "Reply: ", reply, message result = http.getfile().read() print result
The only tricky part of posting your own SOAP packet is ensuring the content length is correctly specified. Of course, when retrieving the SOAP response, you would need to use a parser to determine the success of the request and to utilize the data returned. When the above code was run against my copy of .NET My Services, I received the following SOAP (and HTTP) response back from the server:
Reply: 200 OK <?xml version='1.0'?> <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hs="http://schemas.microsoft.com/hs/2001/10/core"> <s:Header> <path xmlns="http://schemas.xmlsoap.org/rp/"> <action>http://schemas.microsoft.com/hs/2001/10/core#response</action> <rev></rev> <from>http://skweetis</from> <id>89A1D531-E2AE-11D5-B853-00065B41A690</id> <relatesTo>C6B9D29A-D4A9-11D5-AD8F-00B0D0E9071D</relatesTo> </path> <hs:response> </hs:response> </s:Header> <s:Body> <hs:queryResponse xmlns:hs="http://schemas.microsoft.com/hs/2001/10/core" xmlns:m="http://schemas.microsoft.com/hs/2001/10/myContacts" xmlns:mp="http://schemas.microsoft.com/hs/2001/10/myProfile"> <hs:xpQueryResponse status="success"> <m:contact synchronize="no" id="221C4B68-C22B-4247-ABEC-08CB40E8B434"> <m:name id="63231675-ED1D-4937-AD82-2347A9B3A6B3"> <mp:givenName xml:lang="en-us">John</mp:givenName> <mp:surname xml:lang="en-us">Doe</mp:surname> </m:name> <m:address id="BAD1BEF2-5ACB-453C-A443-9C30186F1C33"> <hs:officialAddressLine xml:lang="en-us">One Microsoft Way</hs:officialAddressLine> <hs:primaryCity xml:lang="en-us">Redmond</hs:primaryCity> <hs:subdivision xml:lang="en-us">WA</hs:subdivision> <hs:postalCode>98052</hs:postalCode> <hs:countryCode>US</hs:countryCode> </m:address> <m:emailAddress id="CEAEA6DC-DAD3-483C-9296-D288DCF5D5A0"> <mp:email>email@example.com</mp:email> </m:emailAddress> </m:contact> <m:contact synchronize="no" id="25C91687-2A98-4A93-8168-A3DC0BC8828F"> <m:name id="CCB1AC0C-E9FC-42C5-BAAF-36C81D3FF1FF"> <mp:givenName xml:lang="en-us">Jane</mp:givenName> <mp:surname xml:lang="en-us">Doe</mp:surname> </m:name> <m:address id="85835630-4570-4F9C-B871-7995498BB349"> <hs:officialAddressLine xml:lang="en-us">123 Anystreet</hs:officialAddressLine> <hs:primaryCity xml:lang="en-us">Woodinville</hs:primaryCity> <hs:subdivision xml:lang="en-us">WA</hs:subdivision> <hs:postalCode>98072</hs:postalCode> <hs:countryCode>US</hs:countryCode> </m:address> <m:emailAddress id="A7568B70-9037-4F89-9D75-FCC88DA7AE40"> <mp:email>firstname.lastname@example.org</mp:email> </m:emailAddress> </m:contact> </hs:xpQueryResponse> </hs:queryResponse> </s:Body> </s:Envelope>
As you can see, only two contact elements existed in my .NET Contacts service: one entry each for John and Jane Doe.
If you’re interested in learning more about Python and XML, O’Reilly’s upcoming Python & XML title is sure to please. The code presented here was written to work against the .NET My Services beta that was released at the Professional Developer’s Conference. Newer versions are sure to emerge periodically from Microsoft, and this code may need to be tweaked to support the new features as they become available.
Christopher A. Jones has an extensive background in Internet systems programming and XML.
O'Reilly & Associates will soon release (December 2001) Python & XML.
Sample Chapter 1, Python and XML, is available free online.
For more information, or to order the book, click here.
Return to the .NET DevCenter.
Copyright © 2009 O'Reilly Media, Inc.