dunder-xml-reader 0.1.1

Pythonic XML parsing/reading

dunder_xml_reader

Description/Overview

The dunder_xml_reader package provides classes and functions that aim to make navigating and extracting data from an XML document easier and more "Pythonic". All you have to do is pass a string containing valid XML to the parse_xml() function and it will return a Python object that exposes properties as dictionary keys and child-nodes as attributes. The following sample code illustrates. Consider the following XML:

<cXML payloadID="[e-mail]" xml:lang="en-CA" timestamp="2000-10-12T18:41:29-08:00">
    <Header>
        <To>
            <Credential domain="AribaNetworkUserId" type="marketplace">
                <Identity>[e-mail]</Identity>
            </Credential>
            <Credential domain="AribaNetworkUserId">
                <Identity>[e-mail]</Identity>
            </Credential>
        </To>
    </Header>
</cXML>

It can be easily nagivated with the following Python code:

>>> with open('order_request.xml') as infile:
...     xml = dunder_xml_reader.parse_xml(infile.read())
>>>
>>> print(xml['payloadID'])
[e-mail]
>>>
>>> print(xml.Header.To.Credential[0].Identity.text())
[e-mail]
>>>

As you can see, the property payloadID is accessed as if it were a dictionary key of the xml object. Likewise, the Header child node of the cXML document is accessed as if it were an attribute of the xml object.

The Python object(s) returned by parse_xml() use Python dunder methods to fulfill functionality provided by other Python built-ins:

>>> # Python's "in" operator
>>>
>>> 'payloadID' in xml
True
>>> 'someOtherThing' in xml
False
>>>

>>> # Python's "hasattr" function
>>>
>>> hasattr(xml, 'Header')
True
>> hasattr(xml, 'Footer')
False
>>>

>>> # Python's "len" function
>>>
>>> len(xml.Header.To.Credential)
2
>>>

The Python objects returns by parse_xml() have the following methods:

• tag() - Returns the XML element tag (i.e. "<cXML>" returns 'cXML')
• text() - Returns the text between the tag and it's closer (i.e. "<p>blah</p>" returns 'blah')

Nodes that have siblings with the same element tag (i.e. Credential in the above example) are represented as a XmlNodeList, a subclass of Python's built-in list. The XmlNodeList class has a number of extra convenience methods:

• first() - Returns the first item in the list, or if the list is empty, a default value.
• last() - Returns the last item in the list, or if the list is empty, a default value.
• filter() - Filters the list items given a lambda expression.
• filter_prop() - Filter's the list items looking for items that have a property with a given value.
• filter_text() - Filter's the list items looking for items that have text with the given value.
• map() - Transforms each item in the list to another given a lambda expression.
• map_attr() - Transforms each item in the list to the value of the item's given attribute.
• map_prop() - Transforms each item in the list to the value of the item's given property.
• map_text() - Transforms each item in the list to the value of the item's text().
• join_text() - Returns string created by concatenating the text() values from each item.
• join_prop() - Returns a string craeted by concatenating the given property value from each item.

Installation

The easiest way to install dunder_xml_reader is to use pip:

$ pip install dunder-xml-reader

Some Examples

The following are some examples of how you can use the Python objects returns by parse_xml():

Getting property values by dereferencing as a dictionary

>>> xml = parse_xml(raw_xml_text)
>>> payload_id = xml['payloadID']
>>> print(payload_id)
[e-mail]
>>>

You can even check to see if a property exists using Python's built-in in operator

>>> xml = parse_xml(raw_xml_text)
>>> 'payloadID' in xml
True
>>> 'someOtherWeirdThing` in xml
False

Getting siblings by dereferencing as a list

>>> xml = parse_xml(raw_xml_text)
>>> second_credential = xml.Header.To.Credential[1]
>>> print(second_credential.Identity.text())
[e-mail]
>>>

A nice side effect of being able to dereference an CxmlNode as a list is that you can for-loop over CxmlNodes:

>>> xml = parse_xml(raw_xml_text)
>>> for cred in xml.Header.To.Credential
...   print(cred['domain'], cred.Identity.text())
...
AribaNetworkUserId [e-mail]
AribaNetworkUserId [e-mail]
>>>

You can also use the built-in len() function with CxmlNodes:

>>> xml = parse_xml(raw_xml_text)
>>> print(len(xml.Header.To.Credential))
2
>>>

Looking up the original element name with the .tag() method

>>> request = parse_xml(raw_xml_text).Request.ConfirmationRequest
>>> print(request.tag())
ConfirmationRequest
>>>

Getting the inner-text of the element with the .text() method

>>> xml = parse_xml(raw_xml_text)
>>> node = xml.Header.From.Credential.Identity
>>> print(node.text())
942888711
>>>

Getting the first of a list of siblings with the .first() method

>>> xml = parse_xml(raw_xml_text)
>>> node = xml.Header.To.Credential.first()
>>> print node['type']
marketplace
>>>

Getting the last of a list of siblings with the .last() method

>>> xml = parse_xml(raw_xml_text)
>>> print(xml.Header.To.Credential.last().Identity.text())
[e-mail]
>>>

Filtering nodes with the .filter() method

>>> extrinsics = parse_xml(raw_xml_text).Request.OrderRequest.OrderRequestHeader.Extrinsic
>>> user_nodes = extrinsics.filter(lambda n: n['name'].startswith('User'))
>>> print(len(user_nodes), len(extrinsics))
5 12
>>>

Simpler filtering nodes with the .filter_prop() method

>>> extrinsics = parse_xml(raw_xml_text).Request.OrderRequest.OrderRequestHeader.Extrinsic
>>> user_nodes = extrinsics.filter_prop('name', 'User', startswith_predicate)
>>> print(len(user_nodes), len(extrinsics))
5 12
>>>

Mapping nodes to something else with a lambda expression

>>> line_item = parse_xml(raw_xml_text).Request.OrderRequest.ItemOut[0]
>>> taxes = line_item.tax.map(lambda n: float(n.TaxDetail.TaxAmount.text()))
>>> print(taxes)
[0.805063025, 0.0, 3.703289915]
>>>

Simpler mapping with the .map_attr() method

>>> from_header = parse_xml(raw_xml_text).Header.From
>>> identity_nodes = from_header.Credential.map_attr('Identity')
>>> print(identity_nodes[1].text())
test_customer
>>>

Simpler mapping with the .map_prop() method

>>> from_header = parse_xml(raw_xml_text).Header.From
>>> credential_domains = from_header.Credential.map_prop('domain')
>>> print(credential_domains)
['DUNS', 'CompanyName', 'InteropKey']
>>>

Simpler mapping with the .map_text() method

>>> bill_to = parse_xml(raw_xml_text).Request.OrderRequest.OrderRequestHeader.BillTo
>>> street = bill_to.Address.PostalAddress.Street.map_text()
>>> print(street)
['1242 West Main Street', 'Suite 6a']
>>>

String joining with the .join_text() method

>>> address = parse_xml(raw_xml).Request.OrderRequest.OrderRequestHeader.BillTo.Address
>>> print(address.PostalAddress.Street.join_text("\n"))
1231 West Main Street
Suite 6a
>>>

String joining with the .join_prop() method

>>> xml = parse_xml(raw_xml_text)
>>> print(xml.Header.From.Credential.join_prop('domain'))
DUNS, CompanyName, InteropKey
>>>

XML with namespaces

The dunder_xml_reader package can handle XML with varying defined namespaces. It does this by optionally replacing namespaces in tags with an alias prefix. Take the following XML for example:

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Body>
    <ReceiveClaimResponse xmlns="http://insurance.com/webservices/ReceiveClaim.job">
      <ClaimResponse>
        <status>S</status>
        <errorDetail/>
        <claimNumber>LC0000001742</claimNumber>
      </ClaimResponse>
    </ReceiveClaimResponse>
  </soapenv:Body>
</soapenv:Envelope>

Using the namespaces argument to parse_xml(), you can have the attributes prefixed with an alias:

>>> soap = parse_xml(raw_soap_text, namespaces={
...   'http://schemas.xmlsoap.org/soap/envelope/': 'soap',
...   'http://insurance.com/webservices/ReceiveClaim.job': 'claim'
... }
>>> soap.soap_Body.claim_ReceiveClaimResponse.claim_ClaimResponse.claim_status.text()
S
>>>

If you don't want any prefixes and you just want to strip out the namespace references you can either pass a set of namespace strings to parse_xml():

>>> soap = parse_xml(raw_soap_text, namespaces={
...   'http://schemas.xmlsoap.org/soap/envelope/',
...   'http://insurance.com/webservices/ReceiveClaim.job'
... }
>>> soap.Body.ReceiveClaimResponse.ClaimResponse.status.text()
S
>>>

Bonus: SafeReference

SafeReference is a wrapper that you can wrap a xmlNode instance in (or any other Python object graph for that matter) that will prevent errors like:

TypeError: 'NoneType' object is not subscriptable
AttributeError: 'NoneType' object has no attribute 'blah'

A quick example. Say you want to issue the following line of code:

>>> x = xml.Request.OrderRequest.OrderRequestHeader.BillTo.Address.PostalAddress.City.text()

But, the XML that was parsed in order to create the xml object doesn't have a BillTo element. You'd expect to get the following exception:

AttributeError: 'OrderRequestHeader' object has no attribute 'BillTo'

Now you could test your way through the graph heirchy (aka look-before-you-leap):

>>> if hasattr(xml, 'Request') and \
...    hasattr(xml.Request, 'OrderRequest') and \
...    hasattr(xml.Request.OrderRequest, 'OrderRequestHeader') and \
...    hasattr(xml.Request.OrderRequest.OrderRequestHeader, 'BillTo'):
...   x = xml.Request.OrderRequest.OrderRequestHeader.BillTo.Address.PostalAddress.City.text()
... else:
...   x = ''

Or, you could just wrap your assignment in a try/except (aka ask-for-forgiveness):

>>> try:
...   x = xml.Request.OrderRequest.OrderRequestHeader.BillTo.Address.PostalAddress.City.text()
... except AttributeError:
...   x = ''

But in both cases you're writing a lot of "check to see if it's really there"-type code. The goal of SafeReference is to prevent that code from needing to be written in the first place. If your object-graph is wrapped in a SafeReference instance, it will check at each level for the requested attribute to be there or missing. If it's there, it will return the requested attribute. However, if it's not there, it will return a default-value. Here's an example:

>>> xml = parse_xml(raw_xml_text_thats_missing_a_BillTo_element)
>>> safe_xml = safe_reference(xml, 'n/a')
>>> x = safe_xml.Request.OrderRequest.OrderRequestHeader.BillTo.Address.PostalAddress.City.text()
>>> print(x)
n/a
>>>

This result of n/a will be the result no matter what item is missing (Request, OrderRequest, OrderRequestHeader, BillTo, etc.) along the way. If any of the objects in that long dereferencing are missing, the default will be returned. However, if nothing is missing, the actual graph value will be returned.

>>> xml = parse_xml(raw_xml_text_with_nothing_missing)
>>> safe_xml = safe_reference(xml, 'n/a')
>>> x = safe_xml.Request.OrderRequest.OrderRequestHeader.BillTo.Address.PostalAddress.City.text()
>>> print(x)
New York
>>>

Bonus 2: extract_namespaces()

If you have a chunk of XML or a SOAP message that conains numerous namespace references, figuring our what the namespaces are (so you can pass them as an argument to the parse_xml()), you can use the extract_namespaces() function:

>>> from dunder_xml_reader import extract_namespaces
>>> xml = """
...     <section xmlns="http://www.ibm.com/events" xmlns:bk="urn:loc.gov:books" xmlns:pi="urn:personalInformation"
...              xmlns:isbn='urn:ISBN:0-395-36341-6'>
...         <title>Book-Signing Event</title>
...         <signing>
...             <bk:author pi:title="Mr" pi:name="Jim Ross"/>
...             <book bk:title="Writing COBOL for Fun and Profit" isbn:number="0426070806"/>
...             <comment xmlns=''>What a great issue!</comment>
...         </signing>
...     </section>
...  """
...
>>> ns = extract_namespaces(xml)
>>> print(ns)
{(None, 'http://www.ibm.com/events'), ('bk', 'urn:loc.gov:books'), ('pi', 'urn:personalInformation'), ('isbn', 'urn:ISBN:0-395-36341-6') }
>>>