pygixml: Python Giant XML

pygixml (Python Giant XML) is a Cython framework built on two specialized C++ engines: pugixml for its in-memory DOM parser (XPath, objectify, dictify), and an inlined yxml push parser for true constant-memory streaming. Between the two, pygixml covers everything lxml and xmltodict do: dotted objectify navigation, XPath 1.0, and an xmltodict-compatible dictify, plus a streaming layer neither of them has, which is what makes pygixml the package of choice for big XML and big-data pipelines.

📚 View Full Documentation

Why pygixml?

Speed, memory, and size. pygixml brings pugixml's battle-tested C++ parser directly to Python.

Every parsing/conversion layer pygixml ships (raw DOM, dictify, objectify, jsonify) is measured against lxml, ElementTree, xmltodict, and xmljson in a full, reproducible benchmark suite (see benchmarks/), not summarized here as a handful of numbers that inevitably go stale the moment the code changes. The live, interactive report (refreshed on a schedule, not hand-edited) is on the Performance page.

Two things worth knowing going in, because they're structural (true regardless of which exact numbers the report shows this week), not a one-time measurement:

• jsonify.stream_dump holds memory flat as input grows. Every DOM-based approach (pygixml's own DOM path included, lxml, xmltodict) allocates roughly in proportion to the input, so its memory use grows with file size. stream_dump never builds a tree at all, so it doesn't: this is a different complexity class (O(1) vs O(n)), not a fixed "X times less" you'd see at only one input size. It costs something in return: see the report's own memory panel for the honest trade-off, not just the part that flatters it.
• Package size: pygixml installs in well under a megabyte, without vendoring a build of libxml2/libxslt the way lxml does.

Built for big XML

Those benchmark numbers are for documents that fit comfortably in memory. For the documents that don't (multi-gigabyte exports, logs, data dumps), pygixml's streaming layer is the part lxml and xmltodict simply don't have:

• pygixml.iterfind / dictify.iterdict / jsonify.iterjsonl: yxml-based incremental parsing in constant memory: one element (or one dict, or one JSON line) in flight at a time, regardless of whether the source document is 10 KB or 10 GB.

• jsonify.stream_dump(xml_path, json_path) (the headline feature) converts a giant XML file into a single, valid, giant JSON file, entirely in C++, in constant memory, with an xmltodict-compatible output shape (same @attr / #text / repeated-siblings-as-array conventions as dictify.parse). No DOM tree is ever built, no intermediate Python dict/list/str is ever allocated, and the file never has to fit in RAM: only an in-place seek-and-patch trick on the output file is used to close JSON arrays correctly as repeated siblings are discovered. As far as we know, this is the only Python package that can do this without buffering the document, the output, or both, and without crashing the process once the file gets genuinely large.

  The memory guarantee is unconditional (verified: ~17MB peak, completely flat from 5K to 640K records, see benchmarks/adhoc/bench_record_shaped.py). The time guarantee is O(n) for the shape almost all real giant XML actually has: one repeated tag per nesting level (<orders><order>...</order>...). It degrades towards O(n²) only in an adversarial shape: two or more different tags repeating and interleaving at the same level for many iterations (e.g. <a>1</a><b>1</b><a>2</a><b>2</b>... with no wrapping element around each pair). If that's genuinely your data shape, jsonify.stream_jsonl (below) sidesteps the problem entirely by not trying to preserve a single JSON document at all.

• jsonify.stream_jsonl(xml_path, jsonl_path, tag): the file-to-file counterpart of iterjsonl (filters by tag, unlike stream_dump which always converts the whole document): streams straight to a .jsonl file, one matched element per line, same constant-memory, all-C++ guarantee.

from pygixml import jsonify

# A multi-GB XML file in, a multi-GB JSON file out: peak memory stays flat.
jsonify.stream_dump("huge_export.xml", "huge_export.json")

# Or, one record per line:
jsonify.stream_jsonl("huge_export.xml", "huge_export.jsonl", "record")

Features

• Fast parsing: pugixml's C++ DOM parser, consistently at or near the front across the benchmark suite (see the Performance page)
• Constant-memory streaming: jsonify.stream_dump holds flat regardless of input size; nothing else compared here does
• Small footprint: well under a megabyte installed, no bundled libxml2/libxslt
• Full XPath 1.0: complete query engine with all standard functions
• Pythonic API: intuitive properties and methods, not a direct C++ mirror
• objectify: lxml.objectify-style dotted navigation
• dictify: xmltodict-compatible XML → dict conversion
• jsonify: direct XML → JSON, in memory or streamed straight to disk in constant memory (stream_dump, stream_jsonl)
• HTML ↔ XML: parse_html / html_to_xml turn real-world HTML into well-formed XML you can query with XPath, xml_to_html goes back; names XML can't hold are mapped reversibly (encode_name / decode_name)
• Tree diff / patch: a minimal edit script between two documents (HTML or XML), plus similarity / signature for structural comparison
• Streaming (iterfind, iterdict, iterjsonl): constant-memory, yxml-based incremental parsing for documents too big to load whole
• CLI tools: pygixml cat (pretty-print/colorize), pygixml query (XPath/dotted query), pygixml jsonify (convert to JSON), pygixml convert (XML/JSON/YAML/TOON, any direction), pygixml stream (filter giant files in bounded memory): see Command Line Tools
• Cross-platform: Windows, Linux, macOS
• Text extraction: recursive text gathering with configurable joins
• XML serialization: output with custom indentation
• Node iteration: depth-first traversal of the entire document

Installation

# From PyPI
pip install pygixml

# Or from GitHub
pip install git+https://github.com/MohammadRaziei/pygixml.git

Quick Start

import pygixml

# Parse XML from string
xml = """
<library>
    <book id="1" category="fiction">
        <title>The Great Gatsby</title>
        <author>F. Scott Fitzgerald</author>
        <year>1925</year>
    </book>
    <book id="2" category="fiction">
        <title>1984</title>
        <author>George Orwell</author>
        <year>1949</year>
    </book>
</library>
"""

doc = pygixml.parse_string(xml)
root = doc.root                           # <library>

# Access children and attributes
book = root.child("book")
print(book.name)                          # book
print(book.attribute("id").value)         # 1
print(book.child("title").text())         # The Great Gatsby

# XPath queries
fiction = root.select_nodes("book[@category='fiction']")
print(f"Found {len(fiction)} fiction books")

# Create & save
doc = pygixml.XMLDocument()
root = doc.append_child("catalog")
root.append_child("item").set_value("Hello")
doc.save_file("output.xml")

Properties vs Methods

A quick reference so you don't get tripped up:

objectify: dotted navigation

pygixml.objectify provides an lxml.objectify-inspired interface for navigating XML with plain Python attribute access.

from pygixml import objectify

xml = """
<database name="users_db" version="1.2">
    <user-profile id="101" verified="true">
        <first_name>Mohammad</first_name>
        <balance>450.75</balance>
    </user-profile>
    <entry>Value A</entry>
    <entry>Value B</entry>
</database>
"""

root = objectify.from_string(xml)

# Dotted navigation: underscores map to hyphens automatically
print(root.user_profile.first_name)        # ObjectifiedElement(<first_name>)
print(str(root.user_profile.first_name))   # 'Mohammad'

# Automatic type inference for attributes
print(root.version)                        # 1.2   (float)
print(root.user_profile.id)               # 101   (int)
print(root.user_profile.verified)         # True  (bool)

# Text content
print(str(root.user_profile.first_name))  # 'Mohammad'   always str
print(root.user_profile.balance())        # 450.75        type-inferred

# Repeated siblings: indexing and iteration
print(root.entry[0])                      # ObjectifiedElement
print([str(e) for e in root.entry])       # ['Value A', 'Value B']

# Safe attribute access: never raises
print(root.get('version'))                # 1.2
print(root.get('missing', 'default'))     # 'default'

# Search descendants
print(root.find('balance'))               # ObjectifiedElement(<balance>)
print(root.find('balance', recursive=False))  # None  (not a direct child)
print(root.findall('entry'))              # [ObjectifiedElement, ...]

# Write support: modify in place
root.user_profile.first_name = "Ali"      # update child element text
root.version = 2.0                        # update attribute
root.timeout = 30                         # create new child element

# Delete
del root.timeout                          # remove child element
del root.version                          # remove attribute

objectify API

dictify: XML to dict

pygixml.dictify converts XML to a nested dict, compatible with the xmltodict library.

from pygixml import dictify

xml = """
<database name="users_db" version="1.2">
    <user-profile id="101" verified="true">
        <first_name>Mohammad</first_name>
        <balance>450.75</balance>
    </user-profile>
    <entry>Value A</entry>
    <entry>Value B</entry>
</database>
"""

# Parse XML → dict
d = dictify.parse(xml)
# {
#   'database': {
#     '@name': 'users_db',
#     '@version': '1.2',
#     'user-profile': {
#       '@id': '101', '@verified': 'true',
#       'first_name': 'Mohammad', 'balance': '450.75'
#     },
#     'entry': ['Value A', 'Value B']
#   }
# }

# Repeated siblings → list automatically
print(d['database']['entry'])             # ['Value A', 'Value B']

# Attributes prefixed with '@'
print(d['database']['@name'])             # 'users_db'

# Custom options
d = dictify.parse(xml,
    attr_prefix='',       # no prefix: attrs and children in same namespace
    cdata_key='text',     # key for text content (default '#text')
    force_list={'entry'}, # always a list, even with one element
    cdata_separator='',   # join text split by child elements (default: first run)
)

# Parse from file
d = dictify.parse_file('data.xml')

# Convert back to XML
xml_out = dictify.unparse(d, pretty=True, indent='\t')
print(xml_out)

dictify API

HTML, diff and patch

import pygixml

# Messy HTML -> well-formed XML, and back
pygixml.html_to_xml("<ul><li>a<li>b<br></ul>")
# '<ul><li>a</li><li>b<br/></li></ul>'
doc = pygixml.parse_html("<ul><li>a<li>b<br></ul>")      # a normal XMLDocument
pygixml.xml_to_html(doc, unwrap="root")
# '<ul><li>a</li><li>b<br></li></ul>'

# Names XML can't hold (o:p, @click, [prop] ...) are mapped reversibly with the
# SQL/XML rules; `case` picks lower-case (default), "keep" or "escape"
pygixml.html_to_xml("<o:p @Click=1>x</o:p>", case="keep")
# '<o_x003A_p _x0040_Click="1">x</o_x003A_p>'
pygixml.decode_name("_x0040_Click")                       # '@Click'

# Edit script between two documents, and applying it
a = pygixml.parse_html("<ul><li>one</li></ul>")
b = pygixml.parse_html("<ul><li>one</li><li>two</li></ul>")
actions = pygixml.diff(a, b)
# [Insert(parent='/ul[1]', position=1, xml='<li>two</li>')]
pygixml.similarity(a, b)                                   # 0.72
pygixml.patch(a, actions)
pygixml.is_equal(a, b)                                     # True

A document from parse_html remembers it came from HTML, so to_html() restores the original names; a document from parse_string keeps its names as they are.

See the HTML and diff pages of the documentation for the options (ignore_whitespace, ignore_comments, class_as_set, keys, ignore_attrs, moves, text_diff).

Command Line Tools

pygixml installs one CLI entry point, pygixml, with three subcommands: no Python code required for one-off queries, conversions, or filtering giant files from a shell pipeline. Every subcommand also works as python -m pygixml <subcommand>.

pygixml cat: pretty-print and colorize

pygixml cat data.xml                    # pretty + colorized (auto: only
                                         # on a real terminal, and only
                                         # if colorama is installed)
cat data.xml | pygixml cat -            # from stdin
pygixml cat data.xml --color always     # force color even when piped
pygixml cat data.xml --color never      # force plain, no color
pygixml cat data.xml --indent 4         # 4-space indent instead of 2
pygixml cat data.xml -o pretty.xml      # write to a file (never colorized
                                         # by 'auto': a file isn't a tty)

python -m pygixml cat data.xml

Colorizing is fully optional: install it with pip install pygixml[all] (pulls in colorama, along with YAML/TOON support for pygixml convert). Without it, cat still works exactly the same, just without color: plain, pretty-printed XML, same as --color never.

pygixml query

# XPath (starts with / or //)
pygixml query data.xml "//user-profile[@id='101']/first_name"

# dotted, objectify-style (starts with .): first segment names the root
pygixml query data.xml ".database.user_profile.first_name"
pygixml query data.xml ".database.user_profile.@id"      # attribute
pygixml query data.xml ".database.entry[1]"              # index
pygixml query data.xml ".database.entry[*]"              # all siblings
pygixml query data.xml ".database.user_profile.text()"   # text content

# output formats
pygixml query data.xml ".database" --format xml
pygixml query data.xml ".database" --format json --pretty

# just the count, exit code like grep (0 = match, 1 = no match)
pygixml query data.xml ".database.entry[*]" --count

# multiple files, stdin, NUL-separated for xargs -0
pygixml query *.xml ".config.host"
cat data.xml | pygixml query - ".config.host"
pygixml query data.xml ".database.entry[*]" --null | xargs -0 -n1 echo

# python -m form works identically
python -m pygixml query data.xml ".database.user_profile.first_name"

pygixml jsonify: convert a whole file

pygixml jsonify data.xml                        # compact JSON to stdout
pygixml jsonify data.xml -p                     # pretty (2-space indent)
pygixml jsonify data.xml -o data.json           # write to a file
pygixml jsonify data.xml --force-list item      # always make <item> a list
cat data.xml | pygixml jsonify -                # read from stdin

python -m pygixml jsonify data.xml -o data.json
pygixml json data.xml                           # `json` is a short alias

Automatically switches to jsonify.stream_dump (constant memory) for files over 64MB; --stream / --no-stream force one mode or the other regardless of size.

pygixml stream: filter a giant file

Reads the file once via iterparse, tag by tag, holding at most one matched element's subtree in memory: the right tool once a file is too large for pygixml query/pygixml jsonify's DOM mode.

# every <order>, one JSON object per line (JSONL)
pygixml stream orders.xml --tag order

# simple filters: numeric or string, on a child path or an @attribute
pygixml stream orders.xml --tag order --where "total>100"
pygixml stream orders.xml --tag order --where "@status=shipped"

# multiple --where are AND'ed together
pygixml stream orders.xml --tag order \
    --where "@status=shipped" --where "customer=acme"

# a real JSON array instead of JSONL: still one pass, still bounded memory
pygixml stream orders.xml --tag order --where "total>100" --format array -p

# just count, or stop after N
pygixml stream orders.xml --tag order --where "@status=shipped" --count
pygixml stream orders.xml --tag order --limit 10

cat orders.xml | pygixml stream - --tag order
python -m pygixml stream orders.xml --tag order

--where supports =, !=, >, <, >=, <=; the left-hand side is either @attr or a child path like customer / items/item/sku (same syntax as StreamElement.findtext).

pygixml convert: XML, JSON, YAML, TOON, any direction

pygixml convert data.xml -o data.json         # format guessed from extensions
pygixml convert data.xml -o data.yaml
pygixml convert data.json -o data.xml -p      # pretty XML
pygixml convert data.yaml --to toon           # stdout needs --to explicitly
cat data.xml | pygixml convert - --from xml --to json   # stdin needs --from too

python -m pygixml convert data.xml -o data.yaml

Built on :class:pygixml.formats.FormatDocument, a small class wrapping a plain dict (the same @attr/#text/list shape dictify.parse produces), with to_json/to_xml/to_yaml/to_toon and matching from_* classmethods. JSON and XML always work (stdlib json + this package's own dictify); YAML needs PyYAML and TOON needs ctoon (also written by the author of pygixml). Both are optional, and asking for one you don't have gives a clear pip install ... error instead of crashing:

pip install pygixml[all]   # colorama (for `cat`) + PyYAML + ctoon

Like cat/query, this loads the whole document: for XML too big to fit in memory, convert with pygixml jsonify --stream instead.

Advanced Features

Text Content Extraction

import pygixml

xml = """
<root>
    <simple>Hello World</simple>
    <nested>
        <child>Child Text</child>
        More text
    </nested>
    <mixed>Text <b>with</b> mixed <i>content</i></mixed>
</root>
"""

doc = pygixml.parse_string(xml)
root = doc.root

print(root.child("simple").text())                # Hello World
print(root.child("nested").text(join=" | "))      # Child Text | More text
print(root.child("mixed").text(recursive=False))  # Text

XML Serialization

import pygixml

doc = pygixml.XMLDocument()
root = doc.append_child("root")
root.append_child("item").set_value("content")

print(root.xml)
# <root>
#   <item>content</item>
# </root>

print(root.to_string("    "))  # 4-space indent

Document Iteration

import pygixml

doc = pygixml.parse_string("<root><a/><b/></root>")

for node in doc:
    print(f"{node.type:12s} {node.name}")
# document
# element       root
# element       a
# element       b

Modifying XML

import pygixml

doc = pygixml.parse_string("<person><name>John</name></person>")
root = doc.root

root.child("name").set_value("Jane")
root.child("name").name = "full_name"
root.append_child("age").set_value("30")

print(root.xml)
# <person>
#   <full_name>Jane</full_name>
#   <age>30</age>
# </person>

XPath Support

Full XPath 1.0 via pugixml's engine:

import pygixml

xml = """
<library>
    <book id="1" category="fiction">
        <title>The Great Gatsby</title>
        <author>F. Scott Fitzgerald</author>
        <year>1925</year>
        <price>12.99</price>
    </book>
    <book id="2" category="fiction">
        <title>1984</title>
        <author>George Orwell</author>
        <year>1949</year>
        <price>10.99</price>
    </book>
</library>
"""

doc = pygixml.parse_string(xml)
root = doc.root

# Select nodes
books = root.select_nodes("book")
print(f"Found {len(books)} books")

# Predicates
fiction = root.select_nodes("book[@category='fiction']")
print(f"Found {len(fiction)} fiction books")

# Single node
book = root.select_node("book[@id='2']")
if book:
    print(book.node.child("title").text())    # 1984

# Pre-compiled query for repeated use
query = pygixml.XPathQuery("book[year > 1930]")
recent = query.evaluate_node_set(root)
print(f"Found {len(recent)} books published after 1930")

# Scalar evaluations
avg = pygixml.XPathQuery("sum(book/price) div count(book)").evaluate_number(root)
print(f"Average price: ${avg:.2f}")           # Average price: $11.99

has_orwell = pygixml.XPathQuery("book[author='George Orwell']").evaluate_boolean(root)
print(f"Has Orwell books: {has_orwell}")       # Has Orwell books: True

Supported XPath

Core API

Module-level functions: parse_string(xml), parse_file(path).

Benchmarks

cmake -S benchmarks -B benchmarks/build
cmake --build benchmarks/build --target pygixml_bench_report

Compares pygixml's parse/dictify/objectify/jsonify against lxml, ElementTree, xmltodict, and xmljson: speed, memory at scale, and install size. Writes the raw numbers to benchmarks/results/*.json; the docs build (sphinx_docs) renders the interactive page from it. See benchmarks/README.md for the full methodology, or the live copy on the Performance page.

Documentation

📖 Full docs: https://mohammadraziei.github.io/pygixml/

Security

pygixml is not affected by the common XML attacks (XXE, Billion Laughs, external DTD retrieval) that lxml/defusedxml guard against, because its two embedded parsers, pugixml (DOM API) and yxml (streaming API), never implement custom entity resolution or external DTD fetching in the first place, so there's no "safe mode" to configure. See #8 for the full technical breakdown.

License

MIT License: see LICENSE.

Enjoy pygixml? Star the repository ⭐ 👉 Star pygixml on GitHub

Acknowledgments

• pugixml: Fast and lightweight C++ XML library
• yxml: Tiny, dependency-free streaming XML parser, powering pygixml's constant-memory streaming layer
• Cython: C extensions for Python
• scikit-build: Modern Python build system