pygixml — Python Giant XML

pygixml — Python Giant XML — is a Cython framework built on two specialized C++ engines: pugixml for its in-memory DOM parser (XPath, objectify, dictify), and an inlined yxml push parser for true constant-memory streaming. Between the two, pygixml covers everything lxml and xmltodict do — dotted objectify navigation, XPath 1.0, and an xmltodict-compatible dictify — plus a streaming layer neither of them has, which is what makes pygixml the package of choice for big XML and big-data pipelines.

📚 View Full Documentation

Why pygixml?

Speed, memory, and size. pygixml brings pugixml's battle-tested C++ parser directly to Python — with numbers that speak for themselves.

Parsing Performance (5 000 elements, 50 iterations)

Memory Usage (5 000 elements, peak)

Package Size

All numbers from benchmarks/full_benchmark.py. See the Performance page for the full comparison across 6 XML sizes.

Built for big XML

Those benchmark numbers are for documents that fit comfortably in memory. For the documents that don't — multi-gigabyte exports, logs, data dumps — pygixml's streaming layer is the part lxml and xmltodict simply don't have:

• pygixml.iterfind / dictify.iterdict / jsonify.iterjsonl — yxml-based incremental parsing in constant memory: one element (or one dict, or one JSON line) in flight at a time, regardless of whether the source document is 10 KB or 10 GB.
• jsonify.stream_dump(xml_path, json_path) — the headline feature: converts a giant XML file into a single, valid, giant JSON file, entirely in C++, in constant memory, with an xmltodict-compatible output shape (same @attr / #text / repeated-siblings-as-array conventions as dictify.parse). No DOM tree is ever built, no intermediate Python dict/list/str is ever allocated, and the file never has to fit in RAM — only an in-place seek-and-patch trick on the output file is used to close JSON arrays correctly as repeated siblings are discovered. As far as we know, this is the only Python package that can do this without buffering the document, the output, or both, and without crashing the process once the file gets genuinely large.
• jsonify.stream_jsonl(xml_path, jsonl_path, tag) — the file-to-file counterpart of iterjsonl (filters by tag, unlike stream_dump which always converts the whole document): streams straight to a .jsonl file, one matched element per line, same constant-memory, all-C++ guarantee.

from pygixml import jsonify

# A multi-GB XML file in, a multi-GB JSON file out -- peak memory stays flat.
jsonify.stream_dump("huge_export.xml", "huge_export.json")

# Or, one record per line:
jsonify.stream_jsonl("huge_export.xml", "huge_export.jsonl", "record")

Features

• Blazing-fast parsing — up to 14× faster than ElementTree
• Low memory — 7× less than ElementTree, on par with lxml
• Tiny footprint — 0.43 MB installed (12.7× smaller than lxml)
• Full XPath 1.0 — complete query engine with all standard functions
• Pythonic API — intuitive properties and methods, not a direct C++ mirror
• objectify — lxml.objectify-style dotted navigation
• dictify — xmltodict-compatible XML → dict conversion
• jsonify — direct XML → JSON, in memory or streamed straight to disk in constant memory (stream_dump, stream_jsonl)
• Streaming (iterfind, iterdict, iterjsonl) — constant-memory, yxml-based incremental parsing for documents too big to load whole
• Cross-platform — Windows, Linux, macOS
• Text extraction — recursive text gathering with configurable joins
• XML serialization — output with custom indentation
• Node iteration — depth-first traversal of the entire document

Installation

# From PyPI
pip install pygixml

# Or from GitHub
pip install git+https://github.com/MohammadRaziei/pygixml.git

Quick Start

import pygixml

# Parse XML from string
xml = """
<library>
    <book id="1" category="fiction">
        <title>The Great Gatsby</title>
        <author>F. Scott Fitzgerald</author>
        <year>1925</year>
    </book>
    <book id="2" category="fiction">
        <title>1984</title>
        <author>George Orwell</author>
        <year>1949</year>
    </book>
</library>
"""

doc = pygixml.parse_string(xml)
root = doc.root                           # <library>

# Access children and attributes
book = root.child("book")
print(book.name)                          # book
print(book.attribute("id").value)         # 1
print(book.child("title").text())         # The Great Gatsby

# XPath queries
fiction = root.select_nodes("book[@category='fiction']")
print(f"Found {len(fiction)} fiction books")

# Create & save
doc = pygixml.XMLDocument()
root = doc.append_child("catalog")
root.append_child("item").set_value("Hello")
doc.save_file("output.xml")

Properties vs Methods

A quick reference so you don't get tripped up:

objectify — dotted navigation

pygixml.objectify provides an lxml.objectify-inspired interface for navigating XML with plain Python attribute access.

from pygixml import objectify

xml = """
<database name="users_db" version="1.2">
    <user-profile id="101" verified="true">
        <first_name>Mohammad</first_name>
        <balance>450.75</balance>
    </user-profile>
    <entry>Value A</entry>
    <entry>Value B</entry>
</database>
"""

root = objectify.from_string(xml)

# Dotted navigation — underscores map to hyphens automatically
print(root.user_profile.first_name)        # ObjectifiedElement(<first_name>)
print(str(root.user_profile.first_name))   # 'Mohammad'

# Automatic type inference for attributes
print(root.version)                        # 1.2   (float)
print(root.user_profile.id)               # 101   (int)
print(root.user_profile.verified)         # True  (bool)

# Text content
print(str(root.user_profile.first_name))  # 'Mohammad'   always str
print(root.user_profile.balance())        # 450.75        type-inferred

# Repeated siblings — indexing and iteration
print(root.entry[0])                      # ObjectifiedElement
print([str(e) for e in root.entry])       # ['Value A', 'Value B']

# Safe attribute access — never raises
print(root.get('version'))                # 1.2
print(root.get('missing', 'default'))     # 'default'

# Search descendants
print(root.find('balance'))               # ObjectifiedElement(<balance>)
print(root.find('balance', recursive=False))  # None  (not a direct child)
print(root.findall('entry'))              # [ObjectifiedElement, ...]

# Write support — modify in place
root.user_profile.first_name = "Ali"      # update child element text
root.version = 2.0                        # update attribute
root.timeout = 30                         # create new child element

# Delete
del root.timeout                          # remove child element
del root.version                          # remove attribute

objectify API

dictify — XML to dict

pygixml.dictify converts XML to a nested dict, compatible with the xmltodict library.

from pygixml import dictify

xml = """
<database name="users_db" version="1.2">
    <user-profile id="101" verified="true">
        <first_name>Mohammad</first_name>
        <balance>450.75</balance>
    </user-profile>
    <entry>Value A</entry>
    <entry>Value B</entry>
</database>
"""

# Parse XML → dict
d = dictify.parse(xml)
# {
#   'database': {
#     '@name': 'users_db',
#     '@version': '1.2',
#     'user-profile': {
#       '@id': '101', '@verified': 'true',
#       'first_name': 'Mohammad', 'balance': '450.75'
#     },
#     'entry': ['Value A', 'Value B']
#   }
# }

# Repeated siblings → list automatically
print(d['database']['entry'])             # ['Value A', 'Value B']

# Attributes prefixed with '@'
print(d['database']['@name'])             # 'users_db'

# Custom options
d = dictify.parse(xml,
    attr_prefix='',       # no prefix — attrs and children in same namespace
    cdata_key='text',     # key for text content (default '#text')
    force_list={'entry'}, # always a list, even with one element
)

# Parse from file
d = dictify.parse_file('data.xml')

# Convert back to XML
xml_out = dictify.unparse(d, pretty=True, indent='\t')
print(xml_out)

dictify API

Advanced Features

Text Content Extraction

import pygixml

xml = """
<root>
    <simple>Hello World</simple>
    <nested>
        <child>Child Text</child>
        More text
    </nested>
    <mixed>Text <b>with</b> mixed <i>content</i></mixed>
</root>
"""

doc = pygixml.parse_string(xml)
root = doc.root

print(root.child("simple").text())                # Hello World
print(root.child("nested").text(join=" | "))      # Child Text | More text
print(root.child("mixed").text(recursive=False))  # Text

XML Serialization

import pygixml

doc = pygixml.XMLDocument()
root = doc.append_child("root")
root.append_child("item").set_value("content")

print(root.xml)
# <root>
#   <item>content</item>
# </root>

print(root.to_string("    "))  # 4-space indent

Document Iteration

import pygixml

doc = pygixml.parse_string("<root><a/><b/></root>")

for node in doc:
    print(f"{node.type:12s} {node.name}")
# document
# element       root
# element       a
# element       b

Modifying XML

import pygixml

doc = pygixml.parse_string("<person><name>John</name></person>")
root = doc.root

root.child("name").set_value("Jane")
root.child("name").name = "full_name"
root.append_child("age").set_value("30")

print(root.xml)
# <person>
#   <full_name>Jane</full_name>
#   <age>30</age>
# </person>

XPath Support

Full XPath 1.0 via pugixml's engine:

import pygixml

xml = """
<library>
    <book id="1" category="fiction">
        <title>The Great Gatsby</title>
        <author>F. Scott Fitzgerald</author>
        <year>1925</year>
        <price>12.99</price>
    </book>
    <book id="2" category="fiction">
        <title>1984</title>
        <author>George Orwell</author>
        <year>1949</year>
        <price>10.99</price>
    </book>
</library>
"""

doc = pygixml.parse_string(xml)
root = doc.root

# Select nodes
books = root.select_nodes("book")
print(f"Found {len(books)} books")

# Predicates
fiction = root.select_nodes("book[@category='fiction']")
print(f"Found {len(fiction)} fiction books")

# Single node
book = root.select_node("book[@id='2']")
if book:
    print(book.node.child("title").text())    # 1984

# Pre-compiled query for repeated use
query = pygixml.XPathQuery("book[year > 1930]")
recent = query.evaluate_node_set(root)
print(f"Found {len(recent)} books published after 1930")

# Scalar evaluations
avg = pygixml.XPathQuery("sum(book/price) div count(book)").evaluate_number(root)
print(f"Average price: ${avg:.2f}")           # Average price: $11.99

has_orwell = pygixml.XPathQuery("book[author='George Orwell']").evaluate_boolean(root)
print(f"Has Orwell books: {has_orwell}")       # Has Orwell books: True

Supported XPath

Core API

Module-level functions: parse_string(xml), parse_file(path).

Benchmarks

python benchmarks/full_benchmark.py
python benchmarks/benchmark_parsing.py

Compares pygixml against lxml and xml.etree.ElementTree. Results are printed as tables and saved to benchmarks/results/benchmark_full.json.

Documentation

📖 Full docs: https://mohammadraziei.github.io/pygixml/

License

MIT License — see LICENSE.

Enjoy pygixml? Star the repository ⭐ 👉 Star pygixml on GitHub

Acknowledgments

• pugixml — Fast and lightweight C++ XML library
• yxml — Tiny, dependency-free streaming XML parser, powering pygixml's constant-memory streaming layer
• Cython — C extensions for Python
• scikit-build — Modern Python build system