Skip to main content

XML Field Extractor

A flexible, configurable Python tool for extracting fields from XML files and converting them to rectangular data formats like CSV or Excel. Designed to handle complex XML formats like METS, LIDO, PREMIS, and more.

Interactive Config Builder

xml-data-extractor ships with a terminal UI for building configs interactively — no YAML editing needed.

xml-extractor build example.xml

Phase 1 — Select root element: Navigate the XML tree, pick the element that represents one record.

Phase 1: select root element

Phase 2 — Map fields: Navigate to any node, press Enter to map it as a column. Auto-generates XPath.

Phase 2: map fields

The builder saves a ready-to-use config.yaml and can immediately run the extraction.

Disclaimer

🚨 This project was completely developed with the assistance of GitHub Copilot. It is currently in a prototype stage — use at your own risk. Contributions, reviews, and testing are highly appreciated.

✨ Features

  • 🔍 Flexible XPath-based extraction - Extract any field using XPath expressions
  • 🏷️ Automatic namespace detection - No need to manually define namespaces (but you can if needed)
  • 🎯 Advanced filtering - Filter extracted values using regex, startswith, or contains patterns
  • 🔬 Record filtering - Pre-filter records before extraction based on any field criteria (dates, licenses, etc.)
  • 📊 Multiple value handling - Concatenate multiple values with custom separators
  • 🚀 Batch processing - Process entire directories of XML files
  • 📈 Progress tracking - Beautiful progress bars and statistics with Rich
  • 📝 Comprehensive logging - Detailed logs with Loguru
  • 🛡️ Error resilience - Skip invalid XML files and continue processing
  • 🎨 Modern CLI - Built with Click for an intuitive command-line interface

📦 Installation

This project uses uv for fast, reliable package management.

# Clone the repository
git clone <your-repo-url>
cd xml_extractor

# Install dependencies with uv
uv sync

🚀 Quick Start

  1. Prepare your XML files - Place them in a directory (e.g., ./example_data)

  2. Configure extraction - Edit config.yaml to define:

    • Root XPath for record elements
    • Field mappings with XPath expressions
    • Namespaces (optional - auto-detected if omitted)
    • Filters for specific values
  3. Run extraction:

uv run python xml_extractor.py

📖 Configuration

Basic Configuration Structure

# Input/Output
input_directory: "./example_data"
output_file: "output.csv"

# XPath to root element (each match = one CSV row)
root_xpath: ".//oai_dc:dc"

# Namespace definitions (optional)
namespaces:
  oai_dc: "http://www.openarchives.org/OAI/2.0/oai_dc/"
  dc: "http://purl.org/dc/elements/1.1/"

# Field mappings
fields:
  - column: "Title"
    xpath: ".//dc:title/text()"
  
  - column: "Creators"
    xpath: ".//dc:creator/text()"
    separator: " | "  # For multiple values
  
  - column: "URN"
    xpath: ".//dc:identifier/text()"
    filter:
      type: "startswith"
      pattern: "urn:nbn"

Field Configuration Options

Each field can have the following properties:

Property Required Description
column ✅ Yes Name of the CSV column
xpath ✅ Yes XPath expression (relative to root_xpath)
separator ❌ No Separator for multiple values (default: " | ")
filter ❌ No Filter configuration for value selection

Filter Types

Apply filters to select specific values when an XPath matches multiple elements:

Regex Filter:

filter:
  type: "regex"
  pattern: "^urn:nbn"  # Matches URNs starting with "urn:nbn"

StartsWith Filter:

filter:
  type: "startswith"
  pattern: "10."  # Matches DOIs starting with "10."

Contains Filter:

filter:
  type: "contains"
  pattern: "miami.uni-muenster.de"  # Matches URLs containing this domain

Transform

Extract specific parts from text using regex patterns with capture groups:

# Extract date from MARC 008 field and reformat: "180830e20180830||..." → "2018-08-30"
- column: "Publication_Date"
  xpath: ".//marcxml:controlfield[@tag='008']/text()"
  transform:
    regex: "\\d{6}e(\\d{4})(\\d{2})(\\d{2})"
    format: "{0}-{1}-{2}"  # Reformat using capture groups

# Extract language code: "...ger||||||" → "ger"
- column: "Language_Code"
  xpath: ".//marcxml:controlfield[@tag='008']/text()"
  transform:
    regex: "([a-z]{3})\\|{6}$"
    group: 1

Transform options:

  • regex: Regular expression pattern (use \\ for backslashes in YAML)
  • group: Which capture group to extract (default: 0 = full match, 1+ = capture groups)
  • format: Optional format string to reformat the matched data
    • Use {0}, {1}, {2} to reference capture groups
    • Example: "{0}-{1}-{2}" converts 20180830 to 2018-08-30

URL Prefix

Automatically prepend a URL to extracted values:

- column: "DOI"
  xpath: ".//datafield[@tag='024']/subfield[@code='a']/text()"
  url_prefix: "https://doi.org/"  # Converts "10.1234/..." to "https://doi.org/10.1234/..."

Record Filters

Pre-filter records before extraction to only process records matching specific criteria:

# Only extract records from the 1920s with Public Domain licenses
record_filters:
  - xpath: ".//mods:dateCreated/text()"
    condition: "matches"
    value: "192\\d"
  

Record Filters

Pre-filter records before extraction to only process records matching specific criteria:

# Only extract records from the 1920s with Public Domain licenses
record_filters:
  - xpath: ".//mods:dateCreated/text()"
    condition: "matches"
    value: "192\\d"
  
  - xpath: ".//mods:accessCondition/text()"
    condition: "contains"
    value: "publicdomain"

Available filter conditions: exists, not_exists, equals, not_equals, contains, not_contains, matches, not_matches, date_after, date_before, in, not_in

📖 See RECORD_FILTERS.md for complete documentation and examples

🎯 Usage Examples

Basic Usage

# Use default config.yaml
uv run python xml_extractor.py

# Specify custom config
uv run python xml_extractor.py --config my_config.yaml

# Override input/output
uv run python xml_extractor.py --input ./my_xmls --output results.csv

# Enable debug mode
uv run python xml_extractor.py --debug

Command-Line Options

Options:
  -c, --config PATH       Path to configuration YAML file [default: config.yaml]
  -i, --input PATH        Input directory containing XML files (overrides config)
  -o, --output PATH       Output CSV file path (overrides config)
  --debug                 Enable debug mode with verbose logging
  --log-file PATH         Log file path (overrides config)
  --help                  Show this message and exit

📂 Example: Dublin Core OAI-PMH Records

Input XML Structure

Your XML files can have different structures:

Single record per file:

<OAI-PMH xmlns="...">
  <GetRecord>
    <record>
      <metadata>
        <oai_dc:dc xmlns:oai_dc="..." xmlns:dc="...">
          <dc:title>My Title</dc:title>
          <dc:creator>Author Name</dc:creator>
          <!-- more fields -->
        </oai_dc:dc>
      </metadata>
    </record>
  </GetRecord>
</OAI-PMH>

Multiple records per file:

<records>
  <record>
    <metadata>
      <oai_dc:dc xmlns:oai_dc="..." xmlns:dc="...">
        <!-- fields -->
      </oai_dc:dc>
    </metadata>
  </record>
  <record>
    <metadata>
      <oai_dc:dc xmlns:oai_dc="..." xmlns:dc="...">
        <!-- fields -->
      </oai_dc:dc>
    </metadata>
  </record>
</records>

Both formats work seamlessly - the tool finds all oai_dc:dc elements across all files.

Output CSV

Title,Creators,Date,URN,DocType
"Religiöse Traditionen in...","Dam, P. (Peter) van",2018-08-30,urn:nbn:de:hbz:6-87159515859,doc-type:article
"Urinary Dickkopf 3...","Jehn, U. (Ulrich) | Altuner, U. (Ugur)",2024-08-21,urn:nbn:de:hbz:6-05978733071,doc-type:article

🔧 Advanced Examples

METS Configuration

root_xpath: ".//mets:mets"

namespaces:
  mets: "http://www.loc.gov/METS/"
  mods: "http://www.loc.gov/mods/v3"
  xlink: "http://www.w3.org/1999/xlink"

fields:
  - column: "Title"
    xpath: ".//mets:dmdSec/mets:mdWrap/mets:xmlData/mods:mods/mods:titleInfo/mods:title/text()"
  
  - column: "FileID"
    xpath: ".//mets:file/@ID"

LIDO Configuration

root_xpath: ".//lido:lido"

namespaces:
  lido: "http://www.lido-schema.org"

fields:
  - column: "ObjectTitle"
    xpath: ".//lido:titleWrap/lido:titleSet/lido:appellationValue/text()"
  
  - column: "ObjectType"
    xpath: ".//lido:objectWorkType/lido:term/text()"

📊 Statistics & Logging

After extraction, you'll see:

  • ✅ Files processed
  • ⚠️ Files skipped (due to errors)
  • 📝 Total records extracted
  • ⚠️ Fields with missing data
  • ❌ Errors encountered

All details are logged to extraction.log (configurable).

🛠️ Development

Project Structure

xml_extractor/
├── xml_extractor.py      # Main application
├── config.yaml           # Configuration file
├── pyproject.toml        # Project dependencies
├── README.md             # This file
└── extraction.log        # Generated log file

Dependencies

  • lxml - Fast XML processing with XPath support
  • rich - Beautiful terminal output
  • click - CLI framework
  • loguru - Simple, powerful logging
  • pyyaml - YAML configuration parsing

🤝 Contributing

Contributions are welcome! Please feel free to submit issues or pull requests.

📄 License

MIT License - feel free to use this tool for your projects!

💡 Tips

  1. Start with debug mode (--debug) when creating a new configuration to see what's happening
  2. Test with a small subset of files first before processing large batches
  3. Use filters to extract specific identifiers (URNs, DOIs, etc.) from multi-value fields
  4. Check the log file (extraction.log) for warnings about missing XPath matches
  5. Namespaces are auto-detected - you only need to define them manually if auto-detection fails

🆘 Troubleshooting

"No records found with root_xpath"

  • Check your root_xpath expression
  • Verify namespace prefixes match your XML
  • Try without namespace prefix: .//dc instead of .//oai_dc:dc

"Invalid XPath expression"

  • Ensure XPath syntax is correct
  • Check that namespace prefixes are defined
  • Use .// for descendant search, / for direct children

"XML syntax error"

  • Enable skip_invalid_xml: true in config to skip bad files
  • Check XML file encoding (should be UTF-8)
  • Validate XML structure with an XML validator

Metadata

Release files for xml-data-extractor 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for xml-data-extractor 0.1.0
File Size Uploaded Provenance Actions
xml_data_extractor-0.1.0.tar.gz 24.9 kB Without provenance Details for xml_data_extractor-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for xml-data-extractor 0.1.0
File Interpreter ABI Platform Actions
xml_data_extractor-0.1.0-py3-none-any.whl Python 3 none any Details for xml_data_extractor-0.1.0-py3-none-any.whl

Total release size: 43.1 kB

Release files / xml_data_extractor-0.1.0.tar.gz

Download URL xml_data_extractor-0.1.0.tar.gz
Size 24.9 kB
Tags Source
SHA-256 checksum
How to use checksums
c1c84367f15a898c818e9e853a3ae162cc665a4e49af8fee9780ea6411096657
BLAKE2b-256 checksum
How to use checksums
c169d278bf62b1fec9153ec363f880bee892ebcde23b183b047056c683835155
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.0

Release files / xml_data_extractor-0.1.0-py3-none-any.whl

Download URL xml_data_extractor-0.1.0-py3-none-any.whl
Size 18.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a1dbcecb5645037bdcd8d4511a9f5f94e5e92648d3feff2acb9d472bfa9a93a7
BLAKE2b-256 checksum
How to use checksums
5ccc891b85a85323344ab4dd83d7f4dd9a715cc805bfc74af774ee1784de380a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.0

Security for xml-data-extractor 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page