ddigraph

A modern Python toolkit that transforms DDI (Data Documentation Initiative) XML metadata into knowledge graphs. Supports DDI Codebook and DDI-L FragmentInstance formats with streaming parsing, batched writes, and full async I/O across multiple graph backends.

Documentation | Getting Started | PyPI | Source Code

Features

• Neo4j and RDF -- load into Neo4j, or read and write RDF with a documented, versioned vocabulary aligned to DISCO, SKOS and XKOS
• File export -- ddigraph export writes Turtle, JSON-LD, N-Triples, RDF/XML, JSON or CSV with no database involved
• SHACL shapes -- generated from the same schema that builds the Neo4j constraints, so they cannot drift
• Preview before you load -- ddigraph preview summarises a file as text, Mermaid, or one self-contained HTML page
• Streaming XML processing -- Memory-bounded iterparse for files of any size
• Batched writes -- UNWIND-based Cypher for 10-100x fewer database round trips
• Async I/O -- Concurrent parsing and writing with back-pressure control
• Format auto-detection -- identifies DDI Codebook, DDI-L and DDI-CDI input
• Unified schema -- Single source of truth for all node and relationship definitions
• Adapter pattern -- Plug in custom graph backends via GraphWriteAdapter protocol
• Production-ready -- Retry logic, observability hooks, pydantic-based configuration

Quick Start

Install

pip install ddigraph

Load DDI metadata (CLI)

# Set Neo4j connection
export DDIGRAPH_NEO4J_URI=bolt://localhost:7687
export DDIGRAPH_NEO4J_USER=neo4j
export DDIGRAPH_NEO4J_PASSWORD=secret

# Bootstrap schema and load data (format is auto-detected)
ddigraph bootstrap
ddigraph load survey.xml --dataset-id my-survey

Load DDI metadata (Python)

import ddigraph

ddigraph.bootstrap(target="bolt://localhost:7687")
result = ddigraph.load("survey.xml", target="bolt://localhost:7687")
print(result.nodes_written, "nodes,", result.relationships_written, "relationships")

Export without a database

import ddigraph

result = ddigraph.export("survey.xml", "survey.ttl", format="turtle")
print(result.triples, "triples")

Supported Formats

XSD Coverage

ddigraph ships with 100 % coverage of every concrete identifiable element declared in the bundled XSD schemas (schemas/). Coverage is enforced by the audit script and a pytest guardrail so new schema releases surface any gaps:

Run python scripts/xsd_coverage.py to regenerate the audit or python scripts/xsd_coverage.py --json for machine-readable output.

Supported Backends

Every backend reads DDI XML of all three flavors, or an RDF export:

from ddigraph.backends.networkx import to_networkx
from ddigraph.backends.pandas import to_dataframes

G = to_networkx("survey.xml")
frames = to_dataframes("survey.ttl")

To feed a store ddigraph does not cover, use ddigraph.iter_graph(), which yields backend-neutral nodes and relationships for any DDI flavor.

Docker Quick Start

docker run --rm --name neo4j-demo \
  -p 7474:7474 -p 7687:7687 \
  -e NEO4J_AUTH=neo4j/password \
  neo4j:5

export DDIGRAPH_NEO4J_URI=bolt://localhost:7687
export DDIGRAPH_NEO4J_USER=neo4j
export DDIGRAPH_NEO4J_PASSWORD=password

ddigraph bootstrap
ddigraph load your-file.xml --dataset-id demo

Documentation

Full documentation is available at pbisson44.github.io/ddigraph in English and French.

• Getting Started -- Installation, quick start, 10-minute tutorial
• User Guide -- Architecture, DDI formats, relationships, adapters
• Graph Backends -- Neo4j, RDF/SPARQL, Gremlin, NetworkX
• Reference -- CLI commands, configuration
• Advanced -- Performance tuning, AI readiness, standards interoperability
• Contributing -- How to contribute

Development

git clone https://github.com/pbisson44/ddigraph.git
cd ddigraph
pip install -e ".[dev,docs]"

ruff check . && ruff format .
mypy .
pytest
mkdocs serve

License

MIT -- see LICENSE for details.