ddigraph 0.4.0

DDI to Knowledge Graph toolkit - transform DDI metadata into graph databases (Neo4j, RDF, Gremlin, NetworkX)

ddigraph

A modern Python toolkit that transforms DDI (Data Documentation Initiative) XML metadata into knowledge graphs. Supports DDI Codebook and DDI-L FragmentInstance formats with streaming parsing, batched writes, and full async I/O across multiple graph backends.

Documentation | Getting Started | PyPI | Source Code

---

Features

• Multi-backend support -- Neo4j, RDF/SPARQL, Gremlin, NetworkX, and pandas
• Streaming XML processing -- Memory-bounded iterparse for files of any size
• Batched writes -- UNWIND-based Cypher for 10-100x fewer database round trips
• Async I/O -- Concurrent parsing and writing with back-pressure control
• Format auto-detection -- Automatically identifies DDI Codebook vs Lifecycle format
• Unified schema -- Single source of truth for all node and relationship definitions
• Adapter pattern -- Plug in custom graph backends via GraphWriteAdapter protocol
• Production-ready -- Retry logic, observability hooks, pydantic-based configuration

Quick Start

Install

pip install ddigraph

Load DDI metadata (CLI)

# Set Neo4j connection
export DDIGRAPH_NEO4J_URI=bolt://localhost:7687
export DDIGRAPH_NEO4J_USER=neo4j
export DDIGRAPH_NEO4J_PASSWORD=secret

# Bootstrap schema and load data (format is auto-detected)
ddigraph bootstrap
ddigraph load survey.xml --dataset-id my-survey

Load DDI metadata (Python)

import asyncio
from neo4j import AsyncGraphDatabase
from ddigraph import DDILoader, DDIFragmentLoader, detect_ddi_format
from ddigraph.config import Settings

async def main():
    settings = Settings()
    driver = AsyncGraphDatabase.driver(
        settings.neo4j_uri,
        auth=(settings.neo4j_user, settings.neo4j_password.get_secret_value()),
    )
    path = "survey.xml"
    if detect_ddi_format(path) == "lifecycle":
        loader = DDIFragmentLoader(driver, settings=settings)
        result = await loader.load(path)
    else:
        loader = DDILoader(driver, settings=settings)
        result = await loader.load(path, dataset_id="my-survey")
    print(result)  # {'Instrument': 1, 'Sequence': 388, 'QuestionItem': 373, ...}
    await driver.close()

asyncio.run(main())

Supported Formats

Format	Description	Use Case
DDI Codebook	Traditional flat format with central Dataset node	Survey archives, data catalogs
DDI-L FragmentInstance	Lifecycle 3.x format with reusable fragments	Questionnaire design, CAPI/CAWI instruments
DDI-CDI 1.0	Cross-Domain Integration metadata	Data integration, statistical production

XSD Coverage

ddigraph ships with 100 % coverage of every concrete identifiable element declared in the bundled XSD schemas (schemas/). Coverage is enforced by the audit script and a pytest guardrail so new schema releases surface any gaps:

Flavor	Scope	Target	Covered
DDI-L 3.x	Concrete Maintainable + Versionable + Identifiable elements	189	100 %
DDI-C 2.x	Codebook elements with the GLOBALS attribute group (no layout tags)	73	100 %
DDI-CDI 1.0	Concrete top-level entity elements (associations excluded)	210	100 %

Run python scripts/xsd_coverage.py to regenerate the audit or python scripts/xsd_coverage.py --json for machine-readable output.

Supported Backends

Backend	Description	Use Case
Neo4j	Native graph database (Bolt)	Production deployments, complex queries
RDF/SPARQL	Semantic web triplestores	Linked data, ontology integration
Gremlin	Graph traversal language	JanusGraph, Neptune, Cosmos DB
NetworkX	Python graph library	Local analysis, prototyping
pandas	DataFrame-based	Tabular analysis, Excel export

Docker Quick Start

docker run --rm --name neo4j-demo \
  -p 7474:7474 -p 7687:7687 \
  -e NEO4J_AUTH=neo4j/password \
  neo4j:5

export DDIGRAPH_NEO4J_URI=bolt://localhost:7687
export DDIGRAPH_NEO4J_USER=neo4j
export DDIGRAPH_NEO4J_PASSWORD=password

ddigraph bootstrap
ddigraph load your-file.xml --dataset-id demo

Documentation

Full documentation is available at pbisson44.github.io/ddigraph in English and French.

• Getting Started -- Installation, quick start, 10-minute tutorial
• User Guide -- Architecture, DDI formats, relationships, adapters
• Graph Backends -- Neo4j, RDF/SPARQL, Gremlin, NetworkX
• Reference -- CLI commands, configuration
• Advanced -- Performance tuning, AI readiness, standards interoperability
• Contributing -- How to contribute

Development

git clone https://github.com/pbisson44/ddigraph.git
cd ddigraph
pip install -e ".[dev,docs]"

ruff check . && ruff format .
# Docstring linting is currently enforced for src/ddigraph only.
pydocstyle src/ddigraph
mypy .
pytest
mkdocs serve

License

MIT -- see LICENSE for details.