eurlex-builder

Build research-ready datasets from EU legislative data — recitals, articles, points, and the network between them.

eurlex-builder is a configurable Python pipeline that turns the EUR-Lex / Cellar corpus into provision-level Parquet tables for quantitative research. It fetches every directive, regulation, decision, or communication you ask for, extracts text at the granularity you need (whole article, numbered paragraph, or lettered point), collects metadata + inter-document relations, and ships the result as one DuckDB file plus four Parquet tables.

It is, to our knowledge, the first open-source tool to extract EU legislative text at configurable sub-article granularity across EUR-Lex's heterogeneous document formats — from modern HTML to 1970s text-only pages to scanned PDFs.

The package accompanies Seidl and Kosti (2026), "Mapping Europe's Digital Acquis: A Granular History of EU Digital Policymaking" (preprint forthcoming on SocArXiv).

Highlights

• Sub-article granularity. One config switch chooses whether each article is one row, one row per numbered paragraph (Art. 5(1)), or one row per lettered point (Art. 5(1)(a)) — following the EU Joint Practical Guide.
• Six HTML structures + PDF fallback. Automatic detection across EUR-Lex's HTML eras (Standard OJ, Manual CSS, class-based, text-only, consolidated-norm, classless) with Docling/pymupdf for older docs that have no machine-readable HTML.
• Two query modes. Fixed (give us a list of CELEX IDs / procedure numbers) or descriptive (date range + doc types + optional EuroVoc keyword filter).
• Inter-document relations. Citations, amendments, legal basis, repeals, consolidations — all in one table for network analysis.
• Guarded translation built in. Non-English-only documents are translated via pinned Helsinki-NLP Opus-MT revisions, separately at document level (works.full_text) and per-unit (text_units.text_translated). Token-bounded retries and quality guards reject degenerate output without replacing the source text.
• Reproducible + resumable. A single YAML defines the dataset; a checkpoint table in DuckDB makes the pipeline restart-safe and incremental.
• Auditable runs. DuckDB stores every validated config, its SHA-256 hash, runtime versions, and completion status; validate checks structural integrity without modifying the database.
• Parallel mode. Multi-threaded fetching with parallel: true; sequential writes keep DuckDB contention-free.

Full-text retrieval works for every document type; granular extraction does not. Whatever the pipeline reaches, works.full_text is fetched and stored — the caveats here concern the decomposition into text units. Regulations, directives, and decisions are validated against official documents and are what the accompanying paper is built on. Communications extract well but have known gaps; proposals (PC) and staff working documents (SC) lose units to modern templates, and case law (sector 6) yields zero text units — silently (and is reachable only in fixed mode). Read Verification status before running anything outside the validated three.

Quick start

pip install "eurlex-builder[all]"

Create a config.yaml. You can either request specific acts by CELEX ID (fixed mode) or search by date range and document type (descriptive mode):

Fixed mode — specific acts:

metadata:
  project_name: "GDPR + AI Act"

data:
  mode: "fixed"
  celex_ids:
    - "32016R0679"    # GDPR
    - "32024R1689"    # AI Act

processing:
  text_extraction:
    article_granularity: paragraph

output:
  output_directory: "./output"

Descriptive mode — all acts matching a search:

metadata:
  project_name: "EU digital regulations 2020–2025"

data:
  mode: "descriptive"
  document_types:
    - "regulation"
    - "directive"
  start_date: 2020-01-01
  end_date: 2025-12-31
  filter_keywords:           # optional EuroVoc filter
    - "artificial intelligence"
    - "data protection"

processing:
  text_extraction:
    article_granularity: paragraph
  parallel: true
  max_workers: 8

output:
  output_directory: "./output"

Run:

eurlex-builder run config.yaml

You'll get output/eurlex_builder.duckdb plus four Parquet files: works.parquet, text_units.parquet, relations.parquet, eurovoc.parquet. The database filename is always eurlex_builder.duckdb — it's the output_directory that separates projects, so give each config its own (e.g. ./output/digital-regs).

import polars as pl
text_units = pl.read_parquet("output/text_units.parquet")

# All paragraph-1 rows of GDPR Article 5
text_units.filter(
    (pl.col("celex_id") == "32016R0679")
    & (pl.col("type") == "article")
    & (pl.col("number") == "5")
)

Why eurlex-builder

Installation

Requires Python 3.11+.

# Core only (HTML extraction + parquet output)
pip install eurlex-builder

# With PDF support (Docling + pymupdf)
pip install "eurlex-builder[pdf]"

# With translation (Opus-MT via Hugging Face)
pip install "eurlex-builder[translate]"

# Everything (recommended for production runs)
pip install "eurlex-builder[all]"

For development, install from a checkout instead:

git clone https://github.com/tseidl/eurlex-builder.git
cd eurlex-builder
python -m venv .venv && source .venv/bin/activate

# Core only (HTML extraction + parquet output)
pip install -e .

# With PDF support (Docling + pymupdf)
pip install -e ".[pdf]"

# With translation (Opus-MT via Hugging Face)
pip install -e ".[translate]"

# Everything (recommended for production runs)
pip install -e ".[all]"

# Plus dev tools (pytest)
pip install -e ".[dev]"

Configuration reference

A single YAML file defines the dataset. Only the data section is required; everything else has sensible defaults. Unknown keys and malformed CELEX IDs are rejected rather than silently ignored.

data:
  mode: "fixed"
  celex_ids:
    - "32016R0679"
  procedure_numbers:
    - "2021/0106"      # resolved to CELEX IDs via SPARQL

data:
  mode: "descriptive"
  document_types:
    - "regulation"
    - "directive"
    - "decision"
    - "communication"
  start_date: 1979-01-01
  end_date: 2026-04-30
  filter_keywords:
    - "artificial intelligence"
    - "digital"
  include_corrigenda: false
  include_consolidated_texts: false

Granularity

EU drafting convention recognises three operative levels (Joint Practical Guide of the Council, Commission, and Parliament):

Article 5
  Paragraph 5(1)
    Point 5(1)(a)
    Point 5(1)(b)

eurlex-builder lets you emit each as a row:

A short preamble before paragraph 1. gets its own row with paragraph_num = "0". Articles in amending acts (lists of edits to other regulations) are flagged subtype = 'amendment_item'; substantive replacement text within them stays untagged so downstream classifiers can keep it.

Recitals are emitted identically across all granularity settings.

For legacy HTML, numbered table rows are accepted as recitals only when they form a credible drafting sequence. Nested numbered tables remain inside their parent recital, embedded sibling lists are excluded only when the main sequence resumes, and a table continuation is joined to preceding paragraph recitals only when every earlier number is present. For legacy PDFs, a page-break fragment such as (1) of Regulation ... remains part of the active recital instead of starting a false numbered sequence.

CLI commands

eurlex-builder run <config.yaml>

Run the pipeline.

The pipeline resumes by default, skipping checkpointed docs. After an interrupted --fresh rebuild, re-run without --fresh to resume the new rebuild rather than clearing its completed checkpoints again. --limit is intended for canary runs and cannot be combined with --fresh.

If any document attempted in this invocation fails, the command exits with status 1 after saving and exporting successful results. Its run manifest records complete_with_failures. Historical failures outside the attempted selection do not affect this exit status. Use --retry-failed to retry failures. A Python call to Pipeline.run() returns RunResult(processed, failed) with counts for the invocation; discovery and other fatal errors still raise.

Keyword filtering stops with an error if no EuroVoc concepts match or all matches are rejected. Set filter_keywords: [] explicitly for an unfiltered date/type search. Failed selection leaves existing checkpoints intact.

eurlex-builder translate <db>

Translate non-English content. Resumable; skips already-translated rows.

Translation is interruption-safe at row level. Rejected outputs remain in the source language and are recorded in _translation_failures; an ordinary resume skips unchanged current-policy rejections, while --retry-rejected tries them again. A source-text or policy-version change invalidates the corresponding ledger entry automatically.

eurlex-builder enrich <db>

Add post-hoc metadata via SPARQL — no re-fetching content. Adds entry-into-force date, ELI, author institutions, subject matter, procedure type/reference/legal basis, EuroVoc descriptors, and repeal relations.

Completion is tracked separately for metadata, relations, and eurovoc, so categories can be run independently and resumed later. A successful response includes category completion sentinels; only then can an empty refresh remove stale values. Databases enriched by older releases are migrated to all-category checkpoints from enriched_at instead of unexpectedly re-fetching every work.

eurlex-builder status <db>

Print checkpoint summary (processed / failed counts + failure reasons).

eurlex-builder validate <db>

Run read-only integrity checks for checkpoint/work consistency, orphaned rows, stable unit identity, duplicate keys/order, translated-fallback markers, and degenerate stored translations. The command exits non-zero when an error is found and reports expected data gaps and recorded translation rejections as warnings.

Output schema

FAQ

eurlex-builder multilingual output/eurlex_builder.duckdb --languages eng deu fra

Verification status

The configuration accepts six document types plus raw CELEX IDs, but they are not equally validated. A smoke test (July 2026, v0.1.0, 17 documents spanning 1962–2021; full report in docs/doc-type-verification.md) established:

eurlex-builder validate flags the silent-failure cases: a work with stored full text but zero text units is an error (exit 1), and a document type outside the supported extraction set is a warning; the pipeline also logs a warning at run time when a document has no extractor branch. What validation still cannot measure is extraction coverage — a database whose modern proposals silently lost their recitals to unhandled CSS classes passes, because no source-text baseline is stored. Do not read a passing validation as evidence that an unverified document type extracted completely.

Architecture

config.yaml (Pydantic-validated)
  ├─ EuroVoc keyword resolution (SPARQL, optional interactive review)
  ├─ Procedure number → CELEX resolution (SPARQL)
  ├─ CELEX ID discovery (SPARQL descriptive query)
  ├─ Per-document processing (parallel or sequential):
  │     metadata fetch         — SPARQL: title, date, relations
  │     content fetch           — REST: XHTML / HTML / PDF (six-language fallback)
  │     text extraction         — lxml: 6 HTML structures + paragraph splitting
  │                               PDF: persistent isolated Docling workers, with
  │                               per-document hard timeouts and pymupdf fallback
  │     translate-before-extract — Opus-MT fallback when a non-English legislative
  │                                PDF misses requested structures (English-only
  │                                markers like "Whereas:" wouldn't fire on a French
  │                                or German PDF). Translates the Docling markdown
  │                                and re-parses from English.
  │     storage                 — DuckDB: works, text_units, relations, checkpoint
  ├─ Translation                — Opus-MT, sequential post-processing
  ├─ Enrichment                 — SPARQL: dates, ELI, procedure, EuroVoc, repeals
  ├─ Export                     — Polars → Parquet/CSV
  └─ Reports                    — missing-content TSV, extraction stats

More on how the six HTML structures are detected and parsed: extraction approach.

All data sourced through official EU APIs:

• SPARQL: https://publications.europa.eu/webapi/rdf/sparql
• REST: https://publications.europa.eu/resource/celex/

No scraping.

Citation

If you use this package, please cite the accompanying paper and the software:

@article{seidl_kosti_2026,
  author  = {Seidl, Timo and Kosti, Nir},
  title   = {Mapping Europe's Digital Acquis: A Granular History of EU Digital Policymaking},
  year    = {2026},
  note    = {Working paper, preprint forthcoming on SocArXiv}
}

@software{eurlex_builder,
  author  = {Seidl, Timo},
  title   = {eurlex-builder: a configurable Python pipeline for EU legislative datasets},
  year    = {2026},
  url     = {https://github.com/tseidl/eurlex-builder},
  doi     = {10.5281/zenodo.21496963}
}

The DOI above is the Zenodo concept DOI, which always points to the latest release. For reproducibility, cite the version DOI of the release you actually used (listed on the Zenodo record) and state the version number. See also CITATION.cff.

Authors

• Timo Seidl — Assistant Professor, Technical University of Munich
• Claude (Anthropic) — Co-author (software design and implementation). Built with Claude Code.

Acknowledgments

• Sebastian Rein (eulex-build) for the initial impetus, the YAML-driven configuration architecture, the fixed-vs-descriptive query mode design, EuroVoc keyword filtering, and the structural-decomposition target schema (recitals / articles / annexes + inter-document relations). The package builds on the foundation of his MA thesis on EU legislative data extraction (TUM, 2026).
• The maintainers of EUR-Lex / Cellar, Docling, Helsinki-NLP Opus-MT, DuckDB, and Polars.

Roadmap

• Discovery by resource-type, not just CELEX-type. Today's descriptive mode filters by CELEX type code (D, R, L) + sector. EUR-Lex also exposes work_has_resource-type URIs (DEC, DEC_IMPL, DEC_DEL, REG_FINANC, …) which form a semantic classification overlapping but not identical to the CELEX letter. Adding type_basis: celex | resource_type | both to the YAML — with explicit per-doc-type resource_types lists — would let researchers opt into broader sets (e.g. merger decisions with CELEX-type M, budget decisions with B, framework decisions, joint decisions). Default stays CELEX so existing configs reproduce the same corpus.
• Modern COM-template support. The July 2026 smoke test (see Verification status) found that current proposal and impact-assessment templates use CSS classes the COM extractor skips (li ManualConsidrant recitals, Titrearticle article headings, li Point*, li Numbered-Para), and that HTTP-300 multipart representations are truncated to the first stream (annexes and part-2 files are never fetched). Fixing both would make DSA/AI-Act-era proposals and impact assessments extractable.
• Case-law extraction (sector 6). CJ/CC/CO documents currently map to unknown and yield zero text units. Judgments have their own stable structure (grounds, decision on costs, operative part) that a dedicated extractor could target; descriptive mode would also need case-law entries in the document_types mapping.
• Dataset linkage layer. Left-join helpers to enrich our works table with EUPROPS (manually curated text resource), EUPLEX (complexity indicators), and EUPOL (policy domain coding) via CELEX ID — combining their derived columns with our structured text for the same acts.
• Pittsburgh Archive fallback (Archive of European Integration) as a secondary content source for documents that EUR-Lex cannot serve. The Pittsburgh archive holds digitised early-period European Community materials (1950s–1990s) that occasionally fill EUR-Lex gaps.
• Gated OCR/VLM fallback for the residual failure set. Keep OCR disabled by default and consider it only when official HTML is absent or low-information and Docling/PyMuPDF fails to recover the requested structure. Start with a bounded canary, record model/version/input/output hashes and cost, emit faithful Markdown, and accept it only when the existing deterministic parser gains a requested structure without regressions. Benchmark managed Mistral OCR against local GLM-OCR before selecting a backend.
• Incremental update mode — delta runs that fetch only acts adopted since the last completed run.

License

MIT