presscorner-builder

Build and maintain research-ready datasets from the European Commission Press Corner — every press release, speech, and statement since 1975, in one Parquet file.

The Press Corner is the European Commission's press release database. presscorner-builder turns it into a clean, citable, always-updatable dataset for social science research. You can download the full pre-built corpus and top it up to today with one command, or define your own sub-corpus (by date, type, keyword, commissioner, or policy area) in a small YAML file.

No web scraping knowledge required. If you can run two commands in a terminal, you can use this.

130,544 documents · 1975-03-11 to 2026-07-24 · 32 document types · EN language edition

The published dataset is refreshed every few months — and whatever its current cut-off, presscorner update brings your local copy to today in minutes.

Quick start

pip install presscorner-builder

presscorner download   # fetch the published full dataset (~460 MB)
presscorner update     # top it up from its cut-off date to today

That's it. data/press-corner.parquet now contains the complete corpus. Load it in R or Python:

library(arrow)
docs <- read_parquet("data/press-corner.parquet")

import pandas as pd
docs = pd.read_parquet("data/press-corner.parquet")

Already have a copy of the dataset from a colleague? Drop it in data/ and run presscorner update — the tool reads the file itself to see where it stops and fetches only what's newer. Older versions produced by the predecessor scraper are migrated automatically.

What's in the dataset

The Press Corner is the successor of RAPID, the Commission's press database running since the mid-1980s. The current website only advertises nine document types, but the archive behind the API still contains the full RAPID legacy — including press material from the Council, the Court of Justice, and other EU institutions, and European Council conclusions digests back to 1975. presscorner-builder collects all of it. To our knowledge this is not documented anywhere else.

Two honest caveats:

1. The legacy series ended around 2013–2015, when the other institutions launched their own newsrooms. For those institutions this is a historical archive, not ongoing coverage. The Commission's own types (IP, SPEECH, STATEMENT, MEX, QANDA, …) are current and continuously updated.
2. Rich metadata is a recent phenomenon. Policy areas, commissioner attribution, places, and subtitles were introduced with the modern content system and never backfilled. What is consistent across the whole archive is the core: reference, date, title, and full text. Plan your research design accordingly:

Why is full text not 100% in the 2020s? Almost all of the gap is factsheets (FS): these are designed as visual PDF documents, so their pages have no body text to extract. Every one of them carries a working pdf_url pointing to the actual content.

Building your own corpus

For a defined sub-corpus, write a small YAML file (presscorner init creates a template):

metadata:
  project_name: "Von der Leyen climate speeches"

data:
  mode: descriptive
  document_types: [SPEECH, STATEMENT]
  start_date: 2019-12-01
  keywords: ["climate"]

output:
  output_directory: ./output
  dataset_name: vdl-climate

presscorner build config.yaml

This produces output/vdl-climate.parquet plus a metadata sidecar recording exactly how the corpus was built (config hash, package version, run date) — share the YAML in your replication package and the corpus is fully reproducible. If you already know which documents you want, use mode: fixed with a list of reference numbers instead.

Keeping the dataset complete: audit

Scrapes fail silently: connections drop, servers hiccup, and you end up with holes you never notice. presscorner-builder treats this as a first-class problem:

• All fetching happens in calendar-month windows, so an interruption costs at most one month, and every run is resumable — failed windows and documents are remembered and retried on the next run.
• presscorner audit compares, month by month, how many documents the EC API reports against how many your local file contains, and prints any mismatch. presscorner audit --fix re-fetches the deficient months.

presscorner audit          # find holes
presscorner audit --fix    # repair them

(This machinery found and repaired ~3,700 silently missing documents in the predecessor scraper's dataset, including four entirely missing months.)

Commands

All commands take --data-dir (default ./data) and are safe to interrupt and re-run.

Data schema

One row per document.

Multi-valued fields are "; "-joined strings. By default the English edition is collected; update/build accept other language codes but the published dataset is English.

Dataset versioning and citation

The full dataset is published on Zenodo with a versioned DOI; versions are named by cut-off (v2026.07 = complete through July 2026). presscorner download always tells you which version you got. Whatever the published version, presscorner update brings your local copy to today.

If you use the dataset or the package, please cite both:

@dataset{seidl_presscorner_data,
  author    = {Seidl, Timo},
  title     = {EC Press Corner Complete Document Dataset (1975--2026)},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.21536427},
  note      = {Dataset version v2026.07}
}

@software{seidl_presscorner_builder,
  author = {Seidl, Timo},
  title  = {presscorner-builder: research-ready datasets from the EC Press Corner},
  url    = {https://github.com/tseidl/presscorner-builder},
  doi    = {10.5281/zenodo.21538765}
}

The dataset DOI above is the Zenodo concept DOI, which always resolves to the latest version. For reproducibility, cite the version DOI of the release you actually used (listed on the Zenodo record; for v2026.07 it is 10.5281/zenodo.21536428) and state the version number.

For maintainers

• Refresh cycle: every few months, run presscorner update && presscorner audit --fix, then python scripts/update-readme-stats.py, publish the new parquet as a Zenodo version, and update dataset-manifest.json (version, cut-off, URL, sha256).
• Scraping is polite by design: ≥1s request delay, honest User-Agent, exponential backoff, no parallel requests. Please keep it that way.

Authors

• Timo Seidl — Assistant Professor, Technical University of Munich
• Claude (Anthropic) — Co-author (software design and implementation). Built with Claude Code.

License

MIT. The documents themselves are © European Union — reuse is governed by the Commission's reuse policy (CC BY 4.0 for most content).