oldp-ingestor

Ingesting legal data like laws and court decisions via OLDP API.

Data sources:

Installation

pip install oldp-ingestor

Some providers require Playwright browsers. Install them after pip:

playwright install chromium

For development, clone the repo and use Make (auto-detects uv or falls back to pip):

git clone https://github.com/openlegaldata/oldp-ingestor.git
cd oldp-ingestor
make install

Configuration

Set the following environment variables (or add them to a .env file):

Usage

Show API info

oldp-ingestor info

Ingest laws

From the RIS API (rechtsinformationen.bund.de)

# Ingest all available legislation
oldp-ingestor laws --provider ris

# Search for specific legislation
oldp-ingestor laws --provider ris --search-term "EinbTestV"

# Limit the number of law books to ingest
oldp-ingestor laws --provider ris --limit 5

# Combine search and limit
oldp-ingestor laws --provider ris --search-term "BGB" --limit 1

Incremental fetching and request pacing

# Only fetch legislation adopted since a given date
oldp-ingestor laws --provider ris --date-from 2025-12-01

# Fetch legislation within a date range
oldp-ingestor laws --provider ris --date-from 2025-01-01 --date-to 2025-06-30

# Override the default request delay (0.2s) for slower pacing
oldp-ingestor laws --provider ris --request-delay 0.5

From gesetze-im-internet.de (gii)

Pulls all German federal laws from the official BMJ/juris feed. See docs/providers/de/gii.md for usage, incremental-run setup, and continuous-ingestion examples.

From a JSON fixture file (dummy provider)

oldp-ingestor laws --provider dummy --path /path/to/fixture.json

Ingest cases

From the RIS API (rechtsinformationen.bund.de)

# Ingest all cases from all federal courts
oldp-ingestor cases --provider ris

# Filter by court and date range
oldp-ingestor cases --provider ris --court BGH --date-from 2026-01-01

# Limit for testing
oldp-ingestor cases --provider ris --limit 10 -v

From a JSON fixture file (dummy provider)

oldp-ingestor cases --provider dummy --path /path/to/fixture.json

# Limit the number of cases to ingest
oldp-ingestor cases --provider dummy --path /path/to/fixture.json --limit 10

The fixture file should contain Django fixture entries with courts.court and cases.case models. Court foreign keys are resolved to court_name strings for the OLDP cases API.

Targeted citation-based lookup

When OLDP can't resolve an extracted citation, an AI agent can use the lookup subcommand group to fetch the specific decision from the right upstream portal. Each call is a single upstream request returning JSON with a three-status contract (ok / not_found / error).

# What lookup-capable providers exist + which courts they cover?
oldp-ingestor lookup providers

# Search RIS by ECLI
oldp-ingestor lookup search --provider ris \
    --ecli "ECLI:DE:BGH:2026:020626BVIAZR482.23.0"

# Search a juris state portal by Aktenzeichen
oldp-ingestor lookup search --provider juris-rlp --file-number "5 T 16/26"

# Fetch the full case (no OLDP write)
oldp-ingestor lookup fetch --provider ris --doc-id KORE615402026

# Fetch + POST to OLDP (idempotent: 409 → already_exists)
oldp-ingestor lookup ingest --provider ris --doc-id KORE615402026

See docs/lookup.md for the full agent loop, JSON schema, and per-provider capability matrix.

Output sinks

By default, data is written to the OLDP REST API. Use --sink json-file to write JSON files to disk instead:

# Export laws to local files
oldp-ingestor --sink json-file --output-dir /tmp/export \
    laws --provider ris --search-term BGB --limit 1

# Export cases to local files
oldp-ingestor --sink json-file --output-dir /tmp/export \
    cases --provider ris --court BGH --limit 5

See docs/sinks.md for details on directory structure and implementing custom sinks.

Architecture

The ingestor uses a provider-based architecture. Each data source implements a provider class (LawProvider or CaseProvider), and shared RIS HTTP logic (retry, pacing, User-Agent) lives in RISBaseClient. Output is routed through a sink (ApiSink or JSONFileSink).

Provider
├── LawProvider   →  DummyLawProvider, RISProvider
└── CaseProvider  →  DummyCaseProvider, RISCaseProvider,
                     RiiCaseProvider, ByCaseProvider,
                     NrwCaseProvider, NsCaseProvider,
                     EuCaseProvider, JurisCaseProvider (10 state variants)

Sink
├── ApiSink        →  OLDP REST API (default)
└── JSONFileSink   →  local JSON files

See docs/architecture.md for the full design.

Politeness and rate limiting

The RIS API allows 600 req/min. The ingestor stays under this with:

• Request pacing — 0.2 s delay between requests (configurable)
• Retry with backoff — exponential backoff on 429/503, respects Retry-After
• Descriptive User-Agent — oldp-ingestor/0.1.0

See docs/politeness.md for details.

Further documentation

• docs/architecture.md — class hierarchy, data flow, file layout
• docs/sinks.md — sink concept, CLI examples, custom sinks
• docs/politeness.md — rate limiting, retry logic, cron operation
• docs/lookup.md — citation-based lookup tools for AI agents

Provider docs

Detailed documentation about each provider can be found in docs/providers/.

Development

# Run tests
make test

# Run tests with coverage
make test-cov

# Lint
make lint

# Auto-format
make format

See CONTRIBUTING.md for the full development setup, how to add new providers, and pull request guidelines.