bnetza_bk6_scraper
bnetza_bk6_scraper mirrors the documents published by the German
Bundesnetzagentur (BNetzA) Beschlusskammer 6 (BK6) into a structured,
git-diffable directory tree. BK6 regulates electricity network access and is a
constant source of consultations, rulings (Festlegungen) and their attachments.
Because the agency publishes these as loose PDFs on HTML pages with no changelog,
tracking what changed and when is painful. This tool discovers every BK6
proceeding, downloads its PDFs and a normalized HTML snapshot of each phase page,
and records structured metadata. Committing the output to git turns every
regulatory update into a reviewable diff.
Installation
pip install bnetza_bk6_scraper
Usage
The package installs a single console command, bnetza-bk6-scraper, with a
mirror subcommand:
bnetza-bk6-scraper mirror --target <dir> [--concurrency N] [--year YYYY] [-v]
| Option | Default | Description |
|---|---|---|
--target |
(required) | Output directory (the mirror repository root). |
--concurrency |
4 |
Number of parallel HTTP fetches. |
--year |
(all) | Restrict the run to a single year, e.g. 2023. |
-v, --verbose |
off | Enable debug logging. |
Example — mirror only the 2023 proceedings into ./mirror:
bnetza-bk6-scraper mirror --target ./mirror --year 2023 -v
Each run logs a summary such as
run summary: 7 proceedings, 16 documents written, 0 failures.
Scoped mirroring (Python API)
The mirror CLI mirrors all proceedings. To mirror only a curated subset (e.g. the
electricity GPKE/WiM/MaBiS Prozessdokumente), use the Python API: give the scraper a list of
seed pages to crawl and a list of predicates (OR semantics — a document is downloaded
if any predicate returns True). Predicates receive a CandidateDocument.
import asyncio
from bnetza_bk6_scraper import BnetzaBk6Scraper, CandidateDocument
GPKE = "https://www.bundesnetzagentur.de/DE/Beschlusskammern/BK06/BK6_83_Zug_Mess/831_gpke/gpke_node.html"
def is_prozessdokument_lesefassung(c: CandidateDocument) -> bool:
name = c.filename.lower()
return any(fw in name for fw in ("_gpke_", "_wim_", "_mabis_")) and "lesefassung" in name
asyncio.run(
BnetzaBk6Scraper().mirror_seeds(
target_dir=".", seeds=[GPKE], keep=[is_prozessdokument_lesefassung]
)
)
Documents that carry an Aktenzeichen are written under <year>/<aktenzeichen>/; documents
without one (e.g. the PID-Liste in the Datenformate tree) go under _other/…. A root
manifest.json lists the kept documents.
Link detection is not limited to PDFs: .pdf, .xlsx and .xls are all mirrored. BNetzA
publishes the Liste der Prüfidentifikatoren (PID) as both a PDF and a machine-friendly
Excel file — the Excel is captured too, so a predicate like the one above picks up
PID_..._info_....xlsx right next to its PDF.
Output layout
Proceedings are written under /{year}/{aktenzeichen}/, with a top-level
index.json listing every mirrored proceeding:
<target>/
├── index.json # summary of all proceedings
└── 2023/
└── BK6-23-241/
├── metadata.json # structured proceeding metadata
├── BK6-23-241_beschluss.html # normalized HTML snapshot of a phase page
├── BK6-23-241_beschluss_vom_07.05.26.pdf
├── BK6-23-241_bilarem.pdf
└── BK6-23-241_anlage_bilarem.pdf
metadata.jsoncaptures the Aktenzeichen, year, title, status,Stand(last-modified date), any submission deadline (Frist), the phase pages, and one entry per document (title, type, source URL, filename).- The normalized
*.htmlfiles are trimmed, stable snapshots of the source phase pages so that content changes surface as small diffs. - The PDFs are the proceeding's documents, downloaded verbatim.
Change detection is intentionally "dumb": the tool always writes the current
state, and git diff in the mirror repository reveals what changed.
Mirror repository
The scraper is designed to feed a separate mirror repository,
Hochfrequenz/bnetza_bk6_mirror. A scheduled GitHub Action there will
periodically:
pip install bnetza_bk6_scraper
bnetza-bk6-scraper mirror --target .
git add -A && git commit -m "update BK6 mirror"
so that regulatory changes at BK6 become visible as reviewable git diffs and commit history. That Action is future work and does not live in this repository.
WAF / browser User-Agent
The BNetzA website sits behind a Web Application Firewall that rejects
non-browser clients by serving a 200 OK "The requested URL was rejected" page
instead of the real content. To get through, the scraper sends browser-like
User-Agent and Accept headers and treats the rejection page as a retryable
error. No credentials or API keys are required.
Contribute
This project uses uv for all quality gates. Create a one-shot development environment with everything installed:
uv sync --group dev
Individual gates: uv run --group tests pytest, uv run --group linting ruff check .,
uv run --group linting ruff format --check ., uv run --group type_check mypy --strict src/bnetza_bk6_scraper,
uv run --group coverage coverage run -m pytest, and uv run --group spell_check codespell src.
Metadata
Release files for bnetza-bk6-scraper 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| bnetza_bk6_scraper-0.1.2.tar.gz | 173.0 kB | With provenance | Details for bnetza_bk6_scraper-0.1.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| bnetza_bk6_scraper-0.1.2-py3-none-any.whl | Python 3 | none | any | Details for bnetza_bk6_scraper-0.1.2-py3-none-any.whl |
Total release size: 190.8 kB
Release files / bnetza_bk6_scraper-0.1.2.tar.gz
| Download URL | bnetza_bk6_scraper-0.1.2.tar.gz |
|---|---|
| Size | 173.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b7b2cd22488b28140c5a339f28c60ede65671957402a5fb06f001f88125bc5d1
|
|
BLAKE2b-256 checksum How to use checksums |
e0663de25329389159563c4e787c6262c070dc4f0c283cda3dc23dc3945c5d1f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.
Transparency logRelease files / bnetza_bk6_scraper-0.1.2-py3-none-any.whl
| Download URL | bnetza_bk6_scraper-0.1.2-py3-none-any.whl |
|---|---|
| Size | 17.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2297acfa02b11000468797b08064e2025d20d3a69e8742c27549bb08b75356a4
|
|
BLAKE2b-256 checksum How to use checksums |
b4d7f06c276d5fb3fdfb2d87597741a81e5942bf543671da5be78e10d58bc33b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.
Transparency logSecurity for bnetza-bk6-scraper 0.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for bnetza_bk6_scraper-0.1.2.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.