pazufa-corelib 0.1.0

Core library for collecting parliamentary data

Scraper-core

Core library for collectors/scrapers of PaZuFa, providing shared functionality and base classes.

Status: Work in Progress (WIP). For a detailed status overview, see the status page in the wiki (German).

The detailed documentation can be found in the wiki (German).

Requests

If you have a request for the Scraper-core, the best way to voice it is to write a Codeberg issue. Please add the label external-request to it.

If it is a bug you can alternatively use the label Bug.

For requests and questions, you can, of course, contact us on Mattermost.

Structure

The library consists of three parts:

1. CoreLib: shared classes and utilities used by all scrapers regardless of implementation approach. This includes Pydantic validation models, API client helpers, common data transformation functions, standardised phrases and tag mappings (e.g. normalising committee names, document types, and Schlagworte across parliaments), and reusable components for tasks like LLM enrichment.
2. Scrapy-based: Opinionated implementation of Corelib in Scrapy based classes.
3. Collector-based: Our project's scaffolding for scrapers, implemented in an opinionated manner using Corelib.

Requirements

• Python 3.12+
• Poetry 2.x

For the full dependency list, see pyproject.toml.

Setup

See SETUP.md for the full setup guide, which is versioned alongside the code.

Contribution

See CONTRIBUTING.md for development setup, git workflow, code generation, documentation and project context.

License

GPL-3.0