Scraper-core
Core library for collectors/scrapers of PaZuFa, providing shared functionality and base classes.
Status: Work in Progress (WIP). For a detailed status overview, see the status page in the wiki (German).
The detailed documentation can be found in the wiki (German).
Requests
If you have a request for the Scraper-core, the best way to voice it is to write a Codeberg issue. Please add the label external-request to it.
If it is a bug you can alternatively use the label Bug.
For requests and questions, you can, of course, contact us on Mattermost.
Structure
The library consists of three parts:
- CoreLib: shared classes and utilities used by all scrapers regardless of implementation approach. This includes Pydantic validation models, API client helpers, common data transformation functions, standardised phrases and tag mappings (e.g. normalising committee names, document types, and Schlagworte across parliaments), and reusable components for tasks like LLM enrichment.
- Scrapy-based: Opinionated implementation of Corelib in Scrapy based classes.
- Collector-based: Our project's scaffolding for scrapers, implemented in an opinionated manner using Corelib.
Requirements
- Python 3.12+
- Poetry 2.x
For the full dependency list, see pyproject.toml.
Setup
See SETUP.md for the full setup guide, which is versioned alongside the code.
Contribution
See CONTRIBUTING.md for development setup, git workflow, code generation, documentation and project context.
License
GPL-3.0
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file pazufa_corelib-0.2.1.tar.gz.
File metadata
- Download URL: pazufa_corelib-0.2.1.tar.gz
- Upload date:
- Size: 127.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.12.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
07ef73b6077bb02a4900b36bd85f3ddef00ec682cae4eda9e77f71bed0ed9645
|
|
| MD5 |
5f95aef8f24fcdcbf19439c774ea215f
|
|
| BLAKE2b-256 |
54747411289ee71c599fd0900473b746c5540fa81ae054143da307039b254664
|
File details
Details for the file pazufa_corelib-0.2.1-py3-none-any.whl.
File metadata
- Download URL: pazufa_corelib-0.2.1-py3-none-any.whl
- Upload date:
- Size: 206.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.12.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
930bf4470b9b17792eee5913103f6d7f97da8e41374f5503506fe03ed940252e
|
|
| MD5 |
d58439ac73d4ca0a171d371503b194aa
|
|
| BLAKE2b-256 |
0d824668f4dc2470fb005622deccb082d349797eaa8ebbe7fa0acb46cb860511
|
