Scraper-core
Core library for collectors/scrapers of PaZuFa, providing shared functionality and base classes.
Status: Work in Progress (WIP). For a detailed status overview, see the status page in the wiki (German).
The detailed documentation can be found in the wiki (German).
Requests
If you have a request for the Scraper-core, the best way to voice it is to write a Codeberg issue. Please add the label external-request to it.
If it is a bug you can alternatively use the label Bug.
For requests and questions, you can, of course, contact us on Mattermost.
Structure
The library consists of three parts:
- CoreLib: shared classes and utilities used by all scrapers regardless of implementation approach. This includes Pydantic validation models, API client helpers, common data transformation functions, standardised phrases and tag mappings (e.g. normalising committee names, document types, and Schlagworte across parliaments), and reusable components for tasks like LLM enrichment.
- Scrapy-based: Opinionated implementation of Corelib in Scrapy based classes.
- Collector-based: Our project's scaffolding for scrapers, implemented in an opinionated manner using Corelib.
Requirements
- Python 3.12+
- Poetry 2.x
For the full dependency list, see pyproject.toml.
Setup
See SETUP.md for the full setup guide, which is versioned alongside the code.
Contribution
See CONTRIBUTING.md for development setup, git workflow, code generation, documentation and project context.
License
GPL-3.0
Metadata
Release files for pazufa-corelib 0.3.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pazufa_corelib-0.3.1.tar.gz | 141.5 kB | Without provenance | Details for pazufa_corelib-0.3.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pazufa_corelib-0.3.1-py3-none-any.whl | Python 3 | none | any | Details for pazufa_corelib-0.3.1-py3-none-any.whl |
Total release size: 367.1 kB
Release files / pazufa_corelib-0.3.1.tar.gz
| Download URL | pazufa_corelib-0.3.1.tar.gz |
|---|---|
| Size | 141.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b08f1b06c50f7f93267b3256a3a62d66c8cb51e1b49d4cc2dc80413483e848bb
|
|
BLAKE2b-256 checksum How to use checksums |
cc0ebac1ba610ee38c094359beec710bc622a883ab868260451183625217f7e7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.15
|
Release files / pazufa_corelib-0.3.1-py3-none-any.whl
| Download URL | pazufa_corelib-0.3.1-py3-none-any.whl |
|---|---|
| Size | 225.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0c25c2467db39540b481c666237c778118d8b548a1901dc6a6bcd2c5b43700f1
|
|
BLAKE2b-256 checksum How to use checksums |
e9b535b1a001c74e523ffaf9f094c84cd6c07df67adc70cc526eb1b8c48ea153
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.15
|
Security for pazufa-corelib 0.3.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
