This scraper offers both public and private scraping. The latter requires your username and password and performs the following tasks for you:
login
download of all agendas and motions related to upcoming meetings of committees and plenary sessions
Only considers meetings where you already have been invited formally through ALLRIS.
IMPORTANT: All districts are supported but official committee abbreviations will only work for Eimsbüttel as of now.
The public scraper ought to be used with care as it accesses all accessible pages of an entire month. Currently, June 2020 is hardcoded and it can only be used in a GUI environment.
Requirements
Python 3.7+
Firefox installed
geckodriver binary in PATH
Initial setup
Install ALLRIS scraper pip install twomartens.allrisscraper (you need Python 3.7+)
Run ALLRIS scraper a first time tm-allrisscraper (creates config ini in your current working directory)
Fill out the config file with your login credentials and an absolute path on your system to store PDFs of files
Configuration
[Default]
; possible values for district: Altona, Bergedorf, Eimsbüttel, Hamburg-Nord,
; Hamburg-Mitte, Harburg, Wandsbek
district = Eimsbüttel
; if you are not from Eimsbüttel your domain ending will differ
username = max.mustermann@eimsbuettel.de
; password is stored in clear text, therefore ini file should have most
; restrictive read permissions
password = VerySecurePassword
; location for storage of PDFs (trailing slash is IMPORTANT)
pdflocation = /path/to/storage/of/PDFs/
; location of the firefox binary
firefoxBinary = /Pfad/zur/firefox.exe
; location of the geckodriver binary
geckodriver = /Pfad/zum/geckodriverUsage after initial setup
Run ALLRIS scraper: tm-allrisscraper (takes a few seconds to finish)
In the specified location for download you will find the following structure:
YYYY-MM-DD_Abbreviation of committee or plenary session/ (one directory for each meeting)
files inside the directory: Einladung.pdf (contains invitation), Mappe.pdf (contains all motions in one document), and Tagesordnung.pdf (agenda)
Packaging new versions
Activate virtual environment and run python -m build --sdist --wheel
1 maintainer
Release files for twomartens.allrisscraper 0.5.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| twomartens.allrisscraper-0.5.9.tar.gz | 20.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| twomartens.allrisscraper-0.5.9-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 42.7 kB
Release files / twomartens.allrisscraper-0.5.9.tar.gz
| Download URL | twomartens.allrisscraper-0.5.9.tar.gz |
|---|---|
| Size | 20.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
88baf3a71811704e5653af2f2403e9441fe33b91c5075e9b44fe6a418036a5da
|
|
BLAKE2b-256 checksum How to use checksums |
aeb7acb7ee75bd2466ea7d5258462896438bfdfbd8352c23c428d096dbc9abf7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/44.0.0 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.10
|
Release files / twomartens.allrisscraper-0.5.9-py3-none-any.whl
| Download URL | twomartens.allrisscraper-0.5.9-py3-none-any.whl |
|---|---|
| Size | 21.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2db9305f2aa06d327649787002e8687fba4562be99617893b50035be12f78058
|
|
BLAKE2b-256 checksum How to use checksums |
523cbcacbb221f79be2668d2ade521edb23b851dd3991d6da11f78c54e2f7222
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/44.0.0 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.10
|
