Collection of python tools to re-use common code across scrapers
Project description
zimscraperlib
Collection of python code to re-use across python-based scrapers
Usage
- This library is meant to be installed via PyPI (
zimscraperlib
). - Make sure to reference it using a version code as the API is subject to frequent changes.
- API should remain the same only within the same minor version.
Example usage:
zimscraperlib>=1.1,<1.2
Dependencies
- libmagic
- wget
- libzim (auto-installed, not available on Windows)
- Pillow
- FFmpeg
- gifsicle (>=1.92)
macOS
brew install libmagic wget libtiff libjpeg webp little-cms2 ffmpeg gifsicle
Linux
sudo apt install libmagic1 wget ffmpeg \
libtiff5-dev libjpeg8-dev libopenjp2-7-dev zlib1g-dev \
libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python3-tk \
libharfbuzz-dev libfribidi-dev libxcb1-dev gifsicle
Alpine
apk add ffmpeg gifsicle libmagic wget libjpeg
Nota: i18n features do not work on Alpine, see https://github.com/openzim/python-scraperlib/issues/134 ; there is one corresponding test which is failing.
Contribution
This project adheres to openZIM's Contribution Guidelines.
This project has implemented openZIM's Python bootstrap, conventions and policies v1.0.2.
pip install hatch
pip install ".[dev]"
pre-commit install
# For tests
invoke coverage
Users
Non-exhaustive list of scrapers using it (check status when updating API):
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
zimscraperlib-3.4.0.tar.gz
(3.2 MB
view hashes)
Built Distribution
Close
Hashes for zimscraperlib-3.4.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f64ec376a262f715e3341adf2425f715224dcf3786e318498e7804442486673c |
|
MD5 | 5d7f26df91e17184bae37ba9c8f5307b |
|
BLAKE2b-256 | 4d871e93bfb27861f0214ca5753dba29655741512a0a1659d335f07216a2a535 |