zimscraperlib
Collection of python code to re-use across python-based scrapers
Usage
- This library is meant to be installed via PyPI (
zimscraperlib). - Make sure to reference it using a version code as the API is subject to frequent changes.
- API should remain the same only within the same minor version.
Example usage:
zimscraperlib>=1.1,<1.2
See documentation at Read the Docs for details.
[!WARNING] While this library brings support for downloading videos with yt-dlp, recent changes in Youtube have forced yt-dlp team to require new dependencies for youtube videos (see https://github.com/yt-dlp/yt-dlp/issues/15012). These dependencies are significantly big and not needed for all other backend supported by yt-dlp (only youtube needs it). These dependencies are hence not included in this library dependencies (yet, see https://github.com/openzim/python-scraperlib/issues/268), you have to install them on your own if you intend to download videos from Youtube.
Dependencies
Most dependencies are installed automatically by pip (from PyPI by default). The following system packages may be required depending on which features you use:
- libmagic — required for file type detection (used in most scrapers)
- wget — required only for
zimscraperlib.downloadfunctions - FFmpeg — required only for video processing functions
- gifsicle (>=1.92) — required only for GIF optimization
- libcairo — required only for SVG-to-PNG conversion
- libzim — auto-installed via PyPI, not available on Windows
- Pillow — auto-installed via PyPI; pre-built wheels are used by default and no system image libraries are needed. Only if you need to build Pillow from source should you install additional system libraries — see Pillow's build documentation for details.
Note: To run the full test suite, all system dependencies listed above must be installed.
macOS
brew install libmagic wget ffmpeg gifsicle cairo
Linux
sudo apt install libmagic1 wget ffmpeg gifsicle libcairo2
Alpine
apk add ffmpeg gifsicle libmagic wget cairo
Contribution
This project adheres to openZIM's Contribution Guidelines.
This project has implemented openZIM's Python bootstrap, conventions and policies v2.0.0.
All instructions below must be run from the root of your local clone of this repository.
If you do not already have it on your system, install hatch:
pip install hatch
Start a hatch shell — this will install all dependencies including dev in an isolated virtual environment:
hatch shell
Set up the pre-commit Git hook (runs linters automatically before each commit):
pre-commit install
Run tests with coverage:
invoke coverage
Users
Non-exhaustive list of scrapers using it (check status when updating API):
1 maintainer
Metadata
1 maintainer
Release files for zimscraperlib 5.4.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| zimscraperlib-5.4.1.tar.gz | 6.6 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| zimscraperlib-5.4.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 6.8 MB
Release files / zimscraperlib-5.4.1.tar.gz
| Download URL | zimscraperlib-5.4.1.tar.gz |
|---|---|
| Size | 6.6 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3c67fe46f5b124e7244a504ef8e3edd51c7dbc3c5087a2bcb769931eb9966bc8
|
|
BLAKE2b-256 checksum How to use checksums |
8fcb43e00ffae9e7ebf0f29691acb6a5231ad8222738c7f866a3a2138577070b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 31, 2026.
Transparency logRelease files / zimscraperlib-5.4.1-py3-none-any.whl
| Download URL | zimscraperlib-5.4.1-py3-none-any.whl |
|---|---|
| Size | 130.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b029c314323c82a93b87ffcf16becce694a167395f9c832b9cc60b2d3b1a45e5
|
|
BLAKE2b-256 checksum How to use checksums |
9c5b9bc1a0423875530def5e7cdcd36a33d1887b2ae2c905f31cc3f648e16d13
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 31, 2026.
Transparency log
