zimscraperlib

QA Status Tests Status CodeFactor License: GPL v3 PyPI version shields.io PyPI - Python Version codecov Read the Docs

Collection of python code to re-use across python-based scrapers

Usage

  • This library is meant to be installed via PyPI (zimscraperlib).
  • Make sure to reference it using a version code as the API is subject to frequent changes.
  • API should remain the same only within the same minor version.

Example usage:

zimscraperlib>=1.1,<1.2

See documentation at Read the Docs for details.

[!WARNING] While this library brings support for downloading videos with yt-dlp, recent changes in Youtube have forced yt-dlp team to require new dependencies for youtube videos (see https://github.com/yt-dlp/yt-dlp/issues/15012). These dependencies are significantly big and not needed for all other backend supported by yt-dlp (only youtube needs it). These dependencies are hence not included in this library dependencies (yet, see https://github.com/openzim/python-scraperlib/issues/268), you have to install them on your own if you intend to download videos from Youtube.

Dependencies

Most dependencies are installed automatically by pip (from PyPI by default). The following system packages may be required depending on which features you use:

  • libmagic — required for file type detection (used in most scrapers)
  • wget — required only for zimscraperlib.download functions
  • FFmpeg — required only for video processing functions
  • gifsicle (>=1.92) — required only for GIF optimization
  • libcairo — required only for SVG-to-PNG conversion
  • libzim — auto-installed via PyPI, not available on Windows
  • Pillow — auto-installed via PyPI; pre-built wheels are used by default and no system image libraries are needed. Only if you need to build Pillow from source should you install additional system libraries — see Pillow's build documentation for details.

    Note: To run the full test suite, all system dependencies listed above must be installed.

macOS

brew install libmagic wget ffmpeg gifsicle cairo

Linux

sudo apt install libmagic1 wget ffmpeg gifsicle libcairo2

Alpine

apk add ffmpeg gifsicle libmagic wget cairo

Contribution

This project adheres to openZIM's Contribution Guidelines.

This project has implemented openZIM's Python bootstrap, conventions and policies v2.0.0.

All instructions below must be run from the root of your local clone of this repository.

If you do not already have it on your system, install hatch:

pip install hatch

Start a hatch shell — this will install all dependencies including dev in an isolated virtual environment:

hatch shell

Set up the pre-commit Git hook (runs linters automatically before each commit):

pre-commit install

Run tests with coverage:

invoke coverage

Users

Non-exhaustive list of scrapers using it (check status when updating API):

Project links

Data verified by PyPI on Jul 31, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for kiwix from gravatar.com kiwix

Credits

Author: openZIM

GitHub Statistics

Data verified by PyPI on Jul 31, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

GNU General Public License v3 or later (GPLv3+) (GPL-3.0-or-later)

Requires

Python <3.15, >=3.14

Provides Extra

dev docs qa scripts test

Tags

offline openzim zim

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

zimscraperlib-5.4.1.tar.gz (6.6 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

zimscraperlib-5.4.1-py3-none-any.whl (130.8 kB view details)

Uploaded Python 3

File details

Details for the file zimscraperlib-5.4.1.tar.gz.

File metadata

  • Download URL: zimscraperlib-5.4.1.tar.gz
  • Upload date:
  • Size: 6.6 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for zimscraperlib-5.4.1.tar.gz
Algorithm Hash digest
SHA256 3c67fe46f5b124e7244a504ef8e3edd51c7dbc3c5087a2bcb769931eb9966bc8
MD5 4e2de317744f61baced0a966754243b9
BLAKE2b-256 8fcb43e00ffae9e7ebf0f29691acb6a5231ad8222738c7f866a3a2138577070b

See more details on using hashes here.

Provenance

The following attestation bundles were made for zimscraperlib-5.4.1.tar.gz:

Publisher: Publish.yaml on openzim/python-scraperlib

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file zimscraperlib-5.4.1-py3-none-any.whl.

File metadata

  • Download URL: zimscraperlib-5.4.1-py3-none-any.whl
  • Upload date:
  • Size: 130.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for zimscraperlib-5.4.1-py3-none-any.whl
Algorithm Hash digest
SHA256 b029c314323c82a93b87ffcf16becce694a167395f9c832b9cc60b2d3b1a45e5
MD5 179ed684dfad01480aa80814dde59cc1
BLAKE2b-256 9c5b9bc1a0423875530def5e7cdcd36a33d1887b2ae2c905f31cc3f648e16d13

See more details on using hashes here.

Provenance

The following attestation bundles were made for zimscraperlib-5.4.1-py3-none-any.whl:

Publisher: Publish.yaml on openzim/python-scraperlib

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page