Skip to main content

Turn StackExchange dumps into ZIM files for offline usage

Project description

Sotoki

sotoki (stackoverflow to kiwix) is an OpenZIM scraper to create offline versions of Stack Exchange websites such as stack overflow.

It is based on Stack Exchange's Data Dumps hosted by The Internet Archive.

CodeFactor Docker License: GPL v3 PyPI version shields.io

⚠️ Warning

sotoki is undergoing a major rewrite to use libzim7 and its python binding in order to bypass filesystem limitations seen in version 1.x. Use tagged version until this warning is removed as current master is not-functionnal.

Usage

sotoki works off a domain that you must provide. That is the domain-name of the stackexchange website you want to scrape. Run sotoki --list-all to get a list of those

Note: when running off the git repository, you'll need to download a few external dependencies that we pack in Python releases. Just run python src/sotoki/dependencies.py

Docker

docker run -v my_dir:/output openzim/sotoki sotoki --help

Virtualenv

sotoki is a Python3 software. If you are not using the Docker image, you are advised to use it in a virtual environment to avoid installing software dependencies on your system.

python3 -m venv env      # Create virtualenv
source env/bin/Activate  # Activate the virtualenv
pip3 install sotoki # Install dependencies
sotoki --help       # Display kolibri2zim help

Call deactivate to quit the virtual environment.

See requirements.txt for the list of python dependencies.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sotoki-2.0.1.tar.gz (1.6 MB view hashes)

Uploaded source

Built Distribution

sotoki-2.0.1-py3-none-any.whl (1.6 MB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page