Skip to main content

Turn StackExchange dumps into ZIM files for offline usage

Project description

Sotoki

Sotoki (Stack Overflow to Kiwix) is an openZIM scraper to create offline versions of Stack Exchange websites such as Stack Overflow.

It is based on Stack Exchange's Data Dumps hosted by The Internet Archive.

CodeFactor Docker License: GPL v3 PyPI version shields.io PyPI - Python Version

Usage

Sotoki works off a dump of a StackExchange website, as regularly created by StackExchange team. You must provide a --mirror to use to download this dump and the --domain you want to scrape.

For instance, to download Sports StackExchange website as of August 2024 and based on dump hosted on archive.org, you have to use --mirror https://archive.org/download/stackexchange_20240829 --domain sports.stackexchange.com. Value of mirror is hence continuously updated as new dumps are published by StackExchange team.

Other CLI parameters are mandatory:

  • --title: ZIM title, must be less than 30 chars. E.g. Gardening and Landscaping
  • --description: ZIM description, must be less than 80 chars

Docker

docker run -v my_dir:/output ghcr.io/openzim/sotoki sotoki --help

Installation

sotoki is a Python3 software. If you are not using the Docker image, you are advised to use it in a virtual environment to avoid installing software dependencies on your system.

python3 -m venv ./env  # creates a virtual python environment in ./env folder
./env/bin/pip install -U pip  # upgrade pip (package manager). recommended
./env/bin/pip install -U sotoki  # install/upgrade sotoki inside virtualenv

# direct access to in-virtualenv sotoki binary, without shell-attachment
./env/bin/sotoki --help
# alias or link it for convenience
sudo ln -s $(pwd)/env/bin/sotoki /usr/local/bin/

# alternatively, attach virtualenv to shell
source env/bin/activate
sotoki --help
deactivate  # unloads virtualenv from shell

Developers

Anybody is welcome to improve the Sotoki.

To run Sotoki off the git repository, you'll need to download a few external dependencies that we pack in Python releases. Just run python src/sotoki/dependencies.py.

See requirements.txt for the list of python dependencies.

Users

You don't have to make your own ZIM files of Stack Exchange's Web sites. Updated ZIM files are built on a regular basis for all of them. Look at https://library.kiwix.org/?category=stack_exchange to download them.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sotoki-2.2.0.tar.gz (1.7 MB view details)

Uploaded Source

Built Distribution

sotoki-2.2.0-py3-none-any.whl (1.7 MB view details)

Uploaded Python 3

File details

Details for the file sotoki-2.2.0.tar.gz.

File metadata

  • Download URL: sotoki-2.2.0.tar.gz
  • Upload date:
  • Size: 1.7 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.0.0 CPython/3.12.11

File hashes

Hashes for sotoki-2.2.0.tar.gz
Algorithm Hash digest
SHA256 bd7906c085de6ef6cb56bdbc9f7c2080f4bc63be413f4a4ca1d491ab1fd4acd3
MD5 ee7bdb09734a94e7fd110c031eafb562
BLAKE2b-256 5aae1077447d6079a30f91c9049294a613a75bbf9abbf65ac8aa756c548c3c93

See more details on using hashes here.

File details

Details for the file sotoki-2.2.0-py3-none-any.whl.

File metadata

  • Download URL: sotoki-2.2.0-py3-none-any.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.0.0 CPython/3.12.11

File hashes

Hashes for sotoki-2.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 8be32c0b0aeedc163a1e9be9680ea0746a68d12216d862d3fca2465f86fa4204
MD5 b967e23876624ce28fe8240a4e9254a1
BLAKE2b-256 674566770f5bcf1a29d3be8f349ad1a03ec0d9543b38f1394dd840ae2a4a7d9b

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page