pywikiscrape

GitHub license GitHub issues Python Version

pywikiscrape is a user-friendly but powerful open-source Wikipedia scraper written in Python that lets users export articles in any language to an SQLite3 database using Beautiful Soup.

Screenshot of CLI pywikiscrape now supports all Wikipedias, English and non-English, all with manual and automatic seeds

Features

  • Automated Scraping: Automatically scrapes Wikipedia articles with requests and BeautifulSoup.
  • Supports Non-English Wikipedias: Supports scraping for all English and non-English Wikipedias, along with automatic and manual seeds.
  • Minimal-Config: Simply run the script, enter a couple of settings or click enter for defaults, and sit back and relax.
  • Two SQLite3 Tables: Stores two tables, one for the content of the article, the other for the links contained in it, in one file.
  • Minimal Dependencies: Relying only on Python standard library modules, requests, and BeautifulSoup.
  • Error Handling: Handles many types of errors on its own, and only asks for extra user input if necessary.
  • Logging: Uses the Python Standard Library module logging over print statements.

Usage

Install with pip or pipx (recommended)

It is recommended for Linux and macOS users to use pipx rather than pip due to the high possibility of your Python being marked as externally managed in order to prevent the overwriting of files needed by your operating system, which can break system tools. Windows users are free to use pip or pipx to install pywikiscrape.

Tool Command
pip pip install pywikiscrape
pipx pipx install pywikiscrape
Clone repository and manually run If you wish to contribute to this project, you must clone this repository to your local machine, make your changes, and make a pull request. To do so, follow this code block. It is recommended for non-Windows users to use pipx instead of pip to install the dependencies, but either may be used if your operating system or distribution allows so. 
# Clone the repository
git clone https://github.com/tyleruploads/pywikiscrape.git
cd pywikiscrape

# Set up a virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pipx install -r requirements.txt

# Runs the script
python3 src/main.py

The script saves its output to an SQLite3 database with the default name of LANGUAGE_CODE-pywikiscrape.db. All required variables are extracted from the user via text prompts. There are no arguments you need to pass to the program.

The seed can either be chosen manually by the user or automatically by the script.

Database Schemas

Table 1: Articles

This table contains the title of the Wikipedia article and the text inside of it.

Column Type Description
id INTEGER Primary Key
title TEXT Wikipedia article Title
text TEXT Content of Wikipedia article

Table 2: Links

This table contains the id correlating an article to its entry in table 1 and the links contained inside of it.

Column Type Description
id INTEGER Primary Key
links_json TEXT The links contained in the Wikipedia article

Roadmap

  • Package and publish to PyPI
  • Have the CLI ask the user for a seed, a custom path for the output database, and other settings, all with defaults the user can easily select
  • Support scraping non-English Wikipedias
  • Store more metadata about article like categories, images, and when it was added to the database
  • Allow more export options for data like JSON, CSV, or Parquet alongside SQLite
  • Add Unit testing
  • Add a Dockerfile and docker-compose.yml for zero-setup execution

Contributing

Contributions are what make open-source projects one of a kind. All contributions are highly appreciated.

  • Found a bug or issue: Open an Issue and show the output of the script, the steps to reproduce it, and as much information as possible
  • Have an idea: Open an Issue and explain your idea as much as possible, why you think it would be a good addition to the project, and any other important information.

License

This project is licensed under the MIT License - see the LICENSE file for more information.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for TylerUploads from gravatar.com TylerUploads

Credits

Author: tyleruploads

License

MIT License (MIT)

Requires

Python >=3.10

Tags

api beautifulsoup beautifulsoup4 bs4 cli os-independent requests script sqlite3 terminal wikipedia wikipedia-scraper

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pywikiscrape-0.3.0.tar.gz (58.5 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

pywikiscrape-0.3.0-py3-none-any.whl (8.7 kB view details)

Uploaded Python 3

File details

Details for the file pywikiscrape-0.3.0.tar.gz.

File metadata

  • Download URL: pywikiscrape-0.3.0.tar.gz
  • Upload date:
  • Size: 58.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for pywikiscrape-0.3.0.tar.gz
Algorithm Hash digest
SHA256 31f135f2325e4dc67f8d5e4edcf86195731e2f9a5d1fa5b65dcf57d7bde1e5ca
MD5 248675f44f01b34fde0fa21abd82a6c4
BLAKE2b-256 c3866182d6557cc98624d18bc097582346b93b6c3ea871aeacfc843cc44cf87d

See more details on using hashes here.

Provenance

The following attestation bundles were made for pywikiscrape-0.3.0.tar.gz:

Publisher: publish.yaml on tyleruploads/pywikiscrape

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pywikiscrape-0.3.0-py3-none-any.whl.

File metadata

  • Download URL: pywikiscrape-0.3.0-py3-none-any.whl
  • Upload date:
  • Size: 8.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for pywikiscrape-0.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 6048298950fa40c30927816b8faf2dd32458665d849bb2d839e327228b029969
MD5 c936e462a0a27801aca6307823281a00
BLAKE2b-256 1438788e049423c2e23ffb6f1fcdfb22c59d3cd9379769587954efa452a23899

See more details on using hashes here.

Provenance

The following attestation bundles were made for pywikiscrape-0.3.0-py3-none-any.whl:

Publisher: publish.yaml on tyleruploads/pywikiscrape

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.3.0 This release

2 files

0.2.0

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page