pywikiscrape

pywikiscrape is a user-friendly but powerful open-source Wikipedia scraper written in Python that lets users export articles in any language to an SQLite3 database using Beautiful Soup.

pywikiscrape now supports all Wikipedias, English and non-English, all with manual and automatic seeds

Features

• Automated Scraping: Automatically scrapes Wikipedia articles with requests and BeautifulSoup.
• Supports Non-English Wikipedias: Supports scraping for all English and non-English Wikipedias, along with automatic and manual seeds.
• Minimal-Config: Simply run the script, enter a couple of settings or click enter for defaults, and sit back and relax.
• Two SQLite3 Tables: Stores two tables, one for the content of the article, the other for the links contained in it, in one file.
• Minimal Dependencies: Relying only on Python standard library modules, requests, and BeautifulSoup.
• Error Handling: Handles many types of errors on its own, and only asks for extra user input if necessary.
• Logging: Uses the Python Standard Library module logging over print statements.

Usage

# Clone the repository
git clone https://github.com/tyleruploads/pywikiscrape.git
cd pywikiscrape

# Set up a virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pipx install -r requirements.txt

# Runs the script
python3 src/main.py

The script saves its output to an SQLite3 database with the default name of LANGUAGE_CODE-pywikiscrape.db. All required variables are extracted from the user via text prompts. There are no arguments you need to pass to the program.

The seed can either be chosen manually by the user or automatically by the script.

Database Schemas

Table 1: Articles

This table contains the title of the Wikipedia article and the text inside of it.

Table 2: Links

This table contains the id correlating an article to its entry in table 1 and the links contained inside of it.

Roadmap

• Package and publish to PyPI
• Have the CLI ask the user for a seed, a custom path for the output database, and other settings, all with defaults the user can easily select
• Support scraping non-English Wikipedias
• Store more metadata about article like categories, images, and when it was added to the database
• Allow more export options for data like JSON, CSV, or Parquet alongside SQLite
• Add Unit testing
• Add a Dockerfile and docker-compose.yml for zero-setup execution

Contributing

Contributions are what make open-source projects one of a kind. All contributions are highly appreciated.

• Found a bug or issue: Open an Issue and show the output of the script, the steps to reproduce it, and as much information as possible
• Have an idea: Open an Issue and explain your idea as much as possible, why you think it would be a good addition to the project, and any other important information.

License

This project is licensed under the MIT License - see the LICENSE file for more information.