ao3scraper

A python webscraper that scrapes AO3 for fanfiction data, stores it in a database, and highlights entries when they are updated.

Table with an updated entry highlighted.

Installation

You can easily install the latest version from pip:

pip3 install ao3scraper

Development Installation

Create a python virtual environment with python3 -m venv dev_venv and activate it. Then, install required packages with:

poetry install

This will also install ao3scraper into the virtual environment.

Usage

Usage: ao3scraper [OPTIONS]

Options:
-s, --scrape          Launches scraping mode.
-c, --cache           Prints the last scraped table.
-l, --list            Lists all entries in the database.
-a, --add TEXT        Adds a single url to the database.
--add-urls            Opens a text file to add multiple urls to the database.
-d, --delete INTEGER  Deletes an entry from the database.
-v, --version         Display version of ao3scraper and other info.
--help                Show this message and exit.

Configuration

ao3scraper is ridiculously customisable, and most aspects of the program can be modified from here. To find the configuration file location, run python3 ao3scraper -v.

ao3scraper uses rich's styling. To disable any styling options, replace the styling value with 'none'.

Fics have many attributes that are not displayed by default. To add these columns, create a new option under table_template, like so:

table_template:
- column: characters # The specified attribute
  name: Characters :) # This is what the column will be labelled as
  styles: none # Rich styling

A complete list of attributes can be found on the wiki.

Migrating the database

If you're updating from a legacy version of ao3scraper (before 1.0.0), move fics.db to the data location. This can be found by running python3 ao3scraper -v. The migration wizard will then prompt you to upgrade your database. If you accept, a backup of the current fics.db will be created in /backups, and migration will proceed.

Contributing

Contributions are always appreciated. Submit a pull request with your suggested changes!

Acknowledgements

ao3scraper would not be possible without the existence of ao3_api and the work of its contributors.