ao3scraper is a python webscraper that scrapes AO3 for fanfiction data, stores it in a database, and highlights entries when they are updated.
Project description
ao3scraper
A python webscraper that scrapes AO3 for fanfiction data, stores it in a database, and highlights entries when they are updated.
Table with an updated entry highlighted.
Installation
You can easily install the latest version from pip:
pip3 install ao3scraper
Development Installation
Create a python virtual environment with python3 -m venv dev_venv
and activate it.
Then, install required packages with:
poetry install
This will also install ao3scraper into the virtual environment.
Usage
Usage: ao3scraper [OPTIONS]
Options:
-s, --scrape Launches scraping mode.
-c, --cache Prints the last scraped table.
-l, --list Lists all entries in the database.
-a, --add TEXT Adds a single url to the database.
--add-urls Opens a text file to add multiple urls to the database.
-d, --delete INTEGER Deletes an entry from the database.
-v, --version Display version of ao3scraper and other info.
--help Show this message and exit.
Configuration
ao3scraper is ridiculously customisable, and most aspects of the program can be modified from here.
To find the configuration file location, run python3 ao3scraper -v
.
ao3scraper uses rich's styling. To disable any styling options, replace the styling value with 'none'.
Fics have many attributes that are not displayed by default. To add these columns, create a new option under table_template, like so:
table_template:
- column: characters # The specified attribute
name: Characters :) # This is what the column will be labelled as
styles: none # Rich styling
A complete list of attributes can be found on the wiki.
Migrating the database
If you're updating from a legacy version of ao3scraper (before 1.0.0), move fics.db
to the data location.
This can be found by running python3 ao3scraper -v
.
The migration wizard will then prompt you to upgrade your database.
If you accept, a backup of the current fics.db
will be created in /backups
, and migration will proceed.
Contributing
Contributions are always appreciated. Submit a pull request with your suggested changes!
Acknowledgements
ao3scraper would not be possible without the existence of ao3_api and the work of its contributors.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ao3scraper-1.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 968774efe95d3c7fdd37e66bb5a46fe8626cf0c3a53e3562b4f806ec7390e141 |
|
MD5 | 487be7ff171425cfb377b6cb13bfe06c |
|
BLAKE2b-256 | 61d0a4b6135e89e300e64972b9fda7e3f418f0619ca534f40b5147b42dfe5675 |