thetrove-downloader

This Python package allows to easily download any resource, be it folder or file, from The Trove.is.

Installation and Requirements

The program requires Python 3.9 or above.

To install the program it is sufficient to use Python pip and get the package thetrove-downloader. The package will add a new command thetrove-downloader.

python3 -m pip install thetrove-downloader

Updates should be performed via pip.

python3 -m pip install --upgrade thetrove-downloader

Usage

thetrove-downloader [-h] [-t, --target TARGET] [--target-url URL]
                    [-j, --json JSON] [-f, --folder FOLDER]
                    [-o, --output OUTPUT] [-b, --blacklist BLACKLIST]
                    [-w, --whitelist WHITELIST] [--no-download]
                    [--no-preserve-root]

• -h, --help show help message

• -t, --target <TARGET> download target (folder or file)
  To download a specific resource, start the program with the --target command and supply a valid path (e.g., " Books/White Star").

• --target-url download target (full URL to folder or file)
  Overrides --target argument with a full URL and sets the download target to the path of the URL ( e.g., "https://thetrove.is/Open%20Source/BIND%20RPG" to "Open Source/BIND RPG").

• -j, --json <JSON> save/read instructions from a JSON file
  The --json option allows to specify a file to save the target, folder, output, whitelist, and blacklist instructions in JSON format. Launching the program with only the --json option will load the instructions previously saved in the file and perform all of them in succession. This can be used to keep a local copy of specific resources up to date.

• -f, --folder <FOLDER> destination folder
  The --folder option allows to specify a folder in which the targeted resource will be downloaded.

• -o, --output <OUTPUT> output name of download target
  The --output option allows to override the folder/file name of the resource. Note that if the targeted resource is a folder, the option will only apply to the top-level folder and leave all contents therein unaltered.

• -b, --blacklist <BLACKLIST> regex blacklist for files/folders
  The --whitelist option allows to specify a regex pattern that will be used to skip files and folder that do not contain the supplied pattern. Overwrites blacklist options.

• -w, --whitelist <WHITELIST> regex whitelist for files/folders, overrides blacklist
  The --whitelist option allows to specify a regex pattern that will be used to skip files and folder that do not contain the supplied pattern. Overwrites blacklist options.

• --no-download list content without downloading
  The --no-download flag allows to list the content under the supplied target without performing any download. The instructions will still be saved in a JSON file if the --json options is used, but not the flag.

• --no-preserve-root allows the download of root folders
  The --no-preserve-root flag allows downloading entire root folders such as "/Books/". This flag is not preserved with the --json option and must be passed manually every time.

NOTE: the program requires to use either --target, --json, or both --target and --json. Without one of these combinations the program won't have a valid download instruction.

Notes on Crawling

The program performs all GET requests without throttling or delay and with no disallowed paths as per the instructions at thetrove.is/robots.txt.

However, please do take care to not overuse this utility and risk overwhelming their connection. Thank you.

Root folders such as "/Books" are protected and cannot be downloaded without using the --no-preserve-root flag.

Contributing

All contributions and suggestions are welcome!

If you have suggestions for fixes or improvements, you can open an issue with your idea, see #Issues for details.

Issues

If you encounter any problem during usage of the program, an issue can be opened on the project's page on GitLab.

When opening an issue for a problem, please copy the error message – if any – and describe the operation in progress when the error occurred.