szurubooru-toolkit

Python package and script collection to manage your szurubooru image board.

Usage: szuru-toolkit [OPTIONS] COMMAND [ARGS]...

  Toolkit to manage your szurubooru image board.

  Defaults can also be set in a config file.

  Visit https://github.com/reluce/szurubooru-toolkit for more information.

Options:
  --url TEXT                      Base URL to your szurubooru instance.
  --username TEXT                 Username which will be used to authenticate with the szurubooru API.
  --api-token TEXT                API token for the user which will be used to authenticate with the
                                  szurubooru API.
  --public                        If your szurubooru instance is reachable from the internet (default:
                                  False).
  --log-enabled                   Create a log file (default: False).
  --log-colorized                 Colorize the log output (default: True).
  --log-file TEXT                 Output file for the log (default: szurubooru_toolkit.log)
  --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]
                                  Set the log level (default: INFO).
  --hide-progress                 Hides the progress bar (default: False).
  -h, --help                      Show this message and exit.

Commands:
  auto-tagger        Tag posts automatically
  create-relations   Create relations between character and parody tag categories
  create-tags        Create tags based on a tag file or query
  delete-posts       Delete posts
  find-duplicates    Find visually duplicate posts via perceptual hashing
  fix-relations      Complete post relation sets via transitive closure
  import-from-booru  Download and tag posts from various Boorus
  import-from-url    Download images from URLS or file containing URLs
  preview-tags       Show WD tagger scores near the thresholds without tagging anything
  reset-posts        Remove tags and sources
  tag-posts          Tag posts manually
  upload-media       Upload media files
  webserver          Run the webserver for the browser extensions

:ballot_box_with_check: Requirements

In order to run szuru-toolkit, Python 3.11 or newer is required.

:hammer_and_wrench: Installation

This package is available on PyPI and can be installed with pip: pip install szurubooru-toolkit

The WD tagger (local machine learning tagging) and Pixiv support are optional extras since they pull in heavy dependencies:

• pip install "szurubooru-toolkit[wd-tagger]" for WD tagger support (installs ONNX Runtime)
• pip install "szurubooru-toolkit[pixiv]" for Pixiv metadata support
• pip install "szurubooru-toolkit[wd-tagger,pixiv]" for both

Alternatively, you can clone the package from GitHub and set everything up with uv. In the root directory of this repository, execute uv sync (add --all-extras for WD tagger and Pixiv support).

Docker Instructions

If you would like to run the toolkit in a Docker container instead, follow the instructions below.

Several image variants are published on each release, matching the optional extras. The default image is slim; the others bundle heavier dependencies, so pick the smallest one that covers what you enable in config.toml:

Use the matching tag in your docker-compose.yml — e.g. -wd-tagger if you set wd_tagger = true. Every tag is also published per version, e.g. :2.0.0, :2.0.0-wd-tagger, :2.0.0-pixiv and :2.0.0-all.

If the mounted volumes should not be owned by root (e.g. on NFS mounts or with rootless containers), set the PUID and PGID environment variables (see the commented block in docker-compose.yml): the container then creates a matching user on startup, chowns its working directory and runs the cron jobs as that user instead of root.

:memo: User configuration

While the script szuru-toolkit can run with just command line options, you can also set your options in a config file.

The script looks for a config.toml file in following locations:

Linux

• Your current working directory from which szuru-toolkit is executed
• ~/.config/szurubooru-toolkit/config.toml
• /etc/szurubooru-toolkit/config.toml

Windows

• Your current working directory from which szuru-toolkit is executed
• $USERPROFILE/szurubooru-toolkit/config.toml
• $APPDATA/szurubooru-toolkit/config.toml

Options passed to the szuru-toolkit script take priority over the config file.

You can find a sample config file in the GitHub repository of this package.

Note that path names have to be specified with forward slashes (/) if you're using Windows.

Creating a SauceNAO account and an API key is recommended. Please consider supporting the SauceNAO team as well by upgrading your plan. With a free plan, you can request up to 200 posts in 24h.

For local machine learning tagging, posts can be tagged with one of SmilingWolf's WD taggers. Install the wd-tagger extra (pip install "szurubooru-toolkit[wd-tagger]") and set wd_tagger = true in the [auto_tagger] section to use it. The model set in wd_tagger_model (default: SmilingWolf/wd-eva02-large-tagger-v3, ~1.2GB) gets downloaded automatically from Hugging Face on first use and is cached locally afterwards. Any of the WD v3/v2 taggers work, e.g. SmilingWolf/wd-swinv2-tagger-v3 or SmilingWolf/wd-vit-tagger-v3 for smaller and faster models. General tags and character tags use separate confidence thresholds (wd_tagger_threshold and wd_tagger_character_threshold), since character predictions are usually either confident or wrong. Use szuru-toolkit preview-tags <file-or-post-id> to see all scores near the thresholds when tuning them, and szuru-toolkit auto-tagger --dry-run <query> to preview which tags a run would change without updating any post.

With wd_tagger_review = true, posts whose best character score lands between wd_tagger_review_threshold and wd_tagger_character_threshold get tagged needs_review — a szurubooru query for exactly the ambiguous character matches worth curating manually.

Post safety is detected automatically: booru matches carry over their source rating, and the WD tagger predicts one (safe/sketchy/unsafe) for everything else. On top of that, safety_overrides in [auto_tagger] forces a minimum safety whenever certain tags are present, e.g. safety_overrides = { sketchy = ["nude"], unsafe = ["sex"] }. Safety is only ever raised by this, never lowered — a booru-provided unsafe stays unsafe.

Videos are tagged as well if ffmpeg is installed (wd_tagger_videos): frames are sampled across the duration — longer videos get more frames — and their scores averaged, so tags that only appear in a single frame don't stick.

Inference runs on the CPU by default. For hardware acceleration, set wd_tagger_providers in config.toml, e.g. ["CoreMLExecutionProvider"] on Apple Silicon or ["CUDAExecutionProvider"] on NVIDIA GPUs (requires the onnxruntime-gpu package). Unavailable providers fall back to the CPU.

:page_with_curl: Commands

The CLI is installed as szuru-toolkit and under the shorter alias szuructl — both are identical.

Following commands are currently available:

• auto-tagger: Tag posts automatically
• create-relations: Create relations between character and parody tag categories
• create-tags: Create tags based on a tag file or query
• delete-posts: Delete posts
• find-duplicates: Find visually duplicate posts via perceptual hashing
• fix-relations: Complete post relation sets so every member of a set references all other members
• import-from-booru: Download and tag posts from various Boorus
• import-from-url: Batch importing of URLs based on gallery-dl
• preview-tags: Show WD tagger scores near the thresholds for a file or post without tagging anything
• reset-posts: Remove tags and sources
• tag-posts: Tag posts manually
• upload-media: Upload media files
• webserver: Run the webserver for the browser extensions

Check szuru-toolkit -h or szuru-toolkit COMMAND -h for a detailed description of supported options.

If you cloned the repo from GitHub, prefix the above scripts with uv run, e.g. uv run szuru-toolkit auto-tagger "date:today". Note that your current working directory has to be the root of the GitHub project.

If your query starts with a dash (-), for example to negate a tag, you have to separate the query from the command with two dashes (This doesn't work with uv run):

szuru-toolkit auto-tagger --no-wd-tagger -- "-foo bar"

While most commands are self explanatory, the following require a bit of extra attention:

:label: create-relations

Examples

• szuru-toolkit create-relations hitori_bocchi
  • Will create the implication bocchi_the_rock for tag hitori_bocchi if other posts are found with query hitori_bocchi containing bocchi_the_rock as the parody (tag has to be of category series or parody)
  • Will also add hitori_bocchi as a suggestion to the parody tag bocchi_the_rock
  • These relations will only get generated if at least X posts are found containing the tags bocchi_the_rock and hitori_bocchi. Control X with threshold under [create_relations] in config.toml.

:label: create-tags

If no tag_file is specified, the script will download the most recent 100 tags from Danbooru which have been used at least ten times.

You can use tools like Grabber to download a tag list from common boorus.

The tag_file has to be in following format:

<tag_a>,<category_name>
<tag_b>,<category_name>,<implied_tag>,<implied_tag>
<tag_..n>,<category_name>

The category has to be created beforehand manually (e.g. default, artist, parody/series, character and meta).

Any columns after the category are added as implications; implied tags get created (category default) if they don't exist yet. A single tag can also be created directly with implications passed as an argument. With import_implications = true (or --import-implications), tags created by a Danbooru query additionally get their active Danbooru implications, with implied tags created under their Danbooru category.

Examples

• szuru-toolkit create-tags
• szuru-toolkit create-tags --query genshin* --overwrite
• szuru-toolkit create-tags --query genshin* --import-implications
• szuru-toolkit create-tags --tag-file tags.txt
• szuru-toolkit create-tags --category character --implications "slime,monster_girl" slime_girl

:link: import-from-url

This scripts imports posts with their tags from the URL passed to this script. In the background, it simply calls the gallery-dl script and parses its output. Alternatively, an input file with multiple URLs can be specified.

It's recommended to use the --cookies flag for authentication, check https://github.com/mikf/gallery-dl#cookies for details.

Posts imported from e-hentai/exhentai get the gallery URL set as their source and the artist: tag from the gallery applied; the namespaced e-hentai tags themselves are not imported.

Examples

• szuru-toolkit import-from-url "https://danbooru.donmai.us/posts?tags=foo"
• szuru-toolkit import-from-url "https://chan.sankakucomplex.com/?tags=foo"
• szuru-toolkit import-from-url "https://beta.sankakucomplex.com/post/show/<id>"
• szuru-toolkit import-from-url "https://e-hentai.org/g/<gid>/<token>/"
• szuru-toolkit import-from-url --cookies "~/cookies.txt" --range ":100" "https://twitter.com/<USERNAME>/likes"
• szuru-toolkit import-from-url --input-file urls.txt "https://danbooru.donmai.us/posts?tags=foo" "https://beta.sankakucomplex.com/post/show/<id>"

:outbox_tray: upload-media

This script uploads media files from a local directory (src_path).

With read_sidecar_tags = true, tags are read from a <file>.txt sidecar file next to each media file, one tag per line — both abc.jpg.txt (as written by gallery-dl --write-tags) and abc.txt are picked up. Files without a sidecar get the configured default tags as before. If a file is already uploaded, update_tags_if_exists = true appends the sidecar tags to the existing post instead. With cleanup = true, consumed sidecar files are removed along with their media files.

Examples

• szuru-toolkit upload-media --cleanup --tags "foo,bar"
• szuru-toolkit upload-media --read-sidecar-tags --update-tags-if-exists

:information_source: Image credit

GitHub repo icon: Code icons created by Smashicons - Flaticon