H2HDB Downloader (h2hdb-downloader)

Automates downloading galleries from exhentai/e-hentai (via hbrowser) and recording their state in an h2hdb database. It has no CLI or standalone runtime of its own — it's a library consumed by another project that owns the browser session and the overall process lifecycle.

Concepts

• Gallery — a single exhentai/e-hentai gallery, identified by a gid (numeric id) and represented as h2h_galleryinfo_parser.GalleryURLParser once its URL is known.
• Dedup — before issuing a real network download, the package reads live h2hdb state to see if the gid is already settled (downloaded, with no redownload flag or durable request). Settled gids are skipped — except periodically, at a random interval (1 to 19 attempts), when one is force-redownloaded as an integrity re-check.
• Durable requests — immediately before a real download starts, the package creates a tokenized request in h2hdb's todownload_gids table. It conditionally completes that exact token only after success. A False result, exception, cancellation, or process termination leaves resumable work behind, while a newer request for the same gid cannot be erased by an older attempt finishing late. h2hdb also uses this table to publish a redownload request after all active deletion-candidate folders for a gid have actually disappeared.
• Database coordination — every short h2hdb read/write section enters h2hdb's cross-process maintenance gate with a five-minute wait interval. Browser search, downloads, retry sleeps, and tag traversal stay outside the gate, so maintenance is never blocked by network work.
• Manual queue — add a (gid, url) row to the CSV configured by csv_path. It is converted into the same durable request and picked up the next time the queue is drained. Before replay, the inbox is atomically rotated to a same-directory hidden claim file; interrupted claims are replayed automatically on the next run.
• Deep download — download a gallery, then look at its artist/group tags and download sibling galleries that match a set of search conditions (e.g. other-language releases of the same work).

API

Downloader is the public service object; TagCascadePolicy is the other public export. Every method either acts on a target you explicitly pass in or, for the two queue-reading methods below, hands back a plain value with no further bookkeeping required from you. There is no "run the whole thing" method: deciding when to stop, what order to process things in, and how to report progress is the calling application's job, not the library's.

Downloader(
    driver: ExHDriver,         # an un-entered driver; see below
    config_path: str,          # path to the h2hdb JSON config
    csv_path: str | None = None,  # path to the manual download-queue CSV
    *,
    wait4client: int,       # seconds to wait before retrying after ClientOfflineException
    retry2download: int,    # seconds to wait before retrying after InsufficientFundsException
)

csv_path only enables the optional "queue a gid/url by editing a CSV file" feature described above. Leave it as None if you don't need that; durable database requests and live deduplication still work.

Downloader is itself an async context manager that opens and closes the browser session for you, so driver is expected un-entered:

async with Downloader(ExHDriver(headless=False), ...) as downloader:
    ...

If you'd rather manage the driver's lifecycle yourself, pass an already-entered driver and skip async with downloader.

Method names follow one rule throughout: no suffix means it operates directly on a GalleryURLParser you already have; _by_gid means it resolves a bare gid to its gallery via search first, then does the same thing.

• await download_by_gallery(target) — download one GalleryURLParser, or an iterable of them. Returns {gid: downloaded} for each. Retries automatically on ClientOfflineException (waits wait4client seconds) and InsufficientFundsException (waits retry2download seconds); a wait of 0 means "don't retry, raise immediately."
• await download_by_gid(gid) — resolve a bare gid to its gallery via search, then download it. If the gid no longer resolves to anything, it's recorded as removed in h2hdb; if it resolves to a different gid (the gallery was merged/redirected), the original gid is flagged for deletion after the replacement downloads successfully.
• await download_by_tag(tag, conditions) — download every gallery under a hbrowser Tag, once per search condition in conditions (or unconditionally if conditions is empty).
• await deep_download_by_gallery(gallery, policy, skip_check=False) — download gallery, then for each tag in policy.filters (e.g. "artist", "group") on that gallery, call download_by_tag with policy.conditions. The cascade only runs if the initial download actually happened, unless skip_check=True forces it to run regardless (useful when you already know the gallery is downloaded from a separate call and just want the cascade). policy is a TagCascadePolicy(filters, conditions) — both fields always travel together, so they're grouped into one frozen value object rather than two parallel parameters.
• await deep_download_by_gid(gid, policy, skip_check=False) — same gid-resolution as download_by_gid, but deep.
• await drain_queue(policy, skip_check=True) — absorb the manual CSV and process one live snapshot of durable database requests. A request is removed only after a successful download, confirmed removal, or successful redirect. The method doesn't loop or wait for new work.
• pending_redownload_gids() — a snapshot list of gids h2hdb currently flags as needing a periodic redownload. Every call reads live database state; read-only and safe to call repeatedly as you work through it.

Example

The calling application owns the loop. A typical one drains the queue once, then walks the pending-redownload list, deep-downloading anything that actually got (re)downloaded:

import asyncio
from h2hdb_downloader import Downloader, TagCascadePolicy
from hbrowser import ExHDriver
from h2h_galleryinfo_parser import GalleryURLParser

policy = TagCascadePolicy(
    filters=("artist", "group"),
    conditions=("language:chinese$", "language:speechless$"),
)


async def main():
    async with Downloader(
        ExHDriver(headless=True),
        config_path="h2hdb-config.json",
        csv_path="todownload_gids.csv",
        wait4client=30 * 60,
        retry2download=4 * 60 * 60,
    ) as downloader:
        gallery = GalleryURLParser("https://exhentai.org/g/123/456/")
        await downloader.download_by_gallery(gallery)
        await downloader.download_by_gid(666)
        await downloader.deep_download_by_gallery(gallery, policy)

        await downloader.drain_queue(policy, skip_check=True)
        for gid in downloader.pending_redownload_gids():
            await downloader.deep_download_by_gid(gid, policy, skip_check=True)


asyncio.run(main())

License

This project is distributed under the terms of the GNU General Public Licence (GPL). For detailed licence terms, see the LICENSE file included in this distribution.