h2hdb-ingest

h2hdb-ingest turns completed Hentai@Home downloads into an H2HDB catalog and an optional comic library for Komga and OPDS readers. It watches your download folders, updates the catalog as the collection changes, and produces CBZ files and thumbnails. Your original downloads remain the source collection.

Use this service to prepare and maintain the library. Use Komga or h2hdb-opds to browse and read it. Readers receive read-only access to the published files.

Before you start

You need:

• Python 3.14 or newer.
• A nonempty download directory containing completed galleries with galleryinfo.txt metadata. Nested collection folders are supported.
• An H2HDB database, using SQLite or MariaDB. This release requires h2hdb>=0.41.0,<0.42.0 and schema epoch 3, version 8.
• For CBZ output, a separate writable library directory and enough disk space for image processing, one gallery's verified render input, database plans, and all output awaiting publication.

Ingest writes to the database and library. Do not point it at a library managed by another writer, and do not edit its generated files manually. The download and library directories must be distinct; neither may contain the other.

Install

Create a Python environment and install the package:

python3.14 -m venv .venv
source .venv/bin/activate
python -m pip install h2hdb-ingest
h2hdb-ingest --help

The package installs its compatible H2HDB and image-processing dependencies. Run the following commands from this activated environment. Replace the example paths with paths available to your service account or container.

Set up the database

For a new SQLite catalog, save this as core.json:

{
  "database": {
    "sql_type": "sqlite",
    "database": "/data/h2hdb/catalog.sqlite"
  }
}

Create /data/h2hdb first, then initialize the empty database:

mkdir -p /data/h2hdb
python -m h2hdb migrate --config core.json
python -m h2hdb check --config core.json

migrate creates a new schema or resumes its matching interrupted initialization. It does not upgrade arbitrary existing databases. Ingest itself never initializes or migrates the schema.

For MariaDB, set sql_type to mariadb and supply host, port, user, password, and database. Use the same connection in the ingest configuration below, with an account that can write to the catalog. See the H2HDB administration guide for database setup and upgrades.

Prepare the library

For CBZs and thumbnails, create these directories before starting ingest:

mkdir -p /data/h2hdb/library/current/acquisitions
mkdir -p /data/h2hdb/library/current/artwork
mkdir -p /data/h2hdb/library/.h2hdb-coordination

They must be real directories, not symlinks, on the same filesystem. For a container deployment, create them on the host before creating the reader containers. Ensure the ingest account can write to them. Ingest creates its own private .h2hdb-state directory; do not create or modify that directory yourself.

Mount these paths for the respective services:

Komga should receive only acquisitions/; artwork/ contains standalone thumbnails, not comic books. Keep .h2hdb-state private to ingest. Other processes must not modify the library, including its coordination directory.

Skip library preparation if you only want catalog metadata.

Configure ingest

Save this as ingest.json:

{
  "core": {
    "database": {
      "sql_type": "sqlite",
      "database": "/data/h2hdb/catalog.sqlite"
    }
  },
  "paths": {
    "download_path": "/data/hath-download",
    "library_path": "/data/h2hdb/library"
  }
}

The database settings match core.json, but ingest places them inside core. Use paths as seen by the ingest process. download_path must already exist and be nonempty. Set library_path to null to publish metadata without decoding images or producing CBZs and thumbnails.

Optional settings can be added to paths or a top-level resident object:

For image output, paths.render_policy accepts page_jpeg_quality (default 90), thumbnail_jpeg_quality (85), optimize (true), and resampler ("lanczos"). Quality values are integers from 0 through 95. Other resamplers are nearest, box, bilinear, hamming, and bicubic. Changing rendering settings can require galleries to be checked and artifacts rebuilt.

Automatic worker selection is capped at 16 and logged at startup. A Docker container uses the CPU availability visible inside the container; it cannot infer the host's macOS performance-core count. An explicit worker count provides control when the automatic choice does not suit your host.

JSON strings consisting exactly of ${ENV_NAME} can read environment variables, for example "password": "${H2HDB_PASSWORD}". Inline substitution such as "db-${INSTANCE}" is unsupported. Missing variables and unknown configuration fields cause startup to fail.

Run

Start the resident service to process existing downloads and watch for changes:

h2hdb-ingest --config ingest.json

For a single coordinated publication attempt:

h2hdb-ingest --config ingest.json --once

A one-shot run selects the complete inventory by default. An explicit admission limit still applies, and incomplete or changing galleries require a later turn. Use resident mode to continue processing that pending work. A one-shot run fails if it cannot complete a publication, for example because of lease contention or insufficient storage.

To require a nonempty first publication in a fresh, initialized catalog:

h2hdb-ingest-bootstrap --config ingest.json

Bootstrap refuses a catalog that already has a publication. It stops after the first nonempty catalog; start the resident service afterward to process the remaining galleries. The equivalent resident module command is python -m h2hdb_ingest --config ingest.json.

Use Ctrl+C or send SIGTERM for a graceful stop. Shutdown completes the current bounded step and resource cleanup. A full database audit already in progress must finish before a graceful stop can take effect.

What to expect

The first run inventories the source collection. By default, resident.publication_batch_galleries is null: ingest selects all eligible complete galleries in one turn before global analysis and publication. This prioritizes total catch-up time by avoiding repeated whole-collection analysis and validation after small additions. Readers see the result after the complete turn; database operations and filesystem pages retain their own bounded limits.

An explicit positive value preserves incremental publication. For example, 100 still admits at most 100 previously unknown galleries per publication, while applying changes and confirmed deletions to known galleries. It does not limit the inventory or total published books. Existing settings are not silently overridden: change an explicit 100 to null to select full-collection catch-up. This setting does not impose a time or disk-space budget.

Source observation retains immutable hashes and metadata, without copying every gallery's images until publication ends. Rendering rereads the original files and verifies them against that observation. Core keeps one gallery's verified render-input spool while preparing its artifact, then releases it. Changed or missing source bytes cannot be published under the earlier observation and must be observed again. Keep the source collection available throughout the turn.

Each source page validates its returned entries and lookahead against the initial index, while directory and metadata guards reject namespace or marker changes. A page is an intermediate observation, not a sealed snapshot. If another process modifies a file from an earlier page in place, a later page may return before the change is detected. The final completion-marker probe always performs a fresh audit of the complete observed entry set; both metadata-only and image-qualified observations are deferred before sealing if that audit fails. This replaces the previous full-gallery audit on every page, so entry stat probes grow linearly with the gallery size while preserving the final snapshot check.

After interruption, ingest first checks an unpublished, sealed source batch. Its root and policies must match, and each existing gallery's completion marker is reread in pages of at most 128 galleries. This recovery pass takes O(G + marker bytes) source work for G galleries in that batch: it reads SQL and galleryinfo.txt markers, without enumerating new galleries or deeply reading images. Missing marker evidence, changed markers, or a temporarily incomplete source requires a fresh scan. Other source or database errors remain failures. Newly downloaded galleries are picked up after a successfully resumed batch publishes; their arrival does not discard already committed analysis or prepared CBZs. Rendering still verifies live source bytes. A source failure requests fresh observation and can require a replacement batch.

The resumed batch has no fresh deferred/waiting inventory counts. INFO reports that a new inventory is pending, and resident mode immediately schedules it after publication instead of declaring initial catch-up complete. For Python callers, Core's prepare_source_resume(adapter, policy=...) requires the source adapter; source-root components alone are insufficient to authorize recovery. VNextIngestSourceSynchronizationResult, VNextIngestSynchronizationResult and ResidentIngestor.deferred_gallery_count can now report None for these unknown counts. Both result counts are None together, and inventory_scan_pending identifies that state. A one-shot run completes the resumed batch; use resident mode or another invocation to include newly arrived galleries.

During the first scan, each gallery's completed image checks and source facts are persisted before checking the next gallery. Restart performs a fresh marker inventory and reuses matching completed observations. At most the current gallery's unsealed checks are lost when markers remain stable; changed galleries must be checked again. These checkpoints retain metadata and hashes, not copies of the entire source image collection.

This update requires the Core schema-7-to-8 offline converter. It preserves the database contents and complete library, including CBZs and private state; do not clear the database or rebuild the library to perform this upgrade.

Keep galleryinfo.txt as the completion marker: finish writing a gallery's images before writing its metadata. Incomplete or changing galleries wait for a later turn while other complete galleries continue. A completed gallery is a discovery leaf, so galleries nested inside it are not discovered. Removing a completion marker temporarily retains the last published observation; confirmed removal of the gallery removes it from the source collection.

After startup, observed changes trigger work using the quiet and maximum-wait settings. An unchanged source does not trigger repeated full synchronization. The old periodic_scan_seconds setting is not accepted.

With library output enabled:

• Supported page suffixes are .avif, .bmp, .gif, .jpeg, .jpg, .png, and .webp, ignoring ASCII case. Other regular files are not rendered.
• Every accepted page becomes a JPEG. Animated GIFs use the first frame.
• A gallery with an undecodable page is excluded as a whole; ingest does not silently publish a book with missing pages. The logs identify the rejection. Repair the source and rewrite its completion marker to trigger another check.
• A selected gallery produces h2h-<gid>.cbz, with galleryinfo.txt and ordered pages. Page zero supplies the full-size cover, and a separate thumbnail has a maximum side of 320 pixels. A gallery without eligible pages has a metadata-only CBZ and no cover or thumbnail.

Output is limited to 4096 pages per gallery, 32 MiB per encoded JPEG page, 8192 pixels on the long side, 40 megapixels per output page, and 2,147,483,647 bytes per CBZ. Large source images are reduced without enlargement; source dimensions and file size alone do not exclude them. Some image codecs still need large memory buffers, so worker limits are not a fixed memory ceiling.

Ingest stores output under current/acquisitions/ and current/artwork/ using managed paths. Do not rename those files. Deduplication and spam decisions use the whole known collection, so adding galleries can replace or remove earlier published books; the book count need not increase with every batch.

Monitor and maintain the service

INFO logs show startup checks, the current activity, measured progress, and publication results. A rendered CBZ is not necessarily published yet; wait for publication completion before expecting it in a reader. An idle service emits no periodic progress message. Enable detailed diagnostics with "logger": {"level": "DEBUG"} inside core.

Each finished or failed source turn emits an INFO ingest_metric summary with its status, work generation, selected/waiting/deferred galleries, source rows, and logical bytes read. Adapter timings separate discovery, gallery indexing, metadata parsing, reads, hashes, and image qualification. These timings are inclusive: qualification can contain reads and hashes, so do not add them to estimate total wall time. Logical bytes include rereads and do not measure physical disk traffic. A killed process can leave a turn without a terminal summary; absence is not zero cost.

Source progress at INFO states whether selection covers all complete galleries or admits up to the explicit number of new galleries. Unbounded admission is not reported as a zero-gallery quota. Core's initial source_prepare record contains inventory size and observation_complete=false. Its terminal source_step record supplies admitted files/galleries, discovered/staged galleries, deferred/waiting counts, and completed inventory observation under the same correlation ID. Initial inventory size therefore must not be interpreted as already observed or admitted work.

Background inventory emits its own INFO scope=source_monitor operation=inventory summary, including status, completed marker rows, logical read bytes, discovery, read/hash work and total pass time including index reconciliation. These passes can overlap foreground ingest; do not add their elapsed times to foreground wall time or interpret a failed/interrupted pass as a completed inventory. The completion_marker_files_observed and completion_marker_bytes_observed counters identify completed observations of galleryinfo.txt, rather than images. Read-call counters also include EOF calls. Each inventory rereads the markers; the index uses their fingerprints to decide which galleries need foreground work. It does not skip marker reads based only on an unchanged filesystem timestamp.

From a development checkout, measure source reads and marker reuse with local, disposable real-image fixtures:

.venv/bin/python scripts/probe-source-io.py --galleries 2 --pages 2 \
  --codec jpeg --workers 1 --output /tmp/source-io.json
.venv/bin/python scripts/probe-source-backlog.py --inventory 129 \
  --output /tmp/source-backlog.json

The source matrix compares a new inventory, unchanged markers, a changed policy, and changed source bytes. Independent read counters check production telemetry. The backlog probe intentionally keeps an eight-gallery quota over three real publications, measuring how a fixed inventory affects repeated selection; it also checks cleanup and a subsequent work claim. Neither probe measures physical disk traffic or proves a NAS completion time. Run without concurrent builds or benchmarks when comparing wall times. The synthetic controller sensitivity tool probe-publication-budget.py --output /tmp/publication-budget.json separately compares first publication, target misses and total catch-up under fixed/per-gallery cost assumptions; its modeled times are not runtime measurements.

Successful batches also emit publication and artifact_totals summaries at INFO. The latter aggregates all completed render calls in that batch; individual artifact metrics remain available at DEBUG. Subtracting the aggregate render_archive.elapsed from publication elapsed excludes the entire archive renderer, including its input checks and final archive inspection, rather than only compression and packing. render_presentation separately measures thumbnail production. These are wall times for completed calls, not the sum of overlapping page worker durations. render_batches includes source verification, decode, resize and JPEG encoding; archive_page_write measures serial ZIP_STORED copying. The existing render_pages is inclusive and may overlap worker execution. Partial failed render calls have unknown remaining cost and must not be treated as zero.

The INFO archive totals also distinguish worker source verification, native decode/shrink, final resize, JPEG encoding, encoded-buffer copying/hashing, and main-thread ZIP metadata writes and close. worker_elapsed_sum adds elapsed worker durations and can exceed archive wall time. worker_thread_cpu_sum excludes other libvips native threads. The worker decoder pipeline includes scheduling/header work, decode/shrink and final resize; decoder input reads are inclusive suboperations. Do not add these overlapping measurements or subtract parallel JPEG worker durations from publication wall time. A compression-free counterfactual requires a separate controlled experiment.

INFO scope=adapter_io summaries correlate with the publication generation and report source opens, protection, layout checks, staging, journal transactions, lock waits, fsync and rename. Snapshots are cumulative, emitted at completed outer operation boundaries after 60 seconds and at completion or failure; subtract consecutive snapshots when computing interval costs. Inclusive wall time contains nested operations; exclusive wall time excludes them. Bytes are actual logical transfers, not device traffic. An operation still in progress is absent until it returns; the progress heartbeat identifies pending work. Core's publication summary separately attributes source copy/rehash, archive and presentation verification, and protection-boundary hashing. These Core and adapter measurements describe overlapping layers and must not be added together. Foreground source, publication, artifact totals and adapter I/O remain distinct summaries; the background monitor is a concurrent measurement.

Library maintenance emits a separate INFO scope=library_cleanup_io report at outcome transitions and at the configured progress interval. It includes startup, pre-claim and post-session calls, scratch cleanup, and failed or interrupted work. Pending totals are flushed at a successful claim, after a single-cycle call and at orderly shutdown; repeated idle polls do not each produce an INFO record. process_id plus observer_started_ns identifies one process-local cumulative series; use snapshot_sequence and subtract consecutive snapshots. elapsed_ns sums only active maintenance calls, excluding polling and other ingest stages. Outcome counts describe completed adapter results, not a new cleanup authority. Exclusive operation totals and the unattributed residual partition active time; inclusive timings overlap. An in-flight call appears after it returns. These records use INFO without changing the configured log level. Cleanup candidate selection and existence probes expose separate query time, calls and rows_returned; the latter is the result size, not SQLite rows examined. The manual cleanup cost tool separately counts actual SQLite VM instructions.

To exercise these boundaries with deterministic real JPEG files, public ingest, independent archive/raster checks, cleanup and a subsequent work claim:

.venv/bin/python scripts/probe-artifact-io.py --galleries 32 --pages 64 \
  --edge 512 --workers 4 --isolated --timeout 900 \
  --output /tmp/artifact-io.json

Only pass --isolated when other benchmarks and builds are stopped. The report records source hashes, raw INFO measurements and logical amplification; its local wall time is not a NAS throughput prediction. A small fixture with --fsync-delay-ms 2 --fsync-delay-kind directory or file injects a known delay at the adapter boundary to check attribution. It still executes the actual fsync and all publication checks; the artificial delay is not a device model. The library_cleanup_adapter report collects the production maintenance observer's per-call measurements, counting its journal, locks, fsync and logical byte operations. It includes scratch, pre-claim and post-session calls; its wall time overlaps the enclosing resident timings and must not be added to them. Publication INFO metrics retain their original scope. --cleanup-journal-delay-ms 2 injects a bounded delay only inside cleanup journal sessions to test that attribution; production logging and execution semantics are unchanged. Runtime or probe-source drift makes the probe exit nonzero with incomplete evidence while retaining measurements and oracle results. A completed diagnostic does not prove the source cost budgets or the full-library time objective.

Core records source actions, ingest permission checks and cleanup candidate checks separately. Long-operation progress identifies a pending connector call; completed SQL totals exclude that call until it returns. Publication completion, cleanup DONE, and the next successful work claim are distinct milestones. Compare all three when investigating a delay between batches.

Ingest periodically audits the database. A first start, unclean previous shutdown, changed validator, or due audit requires a full check. A recent successful audit and clean shutdown can allow a quick startup check. For an explicit full check:

python -m h2hdb check --config core.json

The default audit interval is the larger of seven days and 100 times the last full audit's duration. Advanced deployments can change resident.database_audit_minimum_interval_seconds and resident.database_audit_duration_multiplier; audits run between work sessions.

Keep free space available in the library filesystem for one gallery's verified source spool, page processing, a CBZ being written, and every prepared CBZ and thumbnail awaiting publication. Database and disk-backed discovery/analysis plans also require space. Removing the full-turn source-byte copy does not make total scratch or pending output constant-sized. Disk-full or quota errors keep work pending for retry instead of publishing incomplete files. Free space or increase the quota, then let resident mode retry. Do not manually remove private journal, staging, or coordination files to clear an error.

After an interruption, restart ingest with the same database and complete library. It resumes pending publication and cleanup. Readers may remain unavailable while an ACTIVATING marker or publication lock protects unfinished work. Unknown files, changed bytes, and unexpected symlinks are preserved and reported for inspection instead of being silently removed.

Upgrade or move an existing installation

Upgrade ingest and H2HDB together within their declared dependency ranges. Back up the database and the complete library before offline maintenance. The runtime and relocation command accept only the exact format-v5 private library journal. An existing format-v4 library needs the one-time offline conversion below; keep its CBZs, artwork and Core database. The converter adds a cleanup-selection index and atomically updates the version control table. It preserves the UUID, publication/protection/relocation facts, marker bytes and artifact files. It does not run a Core database migration or full database audit.

Stop ingest, OPDS, Komga and every other process that could modify or read the library, then run the matching checkout and installed wheel:

.venv/bin/python scripts/upgrade-library-journal-v4-to-v5.py \
  --library /data/h2hdb/library --consumers-stopped

An interrupted conversion can be rerun with the same command. SQLite commits the index and version together; an exact v5 replay verifies the journal without changing its facts. Foreign structures and v1–v3 are rejected. Normal startup does not convert an old journal or fall back to its old format.

For Docker, build the portable bundle from explicit, verified Ingest and Core wheels, passing the actual host bind source from your deployment Compose:

.venv/bin/python scripts/build-library-journal-upgrade-bundle.py \
  --wheel /path/to/ingest.whl --core-wheel /path/to/core.whl \
  --library-root /data/h2hdb/library \
  --output /tmp/h2hdb-journal-upgrade.tar.gz

Extract the bundle beside the deployment .env, then run its compose.yaml:

sudo docker compose --env-file .env \
  -f ./h2hdb-journal4-to5-docker-0.28.0/compose.yaml \
  run --rm --build --no-deps upgrade \
  --library /hentai/library --consumers-stopped

The container uses MEDIA_UID and MEDIA_GID from .env, requires no Core credentials, and has no network access while executing the converter. Building the image can download dependencies. Restart consumers only after conversion reports completion and all installed application versions are compatible.

An exact H2HDB schema-version-7 database can use the core project's one-time offline upgrade-source-collection-schema.py tool from historical Core 0.41.2 to reach schema version 8. That tool was removed from Core 0.42; use the Core 0.41.2 checkout and its matching environment, with all consumers stopped. Keep the database, CBZs, thumbnails, and complete private library state in place; this conversion changes database schema, not artifact bytes or the library layout. For schema 6, first use Core 0.40.0's upgrade-audit-schema.py and its environment to reach schema 7, then use the new converter. Leave all consumers stopped throughout both conversions and retain the original database/library backup. Other older schemas require a new database and catalog rebuild from the source; normal ingest startup does not convert them.

Legacy libraries containing current/hash-v1, .h2hdb-state/coordination, or activation journals version 1, 2, or 3 are rejected. Keep their files intact and rebuild into a fresh library paired with a fresh database. Preserve the original download tree; the relocation command does not upgrade these old layouts.

To move a current-format library to another path or filesystem:

1. Stop ingest, readers, and every other process that could modify the library.

2. Move the entire library, including .h2hdb-state and .h2hdb-coordination. Keep the existing core database.

3. Update all reader and writer paths or mounts to the new location.

4. Run verification with the new library path visible to the command:

   h2hdb-ingest-relocate --library /new/location/library
   

5. Restart ingest and readers only after the command reports completion.

Relocation verifies managed files and retains the library identity, catalog, and artifact contents. If interrupted, rerun the same command with the same destination to resume. It preserves incomplete or ambiguous files and reports the condition; it does not adopt or remove unrelated files. Copying only the CBZ files into an unrelated library does not preserve the database binding.

Troubleshooting

For background on the verification evidence and its limits, see Library reliability. For a problem report, include the package versions, database backend, command, and relevant error context in the issue tracker, with credentials and private paths removed.

Local source-cost acceptance

From a development checkout with its environment installed, run this manual, POSIX-only acceptance before drawing conclusions from a source optimization:

.venv/bin/python scripts/check-source-cost.py --output /tmp/source-cost.json

The report path must not already exist. Exit 0 means the declared work budgets are satisfied, 1 means measured work violates a budget, and 2 means evidence is incomplete or execution failed. A completed experiment is not necessarily a passing acceptance. The current duplicate source reads and repeated whole-gallery entry validation are expected to produce violations; the checker preserves these targets instead of relabelling existing costs as acceptable.

The default matrix uses deterministic PNG and JPEG fixtures with 127, 128, 129, and 512 pages, plus 1024-pixel and 2048-pixel images that exercise real decoding and disk-spool boundaries. It repeats fresh observations and metadata-only comparisons, measures completion-marker probes, and injects an interruption before a fresh source retry. It records logical read bytes, decoder calls, actual entry stat/revalidation work, CPU and wall time, and independently reconciles production telemetry. Fixture generation is timed separately. Runtime source digests must match the checkout; report provenance includes dependency digests and the commit.

The work targets are one source-byte pass per complete observation, at most eight entry-stat passes independent of gallery size, and exactly one qualification decode per page when enabled. They are improvement targets, not a claim that the present implementation already meets them. Qualification includes decoding and resizing before CBZ generation. Inclusive decoder, I/O and revalidation times overlap; they cannot be added to wall time.

This adapter acceptance does not exercise durable database reuse, full catalog analysis, publication, CBZ generation or library cleanup. Its marker-only case must not be described as an end-to-end unchanged-run result. The existing probe-source-io.py remains a diagnostic for actual published-baseline reuse. Local synthetic timings do not establish the full-library targets of 24 hours without CBZ work (12 hours desired), or seven days total including CBZ work.

Local library-cleanup cost acceptance

Run the isolated journal experiment separately from source or database benchmarks:

.venv/bin/python scripts/check-library-cleanup-cost.py \
  --full-inventory --output /tmp/library-cleanup-cost.json

This measures the two actual production cleanup-selection queries against the production SQLite journal schema. Default retained-token counts are 0, 127, 128, 129, 4,096 and 32,768; --full-inventory adds 264,092 tokens, representing two resources per 132,046 galleries. Each size has no-eligible and sparse-eligible cases, with three reset/replay cycles and exact selected-row checks.

The acceptance unit is SQLite VM instructions, including work that returns no rows. A fixed input-derived budget rejects repeated full-table scans. The same queries also run with a fixture-only index control and a forced scan: the index control must pass, and the forced scan must fail at large sizes. These controls verify that the checker distinguishes efficient access from the measured regression. Production journal v5 now has the matching partial index; the runtime must pass the original fixed budget independently of either control.

The report separates experiment status from acceptance.status and uses the same exit codes 0, 1, and 2 as source acceptance. Wall time includes the per-instruction measurement callback and is not a production latency estimate. The seeded engine fixture does not exercise filesystem locking, hashing, unlink, fsync, journal-open validation, or the complete public cleanup lifecycle. Core publication/cleanup acceptance is a separate command in an explicitly supplied Core checkout; neither isolated measurement establishes the full-library SLO.

License

GNU General Public License v3.0 only. See LICENSE.