Argus Quarry

Provenance-first acquisition of public-domain / CC0 images — the input stage of the Argus suite. The quarry digs up raw material: it downloads images from upstream archives and lands them, with full provenance and licensing, into a folder the rest of the suite already consumes (DATASET_DIR → /data/images).

Subjects are grouped into LoRA-training categories — identity (people), wardrobe (garments), setting (scenes/environments) and concept (styles/themes) — and everything lands sorted into <category>/<subject>/ subfolders. A category is independent of identity: a subject can be Mark Twain, a red dress, a modern kitchen, or cyberpunk.

It is deliberately lean — acquisition + provenance, nothing more. Quality scoring, near-duplicate detection, faces, embeddings and captioning are owned downstream by argus-curator and argus-lens; quarry never re-implements them.

Want a UI? Quarry is CLI-only by design (see DESIGN.md §9). The suite's web frontend — argus-studio — surfaces the curation and captioning stages that consume quarry's output (e.g. its /curate view scans the <category>/<subject>/ tree quarry publishes).

argus-quarry (NEW)          argus-curator (:8101)        argus-lens (:8100)        argus-studio
─ download  ─┐              ─ scan + score  ─┐           ─ caption ─┐              ─ web UI (:3000)
─ verify    ─┤   images +   ─ near-dup      ─┤  manifest ─ buckets ─┤   dataset    ─ /curate
─ provenance┤   provenance  ─ face-cluster  ─┤           ─ (ident/ ─┤   → LoRA     ─ caption
─ SHA256    ─┴───────────►  ─ select+export ─┴──────────►  wardrobe)─┘
   /data/images (DATASET_DIR) ─────────────────────────────────────────────────►

See DESIGN.md for the full rationale and phased plan.

Why it exists

• Provenance-first. Every image carries its source URL, landing page, licence and attribution. A record with no accepted licence (PD / CC0) is quarantined, never landed.
• Idempotent. Exact-dedup by SHA256 (UNIQUE in the DB) and a status lifecycle mean reruns resume partials and never duplicate bytes.
• Bounded. A per-file resolution/size cap and a total-archive GB budget keep the pool predictable; the full-resolution URL is always retained for later re-fetch.
• Source-independent. Every downloader yields the same SourceRecord, so the pipeline never learns which archive a file came from.
• Category-sorted. Subjects carry a category (identity / wardrobe / setting / concept) and land under <category>/<subject>/, so one pool serves every LoRA workflow.

Install

pip install argus-quarry            # library + downloaders
pip install "argus-quarry[cli]"     # + the argus-quarry command
pip install "argus-quarry[phash]"   # + opportunistic perceptual-hash metadata
pip install "argus-quarry[server]"  # + the read-only provenance HTTP API (serve)

For development the suite uses uv (works on PEP 668 "externally managed" system Pythons):

make dev                            # uv venv + editable install (dev + cli extras)
# or, manually:
uv venv && uv pip install -e ".[dev,cli]"

Quickstart

# Inspect the curated subject seeds (all categories, or one)
argus-quarry subjects
argus-quarry subjects --category wardrobe

# Fetch from Wikimedia Commons into the raw pool, then publish a curator-ready,
# CC0/PD-only tree into $DATASET_DIR (symlinks by default). Omit --category to
# harvest every category (identity + wardrobe + setting + concept).
argus-quarry run --source commons --limit 20 --export --licence CC0,PD
argus-quarry run --category concept --limit 20 --export   # just one category

# Or split the two stages
argus-quarry fetch --source commons --limit 20
argus-quarry export --dest ./data --licence CC0,PD   # add --copy to avoid symlinks

# Inspect what you have
argus-quarry stats
argus-quarry list --category setting --licence CC0
argus-quarry verify              # re-check files decode + match recorded SHA256

Installed into a uv venv? Prefix commands with uv run (e.g. uv run argus-quarry stats) or source .venv/bin/activate first.

CLI

run, fetch, export and list all accept --category (identity / wardrobe / setting / concept); with none given they span every category.

Layout produced

Quarry fetches into a raw pool it fully owns ($QUARRY_HOME, a sibling side-car dir), then export publishes a clean tree into DATASET_DIR:

$QUARRY_HOME/                          # side-car state — NEVER scanned by curator
├── images/
│   ├── identity/Albert_Einstein/…      # the raw pool, sorted by <category>/<subject>/
│   ├── wardrobe/Red_dress/…
│   ├── setting/Modern_kitchen/…
│   └── concept/Cyberpunk/…
├── metadata/portraits.sqlite          # provenance DB (subjects + photographs)
├── cache/  logs/

$DATASET_DIR/                          # published, curator-ready view (via export)
├── identity/Albert_Einstein/…          # symlinks (default) or copies into the pool
└── wardrobe/Red_dress/…

Provenance model

A single SQLite database (portraits.sqlite, WAL mode) with two tables:

• subjects — name · category · wikidata_id · birth_year · death_year · occupation (the identity-only columns stay NULL for wardrobe / setting / concept subjects)
• photographs — category · title · photographer · year · source · source_url · licence · attribution · width · height · file_size · filename · **sha256 (UNIQUE)** · phash · remote_url · status · downloaded_at

sha256 is the exact-dedup key (idempotent reruns); status (pending → downloading → complete, plus duplicate / quarantined / failed) tracks resumability. phash is recorded opportunistically and is informational only — it never drives dedup here (that's argus-curator's job).

Configuration

Copy .env.example to .env. Key knobs:

HTTP server

A tiny read-only provenance API (FastAPI) over the raw pool — no mutation endpoints, ever. It powers the argus-studio /gallery view, the same way /curate talks to argus-curator on :8101.

pip install "argus-quarry[server]"          # fastapi + uvicorn
argus-quarry serve --host 0.0.0.0 --port 8102 --cors

The published image ships the server extra and serves by default on :8102:

docker run --rm -p 8102:8102 -v "$PWD/quarry:/data/quarry" \
  -e QUARRY_HOME=/data/quarry ghcr.io/smk762/argus-quarry:latest

The entrypoint is still argus-quarry, so any acquisition subcommand works by overriding the command (docker run ... ghcr.io/smk762/argus-quarry fetch …).

The pool root comes from $QUARRY_HOME (default ./quarry), exactly like every other command — the compose service just sets QUARRY_HOME=/data/quarry.

Serving never mutates the provenance data — on any mount. The DB is opened through a mode=ro SQLite URI, which still tracks a concurrently running fetch through the WAL; the server never migrates, relocates, or writes a row, so pointing it at an empty or unmigrated pool reports 503 rather than quietly building one.

It is not wholly side-effect-free on a writable mount, though: mode=ro lets SQLite create the -wal/-shm sidecars it uses to follow the WAL, and a read-only connection can never checkpoint or unlink them, so those two files persist beside the DB (reused across opens, not accumulated — two files, not unbounded growth; issue #9). Mounting the pool :ro avoids them entirely:

docker run --rm -p 8102:8102 -v "$PWD/quarry:/data/quarry:ro" \
  -e QUARRY_HOME=/data/quarry ghcr.io/smk762/argus-quarry:latest

A read-only directory also stops SQLite creating the -shm sidecar mode=ro needs, so on that mount quarry falls back to immutable=1, which needs no sidecars. immutable=1 cannot see a -wal, so it is used only when there is none: a pool snapshotted mid-write, or left behind by a killed fetch, answers 503 instead of silently serving stale counts. Checkpoint it (sqlite3 portraits.sqlite 'PRAGMA wal_checkpoint(TRUNCATE)'), or mount the pool writable, and it serves normally.

The image runs as a non-root user (uid 10001), so a bind-mounted $QUARRY_HOME must be readable by that uid to serve, and writable by it for the acquisition subcommands (fetch, run) — e.g. chown -R 10001:10001 quarry/ on the host. This keeps the read-only sidecars above, and anything the acquisition commands land, owned by an unprivileged uid rather than root (issues #8/#9).

The read-only CLI commands (stats, list, export, verify without --repair) open the pool the same way, so they work against a :ro mount too.

Liveness and readiness are separate. GET /health answers 200 whenever the process is up — it never touches the DB — so a container mounted at an empty QUARRY_HOME that is seeded later still comes up live. GET /ready opens the pool and answers 503 with {"status": "unavailable", "service": "argus-quarry", "database": "..."} when it cannot be served (missing, unmigrated, or an unreadable WAL); the database value is a single generic reason ("provenance database unavailable"), deliberately not a per-cause code, so it never leaks a filesystem path. Point an orchestrator's readiness probe at /ready to shed a container that is up but has no data to serve, and its liveness probe at /health.

Suite integration

Quarry ships a gallery profile in the suite's argus-studio compose.yaml. It's a run-to-completion job: fetch into the pool, publish into DATASET_DIR, then let curator/lens (and the web UI's /curate view) consume the result.

docker compose --profile gallery up --build   # fetch → pool → publish DATASET_DIR
docker compose --profile curator up --build    # then curate the published images

The published tree symlinks back into QUARRY_HOME/images. For those links to resolve inside the curator/lens containers, mount QUARRY_HOME read-only there too, or run export --copy.

Sources

New sources register in downloaders/ behind a common Downloader contract, so adding one never touches ingest, storage, or export.

Development

make lint     # ruff
make test     # pytest
make check    # lint + test + build

Related projects

• argus-studio — the suite's Next.js web UI (captioning + /curate).
• argus-curator — training-suitability scoring, near-dup dedup, face clustering.
• argus-lens — intent-aware, multi-model captioning.

Licence

MIT — see LICENSE. Note: the MIT licence covers this software, not the images it downloads. Image licences are recorded per-record and enforced at ingest.