argus-quarry 0.2.0

Provenance-first acquisition of public-domain / CC0 portrait images for the Argus suite

Argus Quarry

Provenance-first acquisition of public-domain / CC0 images — the input stage of the Argus suite. The quarry digs up raw material: it downloads images from upstream archives and lands them, with full provenance and licensing, into a folder the rest of the suite already consumes (DATASET_DIR → /data/images).

Subjects are grouped into LoRA-training categories — identity (people), wardrobe (garments), setting (scenes/environments) and concept (styles/themes) — and everything lands sorted into <category>/<subject>/ subfolders. A category is independent of identity: a subject can be Mark Twain, a red dress, a modern kitchen, or cyberpunk.

It is deliberately lean — acquisition + provenance, nothing more. Quality scoring, near-duplicate detection, faces, embeddings and captioning are owned downstream by argus-curator and argus-lens; quarry never re-implements them.

Want a UI? Quarry is CLI-only by design (see DESIGN.md §9). The suite's web frontend — argus-vision-demo — surfaces the curation and captioning stages that consume quarry's output (e.g. its /curate view scans the <category>/<subject>/ tree quarry publishes).

argus-quarry (NEW)          argus-curator (:8101)        argus-lens (:8100)        argus-vision-demo
─ download  ─┐              ─ scan + score  ─┐           ─ caption ─┐              ─ web UI (:3000)
─ verify    ─┤   images +   ─ near-dup      ─┤  manifest ─ buckets ─┤   dataset    ─ /curate
─ provenance┤   provenance  ─ face-cluster  ─┤           ─ (ident/ ─┤   → LoRA     ─ caption
─ SHA256    ─┴───────────►  ─ select+export ─┴──────────►  wardrobe)─┘
   /data/images (DATASET_DIR) ─────────────────────────────────────────────────►

See DESIGN.md for the full rationale and phased plan.

Why it exists

• Provenance-first. Every image carries its source URL, landing page, licence and attribution. A record with no accepted licence (PD / CC0) is quarantined, never landed.
• Idempotent. Exact-dedup by SHA256 (UNIQUE in the DB) and a status lifecycle mean reruns resume partials and never duplicate bytes.
• Bounded. A per-file resolution/size cap and a total-archive GB budget keep the pool predictable; the full-resolution URL is always retained for later re-fetch.
• Source-independent. Every downloader yields the same SourceRecord, so the pipeline never learns which archive a file came from.
• Category-sorted. Subjects carry a category (identity / wardrobe / setting / concept) and land under <category>/<subject>/, so one pool serves every LoRA workflow.

Install

pip install argus-quarry            # library + downloaders
pip install "argus-quarry[cli]"     # + the argus-quarry command
pip install "argus-quarry[phash]"   # + opportunistic perceptual-hash metadata

For development the suite uses uv (works on PEP 668 "externally managed" system Pythons):

make dev                            # uv venv + editable install (dev + cli extras)
# or, manually:
uv venv && uv pip install -e ".[dev,cli]"

Quickstart

# Inspect the curated subject seeds (all categories, or one)
argus-quarry subjects
argus-quarry subjects --category wardrobe

# Fetch from Wikimedia Commons into the raw pool, then publish a curator-ready,
# CC0/PD-only tree into $DATASET_DIR (symlinks by default). Omit --category to
# harvest every category (identity + wardrobe + setting + concept).
argus-quarry run --source commons --limit 20 --export --licence CC0,PD
argus-quarry run --category concept --limit 20 --export   # just one category

# Or split the two stages
argus-quarry fetch --source commons --limit 20
argus-quarry export --dest ./data --licence CC0,PD   # add --copy to avoid symlinks

# Inspect what you have
argus-quarry stats
argus-quarry list --category setting --licence CC0
argus-quarry verify              # re-check files decode + match recorded SHA256

Installed into a uv venv? Prefix commands with uv run (e.g. uv run argus-quarry stats) or source .venv/bin/activate first.

CLI

Command	What it does
run	Fetch into the raw pool, then (optionally) publish — the compose entrypoint
fetch	Download candidates into the raw pool (no publish)
export	Publish a filtered <category>/<subject>/ tree into DATASET_DIR (symlink / --copy)
list	List landed photographs with provenance (filter by source / licence / category / subject)
stats	Counts by status / category / source / licence + raw-pool size
verify	Re-check landed files exist, decode, and match their recorded SHA256
subjects	Show the subject seed(s) downloaders harvest around (filter by --category)

run, fetch, export and list all accept --category (identity / wardrobe / setting / concept); with none given they span every category.

Layout produced

Quarry fetches into a raw pool it fully owns ($QUARRY_HOME, a sibling side-car dir), then export publishes a clean tree into DATASET_DIR:

$QUARRY_HOME/                          # side-car state — NEVER scanned by curator
├── images/
│   ├── identity/Albert_Einstein/…      # the raw pool, sorted by <category>/<subject>/
│   ├── wardrobe/Red_dress/…
│   ├── setting/Modern_kitchen/…
│   └── concept/Cyberpunk/…
├── metadata/portraits.sqlite          # provenance DB (subjects + photographs)
├── cache/  logs/

$DATASET_DIR/                          # published, curator-ready view (via export)
├── identity/Albert_Einstein/…          # symlinks (default) or copies into the pool
└── wardrobe/Red_dress/…

Provenance model

A single SQLite database (portraits.sqlite, WAL mode) with two tables:

• subjects — name · category · wikidata_id · birth_year · death_year · occupation (the identity-only columns stay NULL for wardrobe / setting / concept subjects)
• photographs — category · title · photographer · year · source · source_url · licence · attribution · width · height · file_size · filename · **sha256 (UNIQUE)** · phash · remote_url · status · downloaded_at

sha256 is the exact-dedup key (idempotent reruns); status (pending → downloading → complete, plus duplicate / quarantined / failed) tracks resumability. phash is recorded opportunistically and is informational only — it never drives dedup here (that's argus-curator's job).

Configuration

Copy .env.example to .env. Key knobs:

Env	Default	Meaning
QUARRY_HOME	./quarry	Raw pool + DB + cache + logs (side-car dir)
QUARRY_MAX_GB	40	Total raw-pool ceiling; 0 = unlimited
COMMONS_USER_AGENT	descriptive default	Polite UA (Commons/LoC expect one)
DATASET_DIR	./data	Published view export writes into

Suite integration

Quarry ships a gallery profile in the suite's argus-vision-demo compose.yaml. It's a run-to-completion job: fetch into the pool, publish into DATASET_DIR, then let curator/lens (and the web UI's /curate view) consume the result.

docker compose --profile gallery up --build   # fetch → pool → publish DATASET_DIR
docker compose --profile curator up --build    # then curate the published images

The published tree symlinks back into QUARRY_HOME/images. For those links to resolve inside the curator/lens containers, mount QUARRY_HOME read-only there too, or run export --copy.

Sources

Source	Status
Wikimedia Commons	Phase 1 (implemented)
Library of Congress, Smithsonian, Rijksmuseum, LAC (Karsh allow-list)	Phase 2
Europeana, Flickr Commons (strict per-record rights)	Phase 3

New sources register in downloaders/ behind a common Downloader contract, so adding one never touches ingest, storage, or export.

Development

make lint     # ruff
make test     # pytest
make check    # lint + test + build

Related projects

• argus-vision-demo — the suite's Next.js web UI (captioning + /curate).
• argus-curator — training-suitability scoring, near-dup dedup, face clustering.
• argus-lens — intent-aware, multi-model captioning.

Licence

MIT — see LICENSE. Note: the MIT licence covers this software, not the images it downloads. Image licences are recorded per-record and enforced at ingest.