argus-quarry 0.1.0

Provenance-first acquisition of public-domain / CC0 portrait images for the Argus suite

Argus Quarry

Provenance-first acquisition of public-domain / CC0 portraits — the input stage of the Argus suite. The quarry digs up raw material: it downloads images from upstream archives and lands them, with full provenance and licensing, into a folder the rest of the suite already consumes (DATASET_DIR → /data/images).

It is deliberately lean — acquisition + provenance, nothing more. Quality scoring, near-duplicate detection, faces, embeddings and captioning are owned downstream by argus-curator and argus-lens; quarry never re-implements them.

Want a UI? Quarry is CLI-only by design (see DESIGN.md §9). The suite's web frontend — argus-vision-demo — surfaces the curation and captioning stages that consume quarry's output (e.g. its /curate view scans the Person_Name/ tree quarry publishes).

argus-quarry (NEW)          argus-curator (:8101)        argus-lens (:8100)        argus-vision-demo
─ download  ─┐              ─ scan + score  ─┐           ─ caption ─┐              ─ web UI (:3000)
─ verify    ─┤   images +   ─ near-dup      ─┤  manifest ─ buckets ─┤   dataset    ─ /curate
─ provenance┤   provenance  ─ face-cluster  ─┤           ─ (ident/ ─┤   → LoRA     ─ caption
─ SHA256    ─┴───────────►  ─ select+export ─┴──────────►  wardrobe)─┘
   /data/images (DATASET_DIR) ─────────────────────────────────────────────────►

See DESIGN.md for the full rationale and phased plan.

Why it exists

• Provenance-first. Every image carries its source URL, landing page, licence and attribution. A record with no accepted licence (PD / CC0) is quarantined, never landed.
• Idempotent. Exact-dedup by SHA256 (UNIQUE in the DB) and a status lifecycle mean reruns resume partials and never duplicate bytes.
• Bounded. A per-file resolution/size cap and a total-archive GB budget keep the pool predictable; the full-resolution URL is always retained for later re-fetch.
• Source-independent. Every downloader yields the same PortraitRecord, so the pipeline never learns which archive a file came from.

Install

pip install argus-quarry            # library + downloaders
pip install "argus-quarry[cli]"     # + the argus-quarry command
pip install "argus-quarry[phash]"   # + opportunistic perceptual-hash metadata

For development the suite uses uv (works on PEP 668 "externally managed" system Pythons):

make dev                            # uv venv + editable install (dev + cli extras)
# or, manually:
uv venv && uv pip install -e ".[dev,cli]"

Quickstart

# Inspect the curated people seed
argus-quarry people

# Fetch portraits from Wikimedia Commons into the raw pool, then publish
# a curator-ready, CC0/PD-only tree into $DATASET_DIR (symlinks by default)
argus-quarry run --source commons --limit 20 --export --licence CC0,PD

# Or split the two stages
argus-quarry fetch --source commons --limit 20
argus-quarry export --dest ./data --licence CC0,PD   # add --copy to avoid symlinks

# Inspect what you have
argus-quarry stats
argus-quarry list --licence CC0
argus-quarry verify              # re-check files decode + match recorded SHA256

Installed into a uv venv? Prefix commands with uv run (e.g. uv run argus-quarry stats) or source .venv/bin/activate first.

CLI

Command	What it does
run	Fetch into the raw pool, then (optionally) publish — the compose entrypoint
fetch	Download candidates into the raw pool (no publish)
export	Publish a filtered Person_Name/ tree into DATASET_DIR (symlink / --copy)
list	List landed photographs with provenance (filter by source / licence / person)
stats	Counts by status / source / licence + raw-pool size
verify	Re-check landed files exist, decode, and match their recorded SHA256
people	Show the people seed downloaders harvest around

Layout produced

Quarry fetches into a raw pool it fully owns ($QUARRY_HOME, a sibling side-car dir), then export publishes a clean tree into DATASET_DIR:

$QUARRY_HOME/                    # side-car state — NEVER scanned by curator
├── images/Albert_Einstein/…     # the raw pool (every byte quarry landed)
├── metadata/portraits.sqlite    # provenance DB (people + photographs)
├── cache/  logs/

$DATASET_DIR/                    # published, curator-ready view (via export)
└── Albert_Einstein/…            # symlinks (default) or copies into the pool

Provenance model

A single SQLite database (portraits.sqlite, WAL mode) with two tables:

• people — name · wikidata_id · birth_year · death_year · occupation
• photographs — title · photographer · year · source · source_url · licence · attribution · width · height · file_size · filename · **sha256 (UNIQUE)** · phash · remote_url · status · downloaded_at

sha256 is the exact-dedup key (idempotent reruns); status (pending → downloading → complete, plus duplicate / quarantined / failed) tracks resumability. phash is recorded opportunistically and is informational only — it never drives dedup here (that's argus-curator's job).

Configuration

Copy .env.example to .env. Key knobs:

Env	Default	Meaning
QUARRY_HOME	./quarry	Raw pool + DB + cache + logs (side-car dir)
QUARRY_MAX_GB	40	Total raw-pool ceiling; 0 = unlimited
COMMONS_USER_AGENT	descriptive default	Polite UA (Commons/LoC expect one)
DATASET_DIR	./data	Published view export writes into

Suite integration

Quarry ships a gallery profile in the suite's argus-vision-demo compose.yaml. It's a run-to-completion job: fetch into the pool, publish into DATASET_DIR, then let curator/lens (and the web UI's /curate view) consume the result.

docker compose --profile gallery up --build   # fetch → pool → publish DATASET_DIR
docker compose --profile curator up --build    # then curate the published images

The published tree symlinks back into QUARRY_HOME/images. For those links to resolve inside the curator/lens containers, mount QUARRY_HOME read-only there too, or run export --copy.

Sources

Source	Status
Wikimedia Commons	Phase 1 (implemented)
Library of Congress, Smithsonian, Rijksmuseum, LAC (Karsh allow-list)	Phase 2
Europeana, Flickr Commons (strict per-record rights)	Phase 3

New sources register in downloaders/ behind a common Downloader contract, so adding one never touches ingest, storage, or export.

Development

make lint     # ruff
make test     # pytest
make check    # lint + test + build

Related projects

• argus-vision-demo — the suite's Next.js web UI (captioning + /curate).
• argus-curator — training-suitability scoring, near-dup dedup, face clustering.
• argus-lens — intent-aware, multi-model captioning.

Licence

MIT — see LICENSE. Note: the MIT licence covers this software, not the images it downloads. Image licences are recorded per-record and enforced at ingest.