dataset-desk 0.2.0

English | 日本語 | 中文 | Español | Français | हिन्दी | Italiano | Português (BR)

A desk for making and reviewing visual datasets: find the copies, the near-copies and the odd ones out,
search a set by words or by picture, and check that every image's description really describes it.

Why

A style LoRA learns from its training images and its captions. Two quiet defects spoil both:

• Repeats. The same picture saved twice, or two renders a few pixels apart, teach one image twice.
• Descriptions that aren't descriptions. A caption copied from the generation prompt, or one ruling pasted onto 54 images, tells the model nothing about the picture. Worse, if that text is later used to test search, it measures recall of the prompt, not understanding of the image.

dataset-desk finds both, on a project folder, without changing it. It writes a report and a review queue for a person to act on. Nothing is removed automatically.

What it does

The thresholds come from the studio's calibration on its own data. Every one is marked provisional in the code, with the measurement it came from:

• Images: on a DINOv2 index, a pair is a duplicate when (cosine ≥ 0.94 and the mirror-aware dHash is within 44 bits) or cosine ≥ 0.985. Other indexes use the dHash within 26 bits, plus cosine ≥ 0.95.
• Text: 0.98 with bge-m3, and 0.95 for code with EmbeddingGemma 2. Any other model needs --semantic.

Two renders of the same object from different camera angles are never duplicates. The view comes from each record's provenance.asset_id and render_id, or from file names like <asset>__y<yaw>_e<elev>.

The description gate runs without any model. A description is flagged when it:

• echoes its prompt: it equals the prompt or sits inside it, shares a run of 6+ words with it, or 90%+ of its words (80%+ of its content words) appear in it;
• wraps the record's subject line: a 4+ word canon.subject appears as a contiguous run of its words;
• is too short: under 8 words or under 5 content words;
• duplicates another description in the set: identical once punctuation is removed, or a content-word Jaccard of 0.85 or more;
• is too long for SigLIP2: more than 64 tokens (the one check that needs a tokenizer; unchecked records are never marked as passing).

Flagged records go to description-review.jsonl and to a blind worksheet: each flagged image beside 3–5 visually nearest siblings, with no prompt, no subject line and no old text in view, so the person rewriting it describes the picture. An optional drafter (plain-sight, Florence-2) can write a first draft from the pixels alone; drafts are recorded as captioner_assisted, and a person edits them.

The look channel is 30 numbers per image (palette, tone, and texture at native resolution), so "this one's rendering is off" shows separately from "this one's subject is odd".

Install

pip install dataset-desk

That gives the lite tier: numpy and Pillow only, no torch. The duplicate checks, the look channel, the description gate and the report all work in it (--backend none).

For the embedding index (search, coverage, semantic duplicates, the description fit check), add the model libraries:

pip install "dataset-desk[models]"

The models are fetched from Hugging Face at pinned revisions the first time they're used, then load from your cache. Each task has a default, and --backend picks another:

EmbeddingGemma 2 is used the way its model card says: a task prefix per command, text-only loading for text, and --dim 768|512|256. Set HF_HUB_OFFLINE=1 to forbid downloads. The plain-sight drafter is always offline: it never downloads.

Runs on: Python 3.11 and 3.12, on Windows and Linux. CPU by default, with every thread pool capped (4 threads unless --threads says otherwise), and a model run drops the process to below-normal priority, so it can share a machine.

Usage

# The lite pass: no model, no GPU
dataset-desk index  --project path/to/project --backend none --out desk-out/project
dataset-desk report --project path/to/project --backend none --out desk-out/project

# With an embedding index
dataset-desk index    --project path/to/project --backend siglip2 --out desk-out/project
dataset-desk search   --project path/to/project --backend siglip2 --out desk-out/project --text "a squat bog creature"
dataset-desk coverage --project path/to/project --backend siglip2 --out desk-out/project --subjects wanted.txt
dataset-desk report   --project path/to/project --backend siglip2 --out desk-out/project --subjects wanted.txt

# Image duplicates with the calibrated DINOv2 rule
dataset-desk index --project path/to/project --backend dinov2 --out desk-out/project
dataset-desk dupes --project path/to/project --out desk-out/project

# Text and code files
dataset-desk text-index  --source path/to/folder --out desk-out/texts               # bge-m3, for text
dataset-desk text-dupes  --source path/to/folder --out desk-out/texts
dataset-desk text-index  --source path/to/folder --out desk-out/texts --task code   # EmbeddingGemma 2, for code
dataset-desk text-search --source path/to/folder --out desk-out/texts --task code --text "where the cache is written"

dataset-desk --help lists every flag. Exit codes: 0 ok, 1 a user error or a refusal, 2 an unexpected error. Errors print as error <CODE>: <message> with a hint: line; --debug adds the stack trace.

The project layout it reads is style-dataset-lab's:

project/
  records/*.json                  one per image: judgment (status, explanation), canon (subject), provenance (prompt)
  outputs/<wave>/*.png            the images; a wave's generation.json and _curation.json are read when present

A project with no records/ works too, as long as the images sit in folders under outputs/ (files at the top of outputs/ are skipped as contact sheets): every image is "unrecorded", and the duplicate and look checks still run.

Threat model and trust

• What it reads: the project folder you name with --project: images, records/*.json, wave metadata, and the generation data embedded in PNG files. The text commands read the --source folder or --rnd-data checkout you name. They refuse rnd-data's private companion and blob stores, and they print relative paths only.
• What it writes: only under --out: its scan cache, embedding caches, the report, description-review.jsonl and the worksheet. It refuses an --out inside the project (PERM_OUT_INSIDE_PROJECT), so a project is never modified.
• What it never touches: anything outside those two folders. It deletes nothing, anywhere.
• Network: the lite tier never uses it. A model backend fetches its pinned weights from Hugging Face on first use, unless HF_HUB_OFFLINE=1 is set; the drafter loads with HF_HUB_OFFLINE=1 and local_files_only=True every time. No model runs remote code.
• Permissions: an ordinary user account. A GPU needs --device plus a --grant id, so a shared card isn't taken by accident. Before loading a model, a VRAM governor checks that memory already in use plus the model's measured need fits under --vram-budget (default 26,624 MiB). It never evicts anything; --wait polls until the load fits.
• Secrets and telemetry: it reads no credentials, and it collects and sends no telemetry.
• Guarded steps: on projects locked for a running benchmark, description embeddings and any model-based acceptance of descriptions are refused unless --allow-multimodal-pool is passed, so the benchmark's queries aren't shaped by the models it measures.

Report a vulnerability as described in SECURITY.md.

Status

v0.2.0, a public prototype. The code is tested offline: 188 tests and 95% line coverage, with small stand-ins for the model libraries.

• Calibration: the studio's runs on its own data, on an RTX 5090, set the duplicate thresholds and the VRAM figures. They're provisional, and they rest on one pool per kind. The code thresholds rest on 17 planted copies. The method and the data are in newsletter issue 16.
• Large projects: duplicates, outliers and the look channel work in bounded blocks. On synthetic sets, the near-duplicate search and the look channel take minutes at 100,000 images, where they took hours or ran out of memory. The description gate is about 6× faster. Its Jaccard check still slows sharply on very large sets that share many words, and that design question is open.
• Expect the command line to change before 1.0.

The design record, its research grounding and its review history live in the studio's research library: rnd / experiments / dataset-desk.

Development

git clone https://github.com/mcp-tool-shop-org/dataset-desk && cd dataset-desk
python -m pip install -e ".[dev]"
bash verify.sh                       # lint, the tests, and a lite-mode smoke run
python -m coverage run -m unittest discover -s tests && python -m coverage report

CI holds line coverage at 90% or more.

License

MIT. The models it can load are not part of this package, and each carries its own licence; read the model card before you use one.

Built by MCP Tool Shop