BOLDcurator (Python)

Offline curation of BOLD specimen records. BOLDcurator searches, scores, ranks and BAGS-grades COI-5P barcode records against a local DuckDB snapshot of the BOLD public data package -- no BOLD API, no API key, no network once you have the snapshot. It is the offline counterpart of the BOLDcuratoR R Shiny app, developed in the same repository.

Curators: downloads, setup and an FAQ are on the project website. Or install with uv (pip/pipx work the same way):

uv tool install --python 3.11 "boldcurator[desktop]"
boldcurator install-shortcut        # Start menu / Applications / app menu

The rest of this README is for developers. Current state, open items and the decisions not to relitigate are in PROGRESS.md.

BOLD data is CC BY-SA 4.0 (Barcode of Life Data System, boldsystems.org); the app and every export carry that attribution.

What the app does

The desktop app (boldcurator-desktop, or boldcurator desktop) opens on a first-run setup screen until it has a snapshot, then shows these tabs:

How names are read. The species field is shown and exported exactly as BOLD has it; a name_status column says what kind of name it is, using BOLD's identification_rank:

A BIN is discordant (grade E for every species in it; Discordant on the BINs tab) when it holds two different species-level names -- strictly, so D. cf. plexippus beside D. plexippus counts -- or records from more than one genus, family or order, whatever rank they were identified to. A record identified only to the BIN's own genus changes nothing. | Phylogeny | Neighbour-joining tree (K2P, reference-anchored alignment) of one representative per selected BIN x country, with monophyly badges for grade-C species; warns above 500 tips, refuses above 1,000 | | Specimens | Every record, server-side paged and sortable |

Across the tables: per-record annotations (flag, curator note, corrected identification), "checked" records and a separate representative pick (auto-selected per BIN x country on a fresh search). Downloads: all / selected / annotated records (TSV), the BOLD curation report, FASTA, CSV, xlsx species and BIN analyses, and the tree as Newick.

The window is a native pywebview window where that works, else a Chromium --app window, else a browser tab (--window auto|native|browser-app|tab). Everything the app keeps is in ~/.boldcurator/: snapshots (each with a .meta.json provenance sidecar), config.json, sessions.sqlite and boldcurator.log.

Development setup

Python 3.10 or newer. From this python/ directory:

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -e ".[dev,desktop]"
python -m pytest tests/ -q

Or, with uv, exactly the versions pinned in uv.lock (what CI tests, pip-audit checks and the desktop builds ship): uv sync --locked --extra dev --extra desktop, then uv run python -m pytest tests/ -q. After changing dependencies in pyproject.toml, run uv lock and commit uv.lock -- CI fails on a stale lock.

Extras: gui (Shiny), desktop (Shiny, uvicorn, pywebview), bench (psutil, for benchmark's memory column), dev (pytest, psutil).

Command line

boldcurator <command>:

Also installed: boldcurator-desktop, boldcurator-build-snapshot, boldcurator-verify-snapshot and boldcurator-fetch-snapshot. Each tools/*.py script runs the same code from a checkout without installing.

Snapshots

Getting one

The published snapshot is on Zenodo under the concept DOI 10.5281/zenodo.22849515, which always resolves to the latest version (bold_snapshot_<date>.duckdb.gz, about 1.9 GB):

boldcurator fetch-snapshot --record 10.5281/zenodo.22849515 --out bold_snapshot.duckdb

The download runs as 4 parallel byte-range requests where the host allows it, identifies itself as BOLDcurator/<version>, waits out 429/503 as Retry-After asks, and verifies the checksum in the same pass that decompresses it. Re-running only downloads when the snapshot has changed.

Building one

You need the BOLD public data package (BOLD_Public.<date>.tsv.gz, ~3 GB compressed, login-gated at bench.boldsystems.org/index.php/datapackage). The extracted .tsv ingests 3-4x faster than the .gz.

# 1. header check only, in a fraction of a second -- run it first
python tools/build_snapshot.py --tsv BOLD_Public.2026-09-11.tsv --dry-run

# 2. optional trial build; recorded in the file as a partial build
python tools/build_snapshot.py --tsv BOLD_Public.2026-09-11.tsv --out trial.duckdb --limit 100000

# 3. the real build, then verify it from a fresh process
python tools/build_snapshot.py --tsv BOLD_Public.2026-09-11.tsv \
    --out bold_snapshot_2026-09-11.duckdb \
    --temp-dir /fast/local/disk --memory-limit 12GB --threads 4
python tools/verify_snapshot.py --snapshot bold_snapshot_2026-09-11.duckdb

Defaults keep COI-5P only, with sequences, and every source column except identifier_email. BIN, BAGS and the 500 bp quality threshold all assume COI-5P; --marker '' keeps every marker.

A failed build keeps its staging database, and the error says so: retry with --reuse-staging --overwrite to skip the ingest. On PowerShell, quote every path and close every quote -- an unclosed one leaves PowerShell silently waiting at a >> prompt, which looks exactly like a hang.

The 2026-09-11 build, for scale (Windows, 32 GB RAM): 33.6 GB extracted TSV, 20,164,595 COI-5P records, 546,861 taxa, 412,637 BINs; 7.95 GB file (1.9 GB gzipped); about 24 minutes from source, 7 reusing staging. 67% of records have no species-level name, so BAGS grades apply to about a third of the data.

tools/reorder_sequences.py rewrites an older snapshot's sequence table into specimen order (about two minutes, no re-ingest); verify fails a snapshot still in ingest order.

Testing

python -m pytest tests/ -q          # against a generated fixture snapshot
python parity/compare.py            # R-vs-Python parity; exit 1 on any unexplained difference

parity/compare.py runs the committed R reference outputs against the Python pipeline and writes parity/REPORT.md; every difference must map to a recorded divergence (see parity/README.md).

Drive the UI in a real browser before believing a UI change works -- every UI bug so far was invisible to the unit tests:

python tests/make_fake_package.py --out /tmp/fake.tsv.gz --rows 5000
python tools/build_snapshot.py --tsv /tmp/fake.tsv.gz --out /tmp/fake.duckdb
python -m boldcurator.cli gui --snapshot /tmp/fake.duckdb --port 8765 &
pip install playwright && playwright install chromium
python tools/drive_ui.py --out /tmp/shots

drive_ui.py makes 42 checks across every tab and exits non-zero on a failure. When adding one, match against the table, not the panel -- the flag <select> contains every flag name, so a panel-wide text match passes for an annotation that never rendered.

Benchmarking needs no real snapshot: tools/make_benchmark_snapshot.py builds a 20 M-row synthetic stand-in with the real shape in about six minutes (no sequences). boldcurator benchmark --snapshot FILE times each stage; a size-limit refusal is reported, not raised, and --max-fetch (default 250,000) skips a fetch too wide to hold in memory.

Performance

Measured on the real 7.95 GB snapshot:

Three rules came out of it; keep them:

• Resolve rows first, then fetch them. A BIN-expanded search is a disjunction DuckDB can't push into the scan, so plan_search finds matching rowids through narrow columns and fetch_planned joins that small set back -- which DuckDB can push down. sid doesn't work for this; rowid does.
• sequence is stored in specimen order, so a taxonomic result's sequences sit in a few row groups instead of all of them.
• No parallel scoring. Threads are slower (the regex loop holds the GIL); processes buy 1.4x for real complexity. On a realistic result, scoring is 15 ms.

Performance is closed: reopen it only with a measurement showing a curator waiting.

Architecture

src/boldcurator/
  config/      scoring criteria, rank ladder, continents, limits, paths, Zenodo DOI
  data/        snapshot schema, read-only connection, queries (plan/fetch, BIN expansion)
  core/        species rule, scoring, ranking, BAGS, BINs, selection, pipeline,
               paged table, BAGS grouping, summaries, phylogeny, reference alignment
  io/          annotations, exports, session persistence (SQLite)
  build/       snapshot builder, verifier, Zenodo/URL fetcher
  ui/          Shiny app and first-run setup -- the ONLY place a GUI framework is imported
  cli.py       the `boldcurator` command
  desktop.py   window strategies, config, local server
  launcher.py  `boldcurator-desktop` entry point
  shortcuts.py desktop shortcuts for uv/pip installs
tools/         build, verify, fetch, reorder, benchmark snapshot, drive-ui
tests/         pytest suite and the fake data-package generator
parity/        R-vs-Python comparison harness
packaging/     PyInstaller entry point, Windows installer, version stamping

• No GUI imports outside ui/ (tests/test_no_gui_dependency.py enforces it), so the core is testable headless and the CLI works without the gui extra.
• No widget ever receives a whole result. core/table.py pages, sorts and holds selections server-side; a page costs the same for 183 rows or two million. Sorting by a computed column (score, rank, grade) is refused, since it would mean scoring the whole result.
• Size policy. The specimen table works at any size. Species, BIN and BAGS screens are whole-result aggregates, computed lazily once and refused above DOWNLOAD_LIMITS["MAX_RECORDS"] (350,000) with an explanation.
• BAGS screens are problem navigators, not flat tables. Grade C (a species split across BINs) and E (a BIN shared by species) are each a set of separate species-BIN problems, so the screen walks them one at a time, and "Check this group" replaces the selection rather than adding to it. Records identified only to genus ride along by BIN membership, since they may be the misidentification.

Releases

Publish a GitHub release; CI builds the Windows, macOS and Linux downloads and publishes to PyPI, with the version taken from the tag. Never edit the version in pyproject.toml (it stays 0.0.0.dev0). Zenodo archives each release too (/.zenodo.json), and that archive is what the app's "a new version is available" check reads. How the builds work, the update check, and why each packaging flag is there: packaging/README.md.

Every release also carries a CycloneDX SBOM, a SHA256SUMS.txt and signed build-provenance attestations; PyPI files carry PEP 740 attestations from trusted publishing. How to check a download: Verifying your download. For IT reviewers -- network use, files, privileges: SECURITY_OVERVIEW.md. Reporting a vulnerability: SECURITY.md.