BOLDcurator (Python)

Offline curation of BOLD specimen records. BOLDcurator searches, scores, ranks and BAGS-grades COI-5P barcode records against a local DuckDB snapshot of the BOLD public data package -- no BOLD API, no API key, no network once you have the snapshot. It is the offline counterpart of the BOLDcuratoR R Shiny app, developed in the same repository.

Curators: downloads, setup and an FAQ are on the project website. Or install with uv (pip/pipx work the same way):

uv tool install --python 3.11 "boldcurator[desktop]"
boldcurator install-shortcut        # Start menu / Applications / app menu

The rest of this README is for developers. Current state, open items and the decisions not to relitigate are in PROGRESS.md.

BOLD data is CC BY-SA 4.0 (Barcode of Life Data System, boldsystems.org); the app and every export carry that attribution.

What the app does

The desktop app (boldcurator-desktop, or boldcurator desktop) opens on a first-run setup screen until it has a snapshot, then opens on the Guide:

How names are read. The species field is shown and exported exactly as BOLD has it; a name_status column says what kind of name it is, using BOLD's identification_rank:

A BIN is discordant (grade E for every species in it; Discordant on the BINs tab) when it holds two different species-level names -- strictly, so D. cf. plexippus beside D. plexippus counts -- or records from more than one genus, family or order, whatever rank they were identified to. A record identified only to the BIN's own genus changes nothing. | Phylogeny | Neighbour-joining tree (K2P, reference-anchored alignment) of one representative per selected BIN x country, with monophyly badges for grade-C species; warns above 500 tips, refuses above 1,000 | | Specimens | Every record, server-side paged and sortable; Find records narrows it to one column's (or any column's) text |

Across the tables: annotations (below), "Find records" on Specimens and every BAGS group (check all it found, then annotate them), "checked" records and a separate representative pick (auto-selected per BIN x country on a fresh search). Downloads: all / selected / annotated records (xlsx, prepared in the background with a progress panel, then saved by themselves; the annotated workbook carries the taxon updates as a second sheet), the selected process IDs (one per line, for a BOLD dataset), the BOLD curation report and the taxon updates (TSV), FASTA, CSV, xlsx species and BIN analyses, and the tree as Newick.

Working through the BAGS screens

Each screen shows one species or BIN at a time; the list above it can be searched by typing. Mark reviewed ✓ records that a curator has finished with it (✓ in the list, "N reviewed" above it, Next unreviewed › skips it); ✎ marks one holding annotated records. Marks are saved with the session, and follow the species or BIN if regrading moves it to another screen.

Grades from regrades the search without touching the snapshot: Original IDs; With curated IDs (each record's updated ID and rank, taxon updates included; contaminated records left out); or Excluding altered records (no record with an ID flag, corrected ID, contamination or taxon update). Counts read "before → after", the checklist and downloads add the view's grade beside the original (bags_grade_curated / bags_grade_unaltered), and group entries note "was X". The BINs tab adds the view's concordance beside the original, with before → after counts. The tree keeps its shape but takes the view's colours and monophyly: a re-identified tip reads "old → new" in purple, a left-out one is grey, and the Newick download carries the same labels (with "->"). Only the Specimens table keeps the original grades. A search over 20,000 records regrades in the background, with a progress panel.

While Find records narrows a table, an amber bar above it says how many of how many records it shows, with Clear filter, and the table is outlined.

Annotations

Check records, then Annotate checked… opens a form that shows each flag's definition, asks only for what the chosen flag needs, and won't apply until the required fields are filled in. A record has at most one ID issue, because each gives one new identification, plus two yes/no issues that combine with it and with each other:

The curator's full name is always required. The rank is suggested from the name; the identification method is one of the iBOL accepted terms.

Taxon updates. Name issue and synonym fix the taxon record in BOLD, so they are saved once, for the name, and reach every record identified as it (listed, downloadable and removable under Taxon updates on the Specimens tab). A record's own corrected ID always wins over a taxon update, whichever came first; if that corrected ID is itself a name with a taxon update, the update is followed too, and Update from shows both steps ("record → taxon (synonym)"). Sessions saved by earlier versions convert on load: Data Issue becomes the Other issue tick, ID Uncertain becomes ID unfounded.

The window is a native pywebview window where that works, else a Chromium --app window, else a browser tab (--window auto|native|browser-app|tab). Everything the app keeps is in ~/.boldcurator/: snapshots (each with a .meta.json provenance sidecar), config.json, sessions.sqlite and boldcurator.log.

Development setup

Python 3.10 or newer. From this python/ directory:

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -e ".[dev,desktop]"
python -m pytest tests/ -q

Or, with uv, exactly the versions pinned in uv.lock (what CI tests, pip-audit checks and the desktop builds ship): uv sync --locked --extra dev --extra desktop, then uv run python -m pytest tests/ -q. After changing dependencies in pyproject.toml, run uv lock and commit uv.lock -- CI fails on a stale lock.

Extras: gui (Shiny), desktop (Shiny, uvicorn, pywebview), bench (psutil, for benchmark's memory column), dev (pytest, psutil).

Command line

boldcurator <command>:

Also installed: boldcurator-desktop, boldcurator-build-snapshot, boldcurator-verify-snapshot and boldcurator-fetch-snapshot. Each tools/*.py script runs the same code from a checkout without installing.

Snapshots

Getting one

The published snapshot is on Zenodo under the concept DOI 10.5281/zenodo.22849515, which always resolves to the latest version (bold_snapshot_<date>.duckdb.gz, about 1.9 GB):

boldcurator fetch-snapshot --record 10.5281/zenodo.22849515 --out bold_snapshot.duckdb

The download runs as 4 parallel byte-range requests where the host allows it, identifies itself as BOLDcurator/<version>, waits out 429/503 as Retry-After asks, and verifies the checksum in the same pass that decompresses it. Re-running only downloads when the snapshot has changed.

Building one

You need the BOLD public data package (BOLD_Public.<date>.tsv.gz, ~3 GB compressed, login-gated at bench.boldsystems.org/index.php/datapackage). The extracted .tsv ingests 3-4x faster than the .gz.

# 1. header check only, in a fraction of a second -- run it first
python tools/build_snapshot.py --tsv BOLD_Public.2026-09-11.tsv --dry-run

# 2. optional trial build; recorded in the file as a partial build
python tools/build_snapshot.py --tsv BOLD_Public.2026-09-11.tsv --out trial.duckdb --limit 100000

# 3. the real build, then verify it from a fresh process
python tools/build_snapshot.py --tsv BOLD_Public.2026-09-11.tsv \
    --out bold_snapshot_2026-09-11.duckdb \
    --temp-dir /fast/local/disk --memory-limit 12GB --threads 4
python tools/verify_snapshot.py --snapshot bold_snapshot_2026-09-11.duckdb

Defaults keep COI-5P only, with sequences, and every source column except identifier_email. BIN, BAGS and the 500 bp quality threshold all assume COI-5P; --marker '' keeps every marker.

A failed build keeps its staging database, and the error says so: retry with --reuse-staging --overwrite to skip the ingest. On PowerShell, quote every path and close every quote -- an unclosed one leaves PowerShell silently waiting at a >> prompt, which looks exactly like a hang.

The 2026-09-11 build, for scale (Windows, 32 GB RAM): 33.6 GB extracted TSV, 20,164,595 COI-5P records, 546,861 taxa, 412,637 BINs; 7.95 GB file (1.9 GB gzipped); about 24 minutes from source, 7 reusing staging. 67% of records have no species-level name, so BAGS grades apply to about a third of the data.

tools/reorder_sequences.py rewrites an older snapshot's sequence table into specimen order (about two minutes, no re-ingest); verify fails a snapshot still in ingest order.

Testing

python -m pytest tests/ -q          # against a generated fixture snapshot
python parity/compare.py            # R-vs-Python parity; exit 1 on any unexplained difference

parity/compare.py runs the committed R reference outputs against the Python pipeline and writes parity/REPORT.md; every difference must map to a recorded divergence (see parity/README.md).

Drive the UI in a real browser before believing a UI change works -- every UI bug so far was invisible to the unit tests:

python tests/make_fake_package.py --out /tmp/fake.tsv.gz --rows 5000
python tools/build_snapshot.py --tsv /tmp/fake.tsv.gz --out /tmp/fake.duckdb
python -m boldcurator.cli gui --snapshot /tmp/fake.duckdb --port 8765 &
pip install playwright && playwright install chromium
python tools/drive_ui.py --out /tmp/shots --url "<the URL gui printed>"

The server only lets in the client that opened its launch URL, which carries a one-time token (localguard.py), so gui prints it ("Open BOLDcurator at ...") and each drive_ui.py run needs a fresh server.

drive_ui.py makes 66 checks across every tab and exits non-zero on a failure. When adding one, match against the table, not the panel -- the annotation form's <select> contains every flag name, so a panel-wide text match passes for an annotation that never rendered.

Benchmarking needs no real snapshot: tools/make_benchmark_snapshot.py builds a 20 M-row synthetic stand-in with the real shape in about six minutes (no sequences). boldcurator benchmark --snapshot FILE times each stage; a size-limit refusal is reported, not raised, and --max-fetch (default 250,000) skips a fetch too wide to hold in memory.

Performance

Measured on the real 7.95 GB snapshot:

Three rules came out of it; keep them:

• Resolve rows first, then fetch them. A BIN-expanded search is a disjunction DuckDB can't push into the scan, so plan_search finds matching rowids through narrow columns and fetch_planned joins that small set back -- which DuckDB can push down. sid doesn't work for this; rowid does.
• sequence is stored in specimen order, so a taxonomic result's sequences sit in a few row groups instead of all of them.
• No parallel scoring. Threads are slower (the regex loop holds the GIL); processes buy 1.4x for real complexity. On a realistic result, scoring is 15 ms.

Performance is closed: reopen it only with a measurement showing a curator waiting. (Round 10 did: large searches and Excel downloads. Anything slow now runs on a background thread with visible progress -- Shiny serves every render, click and download from one event loop, so slow work there freezes the whole app.)

Architecture

src/boldcurator/
  config/      scoring criteria, rank ladder, continents, limits, paths, Zenodo DOI
  data/        snapshot schema, read-only connection, queries (plan/fetch, BIN expansion)
  core/        species rule, scoring, ranking, BAGS, BINs, selection, pipeline,
               paged table, BAGS grouping, summaries, phylogeny, reference alignment
  io/          annotations, exports, session persistence (SQLite)
  build/       snapshot builder, verifier, Zenodo/URL fetcher
  ui/          Shiny app and first-run setup -- the ONLY place a GUI framework is imported
  cli.py       the `boldcurator` command
  desktop.py   window strategies, config, local server
  localguard.py  the local server's host check and one-time launch token
  launcher.py  `boldcurator-desktop` entry point
  shortcuts.py desktop shortcuts for uv/pip installs
tools/         build, verify, fetch, reorder, benchmark snapshot, drive-ui
tests/         pytest suite and the fake data-package generator
parity/        R-vs-Python comparison harness
packaging/     PyInstaller entry point, Windows installer, version stamping

• No GUI imports outside ui/ (tests/test_no_gui_dependency.py enforces it), so the core is testable headless and the CLI works without the gui extra.
• No widget ever receives a whole result. core/table.py pages, sorts and holds selections server-side; a page costs the same for 183 rows or two million. Sorting by a computed column (score, rank, grade) is refused, since it would mean scoring the whole result.
• Size policy. The specimen table works at any size. Species, BIN and BAGS screens are whole-result aggregates, computed lazily once and refused above DOWNLOAD_LIMITS["MAX_RECORDS"] (350,000) with an explanation.
• BAGS screens are problem navigators, not flat tables. Grade C (a species split across BINs) and E (a BIN shared by species) are each a set of separate species-BIN problems, so the screen walks them one at a time, and "Check this group" replaces the selection rather than adding to it. Records identified only to genus ride along by BIN membership, since they may be the misidentification.

Releases

Publish a GitHub release; CI builds the Windows, macOS and Linux downloads and publishes to PyPI, with the version taken from the tag. Never edit the version in pyproject.toml (it stays 0.0.0.dev0). Zenodo archives each release too (/.zenodo.json), and that archive is what the app's "a new version is available" check reads. How the builds work, the update check, and why each packaging flag is there: packaging/README.md.

Every release also carries a CycloneDX SBOM, a SHA256SUMS.txt and signed build-provenance attestations; PyPI files carry PEP 740 attestations from trusted publishing. How to check a download: Verifying your download. For IT reviewers -- network use, files, privileges: SECURITY_OVERVIEW.md. Reporting a vulnerability: SECURITY.md.