specimux-suite

Early preview release. This project is under active development — APIs, event formats, and CLI options may change between versions. Feedback and bug reports are welcome at github.com/joshuaowalker/specimux-suite/issues.

Orchestration and real-time dashboard for the Mycomap fungal DNA barcoding pipeline. Manages the full workflow from raw nanopore reads through demultiplexing, consensus generation, variant summarization, and species identification.

Installation

Requires Python 3.11+.

pip install specimux-suite

Or, for development, from a source checkout:

pip install -e '.[dev]'

The pipeline invokes external bioinformatics tools as subprocesses — specimux, speconsense, and vsearch must be installed and available on your PATH.

See each tool's repository for installation instructions. These tools have system-level dependencies that pip cannot install:

Quick start

Batch mode

Process a single FASTQ file end-to-end:

specimux-suite batch primers.fasta specimens.tsv reads.fastq \
    --reference-db references.fasta

Live mode

Watch a directory for new FASTQ files (e.g., from a running MinION sequencer) and process them incrementally as they appear:

specimux-suite live primers.fasta specimens.tsv /path/to/minknow/output/ \
    --reference-db references.fasta

A web dashboard opens automatically at http://127.0.0.1:8077 showing real-time progress. Press Ctrl+C to finalize — the pipeline will drain remaining files, process all eligible specimens regardless of threshold, and run summarization before exiting.

Profiles

Profiles bundle pipeline settings and tool configurations into reusable presets:

# List available profiles
specimux-suite batch --list-profiles

# Use a profile
specimux-suite batch -p herbarium primers.fasta specimens.tsv reads.fastq

Bundled profiles include default (standard settings) and herbarium (relaxed thresholds for degraded DNA). Custom profiles can be placed in ~/.config/specimux-suite/profiles/.

Input files

Primers — FASTA file containing primer sequences used for demultiplexing.

Specimens — Tab-separated file with at least SampleID and PrimerPool columns. Specimen IDs containing an iNaturalist observation ID (e.g., iNat12345) enable automatic community taxon lookup for on-target/off-target detection.

SampleID	PrimerPool
spec001	pool1
spec002	pool1
specimen-B--iNat12345	pool2

Reads — Standard FASTQ format (batch mode expects a single file; live mode watches a directory for *.fastq files).

Reference database — Optional FASTA file of reference sequences for species identification via vsearch. The sequence ID (first whitespace-delimited token) is used as the match key. An optional name="..." field in the header provides a display name; without it, the name is derived from the ID by replacing underscores with spaces (e.g., Genus_species_authority becomes "Genus species").

>MycoMap_12345_Trametes_versicolor_US_Indiana name="Trametes versicolor"
ACGTACGT...
>MycoMap_67890_Stereum_ostrea_US_Ohio
TGCATGCA...

Options

Common options

Web dashboard options

Live mode options

Pipeline

Processing stages

FASTQ reads
  → specimux (demultiplex into per-specimen FASTQs)
    → speconsense (generate consensus sequences per specimen)
      → vsearch + adjusted-identity (identify species from reference DB)
        → speconsense-summarize (extract and identify variant sequences)

Scheduling

The scheduler uses two-tier prioritization:

1. Never-processed specimens — prioritized by read count (highest first), processed once they reach --min-reads
2. Reprocessing candidates — specimens with enough new reads since last consensus (controlled by --reprocess-ratio)

In live mode, watched specimens (starred in the dashboard) receive a priority boost and are processed first.

Within the reprocessing tier, candidates are ordered by result confidence — uncertain results are revisited first, since additional depth might change the answer:

1. No match — consensus produced but nothing hit the reference database
2. Low identity — best hit below 90% adjusted identity
3. Off-target — no hit matches the iNaturalist community genus (a mycoparasite or yeast contaminant may be dominating the true target), or the community genus appears only in a minority cluster
4. Marginal — identity 90–98%, or ambiguous bases in the consensus
5. Confident — ≥98% identity and on-target

Uncertain results (the first three) also re-enter the queue at half the configured --reprocess-ratio, so depth reaches them sooner. Confidence only reorders work — confident specimens still reprocess whenever workers are free, and finalization always processes every specimen with unprocessed reads, regardless of confidence. The dashboard shows a small ↻ chip on each queued reprocess candidate with the reason it was prioritized.

Live mode concurrency

Consensus jobs read copy-on-write snapshots of their input FASTQs (instant on APFS/btrfs/XFS, a plain copy elsewhere), so when a new FASTQ file stabilizes, specimux demultiplexes it immediately — appending to the live per-specimen files while in-flight consensus jobs keep running on their snapshots. Demultiplexing uses whatever worker threads aren't occupied by consensus jobs, and newly-ready specimens are scheduled as soon as it finishes.

Web dashboard

The built-in dashboard provides a real-time view of pipeline progress, streamed via server-sent events (SSE).

Processing tab

• Specimen table with status, read count, top identification match, and identity score
• Color-coded status badges (queued, processing, identified, no match, error)
• On-target/off-target indicators when community taxa are available
• Identity warnings for low-confidence matches (<98% or <90%)
• Expandable cluster-level detail with per-cluster identification and sequence viewer
• Cluster quality badges: NS/LQ routing preview and CHIMERA (speconsense 0.8.6+ two-parent recombinant flag; routed to the .chimera track when summarize runs with --filter-chimeras, otherwise kept in Summary and badged for review)
• Search, sort, and filter (novel, on-target, off-target, no-match, watched)

Summary tab

• Variant-level results after summarization
• Variant count per specimen with expandable detail rows
• Per-variant identification, read count, and sequence length
• Identification results shown only after variant-level identification completes

Forecast tab

A live stop estimator for the sequencing operator (live mode, after at least two files have been demultiplexed). From the per-file demultiplex history it estimates each specimen's read-accumulation rate on the cumulative-matched-reads clock and projects when below-threshold specimens will cross — with 90% intervals — plus:

• A headline: how many specimens are over threshold, how many more are projected to cross within the next hour, and how many will likely never make it
• A viewer-selectable forecast threshold (chips for common values plus a custom input) — view-only, the scheduler keeps using min_reads
• Threshold sensitivity (≥10 / min_reads / ≥100 / selected), since downstream verification often succeeds well below min_reads
• The specimens-over-threshold accumulation curve for the run so far
• A rate-drift self-check that flags when the forecast's stationarity assumption looks shaky for the current run

Watch feature

Click the star on any specimen row to boost its scheduling priority. Watched specimens are processed ahead of all others in live mode.

Sharing

Use --share to bind the dashboard to your LAN address and display a QR code for easy access from other devices.

Replay

For testing or demos, specimux-replay splits a source FASTQ into timed chunks that mimic MinKNOW output:

specimux-replay source.fastq simulated_output/ --reads-per-file 4000 --delay 30

Files are written atomically with MinKNOW-style filenames. Pair with live mode to replay a sequencing run:

# Terminal 1: start the pipeline
specimux-suite live primers.fasta specimens.tsv simulated_output/ --reference-db refs.fasta

# Terminal 2: replay sequencer output
specimux-replay source.fastq simulated_output/

Output

The output directory contains:

output_dir/
├── events.jsonl                    # Append-only event log (rotates at 100 MB)
├── inat_taxon_cache.json           # Cached iNaturalist community taxa
├── specimux/full/{pool}/
│   └── {specimen_id}.fastq         # Demultiplexed reads per specimen
├── consensus/{specimen_id}/
│   └── {specimen_id}-all.fasta     # Consensus sequences (one or more clusters)
├── summary/
│   ├── {variant_id}-RiC*.fasta     # Individual variant sequences
│   └── summary.fasta               # Aggregated summary sequences
└── identification/
    └── {specimen_id}.tsv           # vsearch hits with adjusted-identity scores

The event log (events.jsonl) records every state change and is the single source of truth for pipeline state. Pipeline state is an in-memory materialized view rebuilt by replaying all events — it is never persisted to disk.

Development

# Install with dev dependencies
pip install -e '.[dev]'

# Run tests
pytest tests/

# Run a single test
pytest tests/test_state.py::test_read_totals -v

License

BSD 3-Clause. See LICENSE.