Mashpit

Mashpit is a lightweight, local genomic epidemiology platform for rapidly comparing a query genome against representative genomes from NCBI Pathogen Detection SNP clusters.

Mashpit is designed for offline or resource-limited analysis, including deployment on small computers such as a Raspberry Pi. It combines compact genomic sketches, epidemiological metadata, and tree-guided representative selection to support fast preliminary genome screening without requiring a centralized computing service.

Mashpit 1.0 provides adaptive tree-based representative selection, local database management, and reproducible screening outputs. See the changelog for changes since 0.9.10. Use Python 3.8–3.11; Python 3.11 is recommended.

Mashpit and IsoScout

Mashpit builds and manages sketch databases and screens assembled genomes locally. IsoScout uses Mashpit screening as the first stage of a broader workflow with input QC, candidate selection, SNP comparison, and optional cluster-member expansion. Mashpit's similarity scores and near-top flags prioritize candidates; they do not establish outbreak membership or replace SNP analysis.

Existing IsoScout databases-v2 packages can supply exact-release cluster membership to Mashpit reports. Taxon merges preserve the optional REP_CLUSTER annotation table and reject conflicting labels. IsoScout containers pin their own Mashpit revision: installing Mashpit 1.0 on the host does not update a container. Follow IsoScout's runtime and database-manifest requirements when integrating a new or merged database.

See the 1.0 validation record for the tested IsoScout revision, real-genome smoke checks, and their limits. These checks do not establish biological accuracy or validate the full IsoScout SNP workflow.

Key features

• Rapid local queries: Compare an assembled genome against a compact database of representative pathogen genomes.
• Tree-based representative selection: Select representatives directly from NCBI Pathogen Detection SNP trees rather than relying on a single centroid for every cluster.
• Adaptive database size: Use more representatives for large or genetically diverse SNP clusters and fewer representatives for compact clusters.
• Optional radius control: Limit the maximum tree distance between cluster members and their selected representatives.
• Robust assembly retrieval: Exclude records without assembly accessions, retry failed genome downloads, and remove persistently unavailable representatives.
• Integrated metadata: Return epidemiological information such as isolation date, location, host, BioSample accession, assembly accession, and SNP-cluster identifiers.
• Local visualization: Produce a query result table and a tree showing the relationship between the query and its closest database representatives.
• Offline operation: After a database has been built, routine queries can be performed without uploading genomic data.
• Custom databases: Build a database directly from local FASTA files, with no NCBI download at all - useful for proprietary or unpublished sequence collections.
• Extensible metadata: Add your own custom columns to any existing database - taxon, accession, or custom - with a built-in, per-database change history.
• Parallel database loading: Large databases are automatically sharded so mashpit query loads and compares them across CPU cores instead of single-threaded.

Mashpit 1.0 database design

An NCBI Pathogen Detection taxon database contains many SNP clusters. A single representative is not sufficient for every cluster because cluster size and within-cluster diversity vary substantially.

Mashpit 1.0 therefore uses an adaptive tree-based strategy:

1. Download the selected NCBI Pathogen Detection metadata and SNP trees.
2. Remove records that do not have a valid assembly accession.
3. Determine the number of representatives required for each SNP cluster from its tree structure.
4. Select representative leaves that cover the cluster, optionally subject to a user-defined tree radius.
5. Download the selected assemblies.
6. Retry failed downloads for a limited number of attempts.
7. Remove representatives whose assemblies remain unavailable.
8. Build the sketch database and retain metadata linking every representative to its SNP cluster.

This approach reduces redundancy in compact clusters while preserving multiple genomic backgrounds in large or diverse clusters.

Installation

Mashpit currently declares support for Python >=3.8,<3.12; use Python 3.11 for the Raspberry Pi instructions below. Python 3.12 and newer are not currently supported.

Option 1: Conda or Mamba

Bioconda packaging can lag behind GitHub and PyPI. Use the pinned pip command below for 1.0.0; the following installs the version currently available on Bioconda:

conda create -n mashpit -c conda-forge -c bioconda mashpit
conda activate mashpit

Option 2: pip

Mashpit requires the NCBI Datasets command-line tool for downloading assemblies when building taxon or accession databases. It is not required for querying an existing database or building a custom database from local FASTA files.

For Intel/AMD 64-bit Linux (x86_64), install it with the following commands. Raspberry Pi users should follow the ARM64 instructions below.

mkdir -p "$HOME/.local/bin"
curl -fL -o "$HOME/.local/bin/datasets" \
  'https://ftp.ncbi.nlm.nih.gov/pub/datasets/command-line/v2/linux-amd64/datasets'
chmod +x "$HOME/.local/bin/datasets"
export PATH="$HOME/.local/bin:$PATH"

Install the stable release:

python -m pip install 'mashpit==1.0.0'

Install the development version from source:

git clone https://github.com/tongzhouxu/mashpit.git
cd mashpit
python -m pip install .

Verify the installation:

mashpit --help

Raspberry Pi: Git clone and a Python virtual environment

Use a 64-bit operating system with Python 3.11. These instructions install the current Git source without Conda and assume a Debian-based system such as Raspberry Pi OS, with python3 pointing to Python 3.11.

Check the architecture, user-space bitness, and Python version first:

uname -m
getconf LONG_BIT
python3 --version

The expected results are aarch64, 64, and Python 3.11.x. If the system uses 32-bit user space, use a 64-bit OS for this installation route. If python3 is 3.12 or newer, install Python 3.11 with its matching venv and development packages separately, then use python3.11 -m venv .venv below. Creating a virtual environment does not change the interpreter version; do not replace the OS's system Python.

Install the system packages needed to create the environment and compile dependencies:

sudo apt update
sudo apt install -y git curl ca-certificates build-essential pkg-config \
  python3-venv python3-dev libhdf5-dev

Clone the repository, create and activate the virtual environment, and install Mashpit:

git clone https://github.com/tongzhouxu/mashpit.git
cd mashpit

python3 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install .

python -m pip check
mashpit --help

Some dependencies compile from source, so installation can take time on a Pi. Keep the virtual environment activated while installing and running Mashpit; do not use sudo pip.

If you will build taxon or accession databases that download NCBI assemblies, install the Linux ARM64 Datasets executable into the activated virtual environment:

curl -fL -o "$VIRTUAL_ENV/bin/datasets" \
  'https://ftp.ncbi.nlm.nih.gov/pub/datasets/command-line/v2/linux-arm64/datasets'
chmod +x "$VIRTUAL_ENV/bin/datasets"
datasets --version

The linux-amd64 executable in the general Linux instructions is for Intel/AMD CPUs and will not run natively on a Pi. See the NCBI Datasets installation documentation for platform downloads. Skip this step if you only query existing databases or build custom databases from local FASTA files.

In a new terminal, change to your cloned repository and reactivate the environment:

cd /path/to/mashpit
source .venv/bin/activate

To update an existing installation from Git, run these commands from the repository with the environment activated:

git pull --ff-only
python -m pip install .
python -m pip check

The repository does not currently run ARM-specific CI tests; these instructions are not a claim of validation on Raspberry Pi hardware.

Database contents

A Mashpit database is stored in a directory containing the sketch database and its metadata. Depending on the database type, it includes files such as:

DATABASE_NAME/
├── DATABASE_NAME.db               # sqlite metadata and representative tables
├── DATABASE_NAME.sig              # merged sourmash signatures (a single file for smaller
│                                   #   databases, or a directory of shards for larger ones -
│                                   #   see "Sharding a database for faster loading")
├── representatives.tsv            # taxon databases only
├── cluster_summary.tsv            # taxon databases only
├── membership/                    # complete source-release TSVs for taxon databases
│   ├── PDG_VERSION.metadata.tsv.gz
│   └── PDG_VERSION.reference_target.all_isolates.tsv.gz
├── unavailable_assemblies.tsv     # taxon/accession databases: accessions that could not be downloaded
└── unavailable_sequences.tsv      # custom databases: local FASTA files that failed to sketch

The build log itself (mashpit-<timestamp>.log) is written to the current working directory alongside the database folder, not inside it.

Keep the entire database directory together when moving it to another computer.

Taxon builds retain the full metadata and cluster-isolate membership from the same resolved PDG release, including isolates without a downloadable assembly. The SQLite CLUSTER_MEMBERS table contains the fields used by cluster reports; the compressed TSVs preserve all original columns for later use and candidate expansion. Clinical/environmental statistics count all cluster members, with unknown sources reported separately, rather than only the sketched representatives. Reports can also read these versioned TSVs from older packages, including IsoScout databases-v2, when CLUSTER_MEMBERS is absent. Older databases with neither source show unavailable cluster statistics. Accession and custom builds have no NCBI cluster membership snapshot.

Building a database

Mashpit supports three database types:

• Taxon database: Build from NCBI Pathogen Detection SNP clusters for a selected organism.
• Accession database: Build from a user-provided list of NCBI BioSample accessions.
• Custom database: Build directly from local FASTA files, with no NCBI download at all.

Taxon database

mashpit build taxon salmonella --species Salmonella

A specific NCBI Pathogen Detection release can be selected with --pd_version:

mashpit build taxon listeria_innocua \
  --species Listeria_innocua \
  --pd_version PDG000000091.9

Supported taxon directory names are listed in the NCBI Pathogen Detection results directory.

Accession database

Prepare a text file containing one BioSample accession per line:

SAMN00000001
SAMN00000002
SAMN00000003

Then run (an Entrez email is required to resolve BioSample accessions via NCBI):

mashpit build accession custom_database --list biosamples.txt --email you@example.com

Custom database

Build a database directly from local FASTA files - no NCBI download, Entrez lookup, or datasets CLI is required for this database type, which makes it suitable for proprietary or unpublished sequence collections. Place one genome per file, named <sample_id>.fasta (.fa/.fna/.fas/.ffn, optionally gzipped), in a directory:

my_genomes/
├── sample1.fasta
├── sample2.fasta
└── sample3.fasta

Then run:

mashpit build custom my_database --input-dir my_genomes/

Optional metadata can be supplied as a TSV with a sample_id column (matching the filenames) plus any subset of the usual metadata fields (strain, collection_date, geo_loc_name, host, etc.) and/or your own custom columns; fields not supplied, or samples not listed, are recorded as missing:

sample_id   strain    collection_date   host
sample1     StrainA   2024-03-01        Homo sapiens

mashpit build custom my_database --input-dir my_genomes/ --metadata metadata.tsv

A custom database has no SNP-cluster concept, so it behaves like an accession database at query time: no cluster-candidates table, and --radius/tree-based representative selection do not apply (every provided genome becomes a database entry directly).

Tree-radius option

Mashpit 1.0 can optionally use a tree-radius threshold during representative selection, via --radius (default 20, in the same units as the SNP tree's branch lengths). A smaller radius generally selects more representatives and gives denser coverage of within-cluster diversity. A larger radius selects fewer representatives and produces a smaller database:

mashpit build taxon salmonella --species Salmonella --radius 10

Use the radius option only after evaluating an appropriate value for the organism and intended surveillance resolution.

Assembly download handling

Before representative selection and download, Mashpit removes rows without valid assembly accessions. Selected assemblies that fail to download are retried automatically. After the configured retry limit, unavailable assemblies are removed from the representative set and recorded in the build log.

This prevents missing assemblies from producing invalid database entries.

Annotating a database

mashpit annotate adds or updates custom metadata columns on a database you've already built - taxon, accession, or custom - since all three share the same underlying metadata table. This is the place to attach your own project-specific fields (an internal lab ID, a project or batch code, an outbreak investigation code, anything not in Mashpit's built-in schema) without rebuilding the database.

To see which assembly/BioSample accessions are actually in the database - the ids your values file needs to match - print them as a starting point:

mashpit annotate my_database --list-accessions > accessions.tsv

Then annotate:

mashpit annotate my_database --values project_metadata.tsv

project_metadata.tsv needs an asm_acc or biosample_acc column to match existing rows, plus whatever columns you want to add or update:

asm_acc            project_batch   internal_lab_id
GCA_000000001.1    Batch01         LAB-001

Columns not already present in the database are added automatically. Rows in the database not mentioned in your file keep their existing values (or are recorded as missing if the column is new); ids in your file with no matching database row are reported, not treated as an error.

Every mashpit annotate run is recorded in the database itself, so you always have a record of what changed and when:

mashpit annotate my_database --history

This works on databases built with older Mashpit versions too - the history table is created automatically the first time you annotate a database that doesn't have one yet, so nothing needs to be rebuilt.

Sharding a database for faster loading

Databases are automatically split into multiple signature shards (a DATABASE_NAME.sig/ directory of shard files instead of a single DATABASE_NAME.sig file) once they reach around 200 representatives, so mashpit query can load and compare shards across CPU cores in parallel rather than parsing one large file single-threaded. The shard count scales with database size (roughly one shard per 100 signatures, capped at 128 - so databases under ~200 representatives stay a single file, since one shard is indistinguishable from none) and query-time parallelism scales with whatever machine actually runs the query (all available cores by default, or --threads to cap it) - a database built once and distributed to others gets the same proportional speedup whether it's queried on a 4-core laptop or a many-core server.

Databases built with an older Mashpit version, or small databases that stayed as a single file, can be resharded without rebuilding - a one-time, local, offline operation that only rewrites the .sig storage, leaving all metadata untouched:

mashpit reshard my_database

Pass --shards to override the automatic count, or --shards 1 to merge shards back into a single file:

mashpit reshard my_database --shards 64
mashpit reshard my_database --shards 1

Merging compatible databases

Merge two or more local database snapshots into a new directory:

mashpit merge combined database_a database_b

Inputs can be database directories or .db files. Inputs remain unchanged and an existing output is refused. The command works offline and loads the input sketches and metadata into memory; allow enough RAM for their combined size. The output supports the usual query, annotate, and reshard commands.

Supported combinations and requirements:

• Custom + custom: matching sketch settings. Identical IDs with identical sketches are deduplicated. Different IDs remain separate even if their sketches match. An ID with different sketches is rejected.
• Taxon + taxon: additionally require the same species, PDG release, representative method, and tree radius. Full cluster membership must be available for every represented cluster. Representatives are unioned and membership is deduplicated by target accession. Cluster summaries are recalculated from membership, never by adding input cluster-size totals.
• Mixed types, accession databases, different PDG releases, and older taxon databases lacking the required settings or membership are rejected. Merging is not a release-update or representative-reselection operation.

Actual signature settings must agree across all inputs: k-mer size, molecule type, seed, hash count, scaled setting, and abundance tracking. Hash counts and k-mer sizes must also agree with database descriptions. No automatic downsampling or re-sketching takes place.

For independent custom datasets with overlapping sample IDs, explicitly prefix both assembly and BioSample IDs, with one prefix per input:

mashpit merge combined lab_a lab_b --prefixes labA_ labB_

Metadata columns are unioned and missing values can be filled from another source. Conflicting nonempty values stop the merge; no input takes priority. Assembly IDs and BioSample IDs must each remain unique. Prefixes are available only for custom inputs. Without prefixes, resolve conflicting metadata in a separate copy of an input before retrying.

Taxon merges also union optional IsoScout REP_CLUSTER annotations by the exact, versioned PDS_acc, retaining all annotation columns. Conflicting nonempty labels or evidence fields reject the merge; a source without annotations does not erase another source's labels. Unknown SQLite tables are rejected to prevent silent loss of extensions.

To save the first conflicting record and field as JSON:

mashpit merge combined database_a database_b --conflict-report conflicts.json

The report path must be new. Compatibility errors are reported directly. No output database is published on failure.

merge_provenance.json records source paths, original settings, ID mappings, and deduplication counts. Source build tables, unavailable-record tables, and prior merge manifests are retained under merge_sources/, and annotation history is preserved with source paths. Original database.json and rep_cluster_annotations.json sidecars, when present, are also archived there. They describe the inputs, not the merged database. Generate a new IsoScout package manifest and checksums before using the merged output as an IsoScout package; source manifests must not be reused for it. Available versioned release TSVs are combined without dropping their columns; conflicting records are rejected. Taxon output includes rebuilt representatives.tsv and cluster_summary.tsv. Source-specific download attempts and original FASTA paths remain in provenance rather than being presented as properties of the merged representatives.

Querying a database

Run a query using an assembled genome in FASTA format:

mashpit query sample.fasta path/to/database

Common query options include:

--number                 maximum number of database matches returned
--threshold              minimum similarity used when constructing the local result tree
--annotation             metadata field used to annotate tree tips
--no-tree                skip all optional tree work; keep identical search tables
--tie-tolerance-hashes   sketch-hash tolerance used to flag near-top SNP clusters (taxon databases only)
--threads                worker processes for a sharded database (default: all available cores)

Example:

mashpit query sample.fasta path/to/salmonella \
  --number 200 \
  --threshold 0.85 \
  --annotation isolation_source

Query output

A query produces:

1. Representative-matches table

   A CSV file (<query>_representative_matches.csv) containing the closest representatives and associated metadata. Results include fields such as BioSample accession, assembly accession, SNP-cluster accession, similarity score, and NCBI links.

2. Cluster-candidates table (taxon databases only)

   A CSV file (<query>_cluster_candidates.csv) that groups the representative matches by SNP cluster, ranked by best similarity score. A near_top column flags clusters whose best hit is within --tie-tolerance-hashes sketch hashes of the single best hit overall, since at typical sketch sizes a small similarity gap can be within normal MinHash sampling noise rather than a meaningful difference.

3. Local tree

   Newick and image files (<query>_tree.newick, <query>_tree.png, <query>_tree.svg) showing the query genome together with its closest database representatives above the similarity threshold. The tree is skipped (with a log message) when the top hit falls below --threshold or fewer than two candidates qualify.

4. Tree status

   <query>_tree_status.json is written after both required search tables (where applicable) succeed, even when the tree is disabled or unavailable. Its schema_version is 1; search_complete is true. The status field is one of:

   Status	Meaning
   generated	Newick, PNG, and SVG were successfully written.
   disabled	The user supplied --no-tree.
   skipped_insufficient_hits	Fewer than two representatives qualify, including when the top hit is below the threshold.
   construction_failed	Tree-specific signature retrieval, pairwise distances, construction, annotation, or Newick writing failed.
   rendering_failed	Newick was saved, but rendering or image writing failed.

   reason explains a skip/failure, and error_type identifies the exception for failures. newick, png, and svg contain filenames for successfully written outputs, otherwise null. A rendering failure retains valid Newick and removes partial PNG/SVG files. Previous tree artifacts with the same query prefix are removed before the optional workflow. Files that cannot be removed are listed in cleanup_errors; use the status fields rather than file existence to decide which artifacts are valid.

5. Standalone HTML report

   <query>_report.html opens directly in a browser, with no Streamlit or web server required. It includes ranked SNP clusters, full cluster sizes, clinical/environmental/unknown counts and percentages, sampling dates, computed serotypes, and representative evidence. The tree image (when available) and downloadable CSV/Newick files are embedded, so the report can be shared as one file and viewed offline. --no-tree still produces the report, with the tree marked as disabled. Accession/custom databases show returned-isolate metadata without SNP cluster statistics.

   The output path is printed in the query log. If report generation fails, Mashpit logs a warning and retains the successful CSV results.

6. Log file

   A timestamped record of the query parameters and processing steps.

A zero process exit code means the required search outputs succeeded; it does not promise a tree. Check the exit code first, then read the tree status to decide whether to display images, offer Newick alone, or show the reason. Database loading, sketching, similarity calculation, and required CSV-writing failures remain nonzero exits, even if an earlier CSV already exists. Use a fresh output directory for each query to avoid confusing earlier outputs with a failed run. If filesystem errors prevent writing the status JSON itself, Mashpit warns in the log and keeps the successful search outputs; callers must treat missing status as unavailable tree information.

For search-only use:

mashpit query sample.fasta path/to/database --no-tree

This skips tree-specific signature retrieval, candidate-to-candidate distances, neighbor joining, annotation, and rendering. Query-to-database comparison still runs normally. Similarity scores, ranking, tie flags, returned candidate counts, and cluster tables are identical with or without the option. The default still attempts to generate a tree.

Tree rendering and annotation use iterative traversal and serialization, avoiding TreeViz's recursive copying and BioPython's recursive traversal/writer helpers. No recursion-limit increase is used. Large trees can still exhaust memory or fail in a dependency; those optional failures are recorded without invalidating search results. Zero-distance trees use unit edges only in the display, with a caption; their Newick distances are preserved.

Mashpit results are intended for rapid screening and prioritization. They do not replace validated SNP-pipeline, cgMLST, or whole-genome phylogenetic analyses when formal outbreak confirmation is required.

Graphical interface

Start the local Mashpit interface with:

mashpit gui

This launches a Streamlit app and opens it in your browser, by default at:

http://localhost:8501

Use --port to run on a different port:

mashpit gui --port 8888

The interface includes a Skip tree generation setting and a downloadable tree-status JSON. Tree and preview errors leave the search tables accessible, including a Newick download when rendering failed.

The interface lets users select a local database, upload a query assembly, run the search, and inspect the result table and tree - all without the genome or results leaving the local machine.

A pre-built Mashpit database is required.

The report has three views:

• Cluster context: ranked full cluster sizes with environmental/clinical/unknown composition; select a cluster to explore computed serotypes, reported serovars, sampling years, geography, isolation sources, hosts, and all member isolates.
• Sketch-distance tree: height adapts to tip count, font size, and wrapped labels. Font and spacing controls redraw the figure without rerunning the query. The scrollable SVG view keeps labels readable; download PNG, SVG, or Newick.
• Isolate evidence & downloads: representative-level similarity scores, complete metadata, and the enriched cluster-summary CSV.

New taxon builds retain CLUSTER_MEMBERS from the complete release membership, including members without assemblies or downloaded genomes. Cluster size counts unique Pathogen Detection target accessions, not representatives. epi_type supplies environmental/clinical categories; unknown values remain explicit. Date summaries use valid ISO years, months, or dates and report their coverage. Raw dates, including ranges, remain in the isolate download. Computed serotype predictions are kept separate from submitted serovar values, and the original computed_types field is retained.

Existing databases: older taxon databases with bundled versioned membership and metadata TSVs (including IsoScout databases-v2) support full cluster statistics without rebuilding. Databases containing only representative metadata show those statistics as unavailable until rebuilt. The report never substitutes representative counts for full cluster size. Accession/custom databases show metadata for returned isolates without implying SNP cluster membership.

The near-top flag uses the configured sketch-hash tolerance; it is a screening heuristic, not a statistical confidence interval. Cluster-member metadata are context: similarity was measured only for the returned representatives.

Intended use

Mashpit is intended for fast, lightweight genomic screening, including:

• preliminary placement of newly sequenced isolates;
• identification of the most relevant NCBI Pathogen Detection SNP clusters;
• local analysis of sensitive genomic data;
• decentralized surveillance in laboratories with limited computing infrastructure;
• prioritization of isolates for higher-resolution analysis.

Mashpit should not be used as the sole evidence for declaring or excluding an outbreak relationship.

Citation

Xu T. et al. (2024). Mashpit: sketching out genomic epidemiology.
Journal of Open Source Software, 9(104), 7306.
https://doi.org/10.21105/joss.07306

Contributing

See CONTRIBUTING.md for development and contribution guidelines.

License

Mashpit is distributed under the GNU General Public License v2.0. See LICENSE.