spt-pipeline 0.1.1

Unified Single-Particle Tracking (SPT) pipeline: tracking, brightness/mobility analysis, two-color co-diffusion, and acquisition-metadata utilities.

spt-pipeline

A consolidated Single-Particle Tracking (SPT) analysis suite that combines tracking, brightness/mobility analysis, two-color co-diffusion, and acquisition-metadata utilities behind a single, unified command-line interface.

Table of Contents

• Overview
• Installation
• Quick Start
• Console Scripts
• Command Structure
• Data Preparation (prep)
• Tracking Operations (track)
• Analysis Operations (analyze)
• Plotting Operations (plot)
• Drawing Operations (draw)
• Utility Operations (utils)
• End-to-End Workflows
• Two-Color Co-Diffusion (spt-coloc)
• Movie Metadata Summarizer (spt-metadata)
• Parameter Reference
• Tips and Best Practices
• Troubleshooting
• Migration Notes
• Publishing
• License and Citation

Overview

spt-pipeline provides a comprehensive suite of tools for single-particle tracking analysis, from raw image processing to advanced statistical analysis and visualization. The package installs three console scripts:

• spt-pipeline - main analysis pipeline (prep / track / analyze / plot / draw / workflow / utils)
• spt-coloc - two-color bead-based registration and co-diffusion analysis
• spt-metadata - lightweight ND2/TIFF acquisition-metadata summarizer

spt-pipeline itself is organized into the following command groups:

• prep - data preparation and format conversion (TrackMate, ND2, OME-TIFF)
• track - particle detection and tracking (trackpy, dwell-time)
• analyze - statistical analysis of trajectories (brightness-mobility, condition comparison)
• plot - visualization (heatmaps, dwell, brightness maps, track overlays)
• draw - movie/track overlay rendering
• workflow - end-to-end automatic workflows
• utils - utility operations (split metrics, find files)

Installation

From PyPI (recommended)

pip install spt-pipeline                # core deps only
pip install "spt-pipeline[all]"         # all optional features

Optional extras:

Extra	Adds	Used for
tracking	trackpy, numba	track trackpy, track dwelltime
imaging	nd2, pims, tifffile	ND2/TIFF I/O, channel stacking
stats	statsmodels	Partial-regression in analyze brightness-mobility
all	everything above	Full functionality
dev	build, twine, pytest	Building/uploading the package

From source

git clone https://github.com/andrewbazley/spt-pipeline.git
cd spt-pipeline
pip install -e ".[all]"

Requirements

• Python 3.10+
• Required (installed automatically): numpy, pandas, matplotlib, scipy, scikit-learn, scikit-image, seaborn

Quick Start

# Get help
spt-pipeline --help
spt-pipeline prep --help
spt-pipeline track --help
spt-pipeline analyze --help
spt-pipeline plot --help
spt-pipeline draw --help
spt-pipeline workflow --help
spt-pipeline utils --help

# Typical end-to-end run
spt-pipeline track trackpy /data/my_experiment --diam 7 --minmass 110 --fps 33.33 --mpp 0.15
spt-pipeline analyze brightness-mobility /data/my_experiment --dt 0.03
spt-pipeline analyze compare-conditions /data/my_experiment
spt-pipeline plot steps-brightness /data/my_experiment --make both

Console Scripts

Command	Module	Purpose
spt-pipeline	spt_pipeline.pipeline:main	Main analysis pipeline
spt-coloc	spt_pipeline.colocalization:main	Two-color registration and co-diffusion analysis
spt-metadata	spt_pipeline.metadata:main	ND2/TIFF acquisition-metadata summarizer

Command Structure

All spt-pipeline invocations follow:

spt-pipeline <command> <subcommand> [arguments] [options]

Example:

spt-pipeline track trackpy /path/to/images --diam 7 --minmass 110

---

Data Preparation (prep)

prep trackmate - TrackMate CSV preprocessing

Converts TrackMate CSVs to a standardized trajectory format with canonical columns (x, y, frame, track_id, condition, replicate).

spt-pipeline prep trackmate <root_dir> [--out-dir DIR] [--include-date] [--date-format FORMAT]

Parameters

• root_dir (required): root directory to recursively scan for TrackMate CSVs.
• --out-dir: output directory (default: <root_dir>/raw).
• --include-date: include date code in output filename.
• --date-format: YYMMDD or YYYYMMDD (default: YYMMDD).

What it does

• Recursively searches for TrackMate CSVs (Spots in tracks statistics.csv, All spots statistics.csv, or any CSV with x, y, frame, track_id columns).
• Cleans/standardizes column names.
• Infers condition and replicate from directory structure or filename.
• Extracts date codes from paths/filenames.
• Outputs Traj_{condition}_{replicate}.csv (or Traj_{condition}-{datecode}_{replicate}.csv with --include-date).

spt-pipeline prep trackmate /data/trackmate_exports --out-dir /data/raw --include-date

prep nd2-to-tiff - ND2 to TIFF conversion

spt-pipeline prep nd2-to-tiff <directory> [--channel N]

• Extracts the requested channel from each ND2 (default channel 0).
• Handles multi-channel images.
• Renames brightfield (BF) images with a BF_ prefix.

prep stack-channels - Stack two-channel images

spt-pipeline prep stack-channels <spool_dir> <output_dir>

Combines TIFF slices from lefts/ and rights/ subdirectories into a two-channel OME-TIFF ({spool_name}_2ch.ome.tif, axes ZCYX, channels left/right, zlib compression).

prep find-ome-tif - Find OME-TIFF files

spt-pipeline prep find-ome-tif <input_folder> [--output-dir DIR]

Recursively searches for files containing _MMStack_Pos0.ome.tif and writes a CSV listing their paths.

---

Tracking Operations (track)

track dwelltime - Dwell-time analysis from ND2

spt-pipeline track dwelltime <input_folder> [--diam INT] [--minmass FLOAT] [--fps FLOAT] [--channel N]

Detects particles, links into tracks, filters short tracks, and computes dwell times (frames each particle is visible). For each movie, writes a {movie_name}/ subdirectory with *_filtered_tracks.csv and diagnostic plots.

track trackpy - Trackpy-based tracking

spt-pipeline track trackpy <input_directory> [options]

Parameters

• input_directory (required).
• --notrack: skip tracking, only generate MSD plots from existing data.
• --mpp (microns/pixel; default 0.15).
• --fps (frames/second; default 33.33).
• --diam (pixels; default 7).
• --range (linking search radius in pixels; default 10).
• --memory (frames a particle can vanish; default 2).
• --minmass (default 110).
• --percentile (default 40).
• --diffusion_tau (lag-time index for D fit; default 1).
• --min-separation / --min_separation.
• --min-track-length (default 10).
• --channel (default 0).
• -o / --out-dir / --output-dir (separate output root; defaults to input directory).
• --correct-brightness / --no-correct-brightness (flatfield correction; default on).

Per-track output columns (*_diffusion_coefficients.csv):

Column	Description
particle	Track ID
n_spots	Number of detections in track
dwell_time	Track duration (frames)
alpha	Anomalous diffusion exponent
total_mass	Cumulative integrated intensity
mean_x_px, mean_y_px	Mean track centroid
mass_per_frame	Raw mean per-frame brightness
mass_per_frame_ff	Flatfield-corrected brightness
flatfield_factor	Local illumination correction factor
mass_per_frame_zscore	Within-movie z-score of corrected brightness
D_fit_um2_s	Diffusion coefficient at fixed lag (µm²/s)
D_last_um2_s	Diffusion coefficient at last finite lag (µm²/s)

Brightness correction:

1. Flatfield correction: a 2D quadratic polynomial is fit to per-detection (x, y, mass) to estimate the illumination profile. Each track's mass_per_frame is divided by the predicted local illumination (normalized to median 1.0) → mass_per_frame_ff.
2. Z-score normalization: mass_per_frame_ff is z-scored within each movie → mass_per_frame_zscore.

spt-pipeline track trackpy /data/images --diam 7 --minmass 110 --fps 33.33 --mpp 0.15 --range 10 --memory 2

---

Analysis Operations (analyze)

analyze brightness-mobility - Brightness vs mobility

spt-pipeline analyze brightness-mobility <input> [options]

Parameters

• input (required): CSV file or directory with trajectory CSVs.
• -o, --outdir: output directory.
• --dt: time per frame in seconds (default 1.0).
• --use-maxlag / --no-use-maxlag (default on).
• --immobile-threshold-px (default 0.01).
• --display-step-cap-px (default 3.0).

Auto-detects column names (TrackMate, GEMspa, custom). Computes per-track step sizes, immobile fraction, mean brightness, sigma, and D_uncorr / D_corr. Generates scatter, histogram, violin, CDF, and (when statsmodels is installed) partial-regression summaries. Writes per_track_metrics.csv and summary_metrics.json per input file.

analyze compare-conditions - Cross-condition comparison

Compares D, step sizes, and brightness across experimental conditions. Reads per-movie CSVs from a track trackpy output tree (using groupnames.csv) and/or flat pre-pooled *_diffusion_coefficients.csv files placed directly in input_dir.

spt-pipeline analyze compare-conditions <input_dir> [options]

Parameters

• input_dir (required).
• --conditions: condition tokens to match in movie/file names (fallback if no groupnames.csv).
• --max-gmm-k: max GMM components to test (default 5).
• --step-subsample: subsample size for step-size GMM (default 100000).
• --d-xlim: linear D histogram x-limit (default 2.0).
• --step-xlim: step histogram x-limit (default 0.5).
• --min-nspots: minimum track length (n_spots) to include (e.g. 20).
• --min-alpha: minimum anomalous exponent to include (e.g. 0.3).
• --subtract-loc-error: subtract per-condition localization-error floor from D values.
• --loc-error-alpha-max: max α for the immobile proxy (default 0.2).
• --loc-error-min-nspots: min n_spots for the immobile proxy (default 6).
• --mpp-rescale PATTERN FACTOR: rescale D-related columns for movies whose movie_name matches PATTERN by FACTOR. Repeatable.

Pooling sources

1. Per-movie subdirectories (*_diffusion_coefficients.csv, *_step_sizes.csv) in input_dir/.
2. Flat pre-pooled files at the root: *_diffusion_coefficients.csv (must contain a movie_name column). Condition is inferred from --conditions tokens, then from filename, with _source_file tagged so loc-error subtraction can be applied per-experiment.

Loc-error subtraction: when --subtract-loc-error is set, D is corrected by the median D of an "immobile" proxy population (alpha < --loc-error-alpha-max, n_spots ≥ --loc-error-min-nspots). Raw values are preserved as <d_col>_raw. Grouping is (_source_file, condition) if source-file info is present, otherwise condition alone.

D column precedence: D_fit_um2_s → D_last_um2_s → D_um2_s.

Outputs: compare_conditions/ with figures (fig1_… to fig11_…), summary_statistics.csv, gmm_subpopulations.csv, gmm_brightness.csv, brightness_x_diffusion.csv, statistical_tests.csv.

spt-pipeline analyze compare-conditions /data/tracking_results
spt-pipeline analyze compare-conditions /data/pooled \
    --conditions NT Tg30 \
    --min-nspots 20 --min-alpha 0.3 \
    --subtract-loc-error \
    --mpp-rescale NT_0 2.25 --mpp-rescale Tg30_0 2.25

---

Plotting Operations (plot)

plot steps-brightness

Creates 2D heatmaps of step size vs brightness and step-size-colored track overlays.

spt-pipeline plot steps-brightness <input_dir> [options]

Key options: --make {both,heatmaps,tracks}, --stepsize-max, --bins-x, --bins-y, --count-cap, --units-label, --line-width, --invert-lut-tracks, --min-track-length, --brightness-col, --x-col, --y-col, --strip-datecodes, --workers.

plot dwell

spt-pipeline plot dwell <input_folder> [filtering options]

Density-normalized dwell-time histograms with GMM fits per condition (using _tracks.csv files and groupnames.csv). Filters: --cutoff, --mass, --size, --ecc, --signal, --raw_mass, --ep, --frame, --particle, --total_mass.

plot brightness-heatmap - Condition-pooled brightness-step heatmaps

spt-pipeline plot brightness-heatmap <input_dir> [options]

Pools replicates by condition (inferred from filename tokens or --conditions), computes step sizes, and renders 2D heatmaps with count caps and red saturation overlays. Supports --log-counts and explicit y-limits.

plot brightness-map - Brightness-colored track overlays

spt-pipeline plot brightness-map <input_dir> [options]

Per-file brightness-colored track overlays (scatter or polylines via --draw-lines), with quantile-based brightness clipping.

---

Drawing Operations (draw)

draw movie-tracks

spt-pipeline draw movie-tracks <input_dir> [--tail-length N] [--line-width W] [--channel N]

Renders movie frames with per-particle track tails, useful for figure-quality overlays.

---

Utility Operations (utils)

utils split-metrics - Split metrics by condition

spt-pipeline utils split-metrics <input_csv> [--cond-col COL] [--params PARAM1 PARAM2 ...]

Reads a master metrics CSV and writes one CSV per parameter (with conditions as columns) to a split metrics by condition/ directory. Defaults to common parameters (mean_step_px, immobile_frac, mean_brightness, median_sigma_px, D_uncorr, D_corr, alpha, dwell_time, total_mass, n_spots, …) when --params is omitted.

---

End-to-End Workflows

workflow auto

spt-pipeline workflow auto <input_dir> -o <outdir> [options]

Drives the typical track-trackpy → analyze-brightness-mobility → plot pipeline against a single input directory.

Complete: ND2 → analysis

spt-pipeline prep nd2-to-tiff /data/nd2_files
spt-pipeline track trackpy /data/images --diam 7 --minmass 110 --fps 33.33 --mpp 0.15
spt-pipeline analyze compare-conditions /data/images
spt-pipeline analyze brightness-mobility /data/images --dt 0.03
spt-pipeline plot steps-brightness /data/images --make both
spt-pipeline plot brightness-heatmap /data/images

TrackMate

spt-pipeline prep trackmate /data/trackmate_exports --out-dir /data/raw
spt-pipeline analyze brightness-mobility /data/raw --dt 0.03
spt-pipeline plot brightness-heatmap /data/raw --conditions vector HRAS HK1

Dwell time

spt-pipeline track dwelltime /data/nd2_movies --diam 7 --minmass 110 --fps 33.33
spt-pipeline plot dwell /data/nd2_movies --cutoff 4

See COMPATIBILITY.md for details on file detection and how commands chain together.

---

Two-Color Co-Diffusion (spt-coloc)

Independent companion CLI for bead-based channel registration and two-color co-diffusion analysis. Consumes track CSVs produced by spt-pipeline track trackpy (columns x, y, frame, particle, mass, …).

# 1. Derive registration from a two-channel bead image
spt-coloc register beads.tif -o registration/

# 2. Pair tracks across channels
spt-coloc pair ch1/ ch2/ registration/transform.json --mpp 0.122 --fps 50 -o coloc/

# 3. Analyze co-diffusion with shuffle controls
spt-coloc analyze coloc/ --n-shuffles 500 -o coloc/

---

Movie Metadata Summarizer (spt-metadata)

Lightweight prescreening utility, separate from the main pipeline. Recursively scans a directory for ND2/TIF/TIFF movies and writes one CSV row per movie with acquisition fields (frame count, exposure, pixel size, objective, lasers, …), leaving cells blank when metadata can't be found.

spt-metadata <input_directory> [--output FILE]

---

Parameter Reference

Common tracking parameters

Parameter	Units	Typical range	Description
--diam	pixels	5–15	Particle diameter for detection. Should match actual particle size.
--minmass	intensity units	50–500	Minimum integrated intensity for detection.
--fps	frames/second	1–100	Acquisition frame rate.
--mpp	µm/pixel	0.05–1.0	Spatial calibration.
--range	pixels	5–20	Maximum distance particles can move between frames for linking.
--memory	frames	0–5	Frames a particle can disappear and still be re-linked.
--percentile	0–100	0–100	Percentile threshold for detection.

Analysis parameters

Parameter	Units	Typical range	Description
--dt	seconds	0.01–1.0	Time per frame; used for MSD and diffusion calculations.
--immobile-threshold-px	pixels	0–1.0	Step-size threshold to classify particles as immobile.
--display-step-cap-px	pixels	1–10	Maximum step size for display.
--use-maxlag	boolean	true/false	Use the maximum lag time for MSD-based D.
--min-nspots	frames	5–50	Minimum n_spots for inclusion in compare-conditions.
--min-alpha	unitless	0–2	Minimum α for inclusion in compare-conditions.
--mpp-rescale	(pat, k)	-	Rescale D for movies matching pattern by factor k.

Output directory structure

track trackpy and track dwelltime create results in the input directory using movie-named subdirectories:

input_directory/
├── movie1/
│   ├── movie1_tracks.csv
│   ├── movie1_IMSD.csv
│   └── …
├── movie2/
│   └── …
└── groupnames.csv

---

Tips and Best Practices

1. Parameter tuning: start with defaults and adjust based on your data:
   • too many false positives → increase --minmass or --percentile.
   • tracks broken → increase --range or --memory.
   • particles missed → decrease --diam or --minmass.
2. Performance: all console subcommands accept --workers (default 8) for independent file/movie work. Use --workers 1 for serial debugging or low-memory systems.
3. File organization: keep input data organized by condition for easier downstream analysis.
4. Column detection: the pipeline auto-detects common column names; override with --x-col, --y-col, etc. when needed.
5. Output management: results default to the input directory. Use --outdir / --output-dir for custom locations.

---

Troubleshooting

Missing required columns

• Check that your CSV has x, y, frame, and track_id (or recognized alternatives).
• Override with --x-col, --y-col, etc.

nd2/trackpy required

• Install the relevant extras: pip install "spt-pipeline[tracking,imaging]".

No particles detected

• Decrease --minmass or --percentile.
• Confirm --diam matches your particle size.
• Verify image quality and contrast.

Tracks too short or broken

• Increase --range to allow larger movements.
• Increase --memory to allow gaps.
• Check frame rate and particle density.

---

Migration Notes

If you are coming from the pre-package single-file scripts (spt_pipeline.py, C_spt_pipeline.py, V10_spt_pipeline.py, C_spt_colocalization.py, summarize_movie_metadata.py):

• All python spt_pipeline.py … invocations become spt-pipeline … after pip install.
• All python C_spt_colocalization.py … invocations become spt-coloc ….
• All python summarize_movie_metadata.py … invocations become spt-metadata ….
• The compare-conditions flags --min-nspots, --min-alpha, --subtract-loc-error, --loc-error-alpha-max, --loc-error-min-nspots, and --mpp-rescale (previously only in V10_spt_pipeline.py) are now available in spt-pipeline analyze compare-conditions.
• The compare-conditions D-column precedence is now D_fit_um2_s → D_last_um2_s → D_um2_s (V10 behavior, previously inverted in C_spt_pipeline.py).
• Default tracking parameters from the older spt_pipeline.py are preserved (--mpp 0.15, --fps 33.33, --memory 2, --minmass 110, --percentile 40, --diam 7).

---

Publishing

python -m pip install -e ".[dev]"
python -m build
python -m twine check dist/*
python -m twine upload dist/*

Build from a clean dist/ directory and bump version in pyproject.toml before each PyPI upload.

---

License and Citation

MIT License - see LICENSE.

This pipeline consolidates functionality from multiple SPT analysis scripts. Please cite the underlying trackpy library and any relevant publications when using this tool.