ByoTrack

ByoTrack is a Python library for tracking biological objects in microscopy videos (2D and 3D).

Its goal is to provide a fast, modular, and research-friendly tracking framework that integrates seamlessly with the Python scientific ecosystem and established bioimage analysis platforms such as Fiji, Icy and Napari.

ByoTrack defines a modular tracking API that can be easily extended to design and evaluate new methods. It also includes implementations of several state-of-the-art detection and tracking approaches following this API.

Some components are implemented natively in Python (e.g. WaveletDetector, KalmanLinker, KOFT, RTSSmoother, EMC2Stitcher), while others wrap existing tools (e.g. StarDistDetector, IcyEMHTLinker, TrackMateLinker, TrackAstraLinker), integrating with external software.

In addition, ByoTrack provides utilities for data loading and preprocessing, as well as evaluation and visualization of tracking results.

[!NOTE] ByoTrack has been primarily developed for scenarios involving up to a few thousand targets in 2D or 3D microscopy data. Some components assume that individual frames fit in memory, which may limit scalability to very large 3D volumes. If you encounter limitations with your use case, feel free to open an issue or contribute a pull request.

[!WARNING] v2.0.0 introduces breaking changes to the Detections and Video APIs. If you are upgrading from v1.x, please read the Changelog for details and migration guidance.

🏆 ByoTrack (PAST-FR) won the Cell Linking Benchmark of the Cell Tracking Challenge with its SKT/KOFT implementation (see our paper for details).

Installation

pip

pip install byotrack

Some components require additional dependencies that are not installed with the library by default. For these components, you need to install their specific dependencies. Here is the complete list:

• StarDistDetector
  • StarDist (+ Tensorflow): Install StarDist
• IcyEMHTLinker
  • Icy: Download Icy
  • Spot Tracking Blocks plugin: Install an Icy plugin
• TrackMateLinker
  • Fiji: Download Fiji
• TrackAstraLinker & TrackOnStraLinker
  • TrackAstra: Install trackastra
• byotrack.geff (io)
  • GEFF: Install GEFF
• byotrack.napari (visualization)
  • Napari: Install Napari
• byotrack.visualize (visualization)
  • Matplotlib: Install Matplotlib.

From source

git clone git@github.com:raphaelreme/byotrack.git  # OR https://github.com/raphaelreme/byotrack.git
cd byotrack
pip install .

Getting started

import byotrack
import byotrack.napari.viewer  # Requires Napari

# Load some specific implementations
from byotrack.implementation.detector.wavelet import WaveletDetector
from byotrack.implementation.linker.frame_by_frame.kalman_linker import KalmanLinkerParameters, KalmanLinker
from byotrack.implementation.refiner.interpolater import ForwardBackwardInterpolater

# Read a video from a path, normalize and select channel
video = byotrack.Video(video_path)  # File/folder on disk or directly from array in RAM
video = video.normalize(q_min=0.01, q_max=0.999)[..., :1]  # First channel. Ignore other channels if any.

byotrack.napari.visualize(video)

# First, let's detect targets with the Wavelet Spot Detector
## Smaller scale <=> search for smaller spots
## The noise threshold is linear with k. If you increase it, you will retrieve less spots.
detector = WaveletDetector(scale=1, k=3.0, min_area=5.0)
detections_sequence = detector.run(video)

byotrack.napari.visualize(video, detections_sequence)

# Then, let's use a KalmanLinker with default parameters estimated from the detections
specs = KalmanLinkerParameters(  # Check the documentation for more parameters
    track_building="detection",  # Position from detections (non-smoothed)
).estimate(detections_sequence)
linker = kalman_linker.KalmanLinker(specs)
tracks = linker.run(video, detections_sequence)

# Last, you may filter short tracks and interpolate over missed detections if any
tracks = [track for track in tracks if len(track) > 50]  # Keep only tracks than last 50 frames
tracks = ForwardBackwardInterpolater().run(video, tracks)  # Interpolate over missed detections

# Final visualization
byotrack.napari.viewer.visualize(video, detections_sequence, tracks)

# Export tracks
byotrack.Track.save(tracks, output_path)

Please refer to the official documentation (https://byotrack.readthedocs.io/en/latest/).

Tracking Pipeline

ByoTrack implements a modular multi-step tracking pipeline:

Video → Detection → Detection Refinement → Linking → Track Refinement

Detection

Detect objects in each frame.

Implemented detectors:

• Wavelet detector [2]
  Similar to the Icy implementation but rewritten in PyTorch

• StarDist [3]
  Wrapper for inference (training is performed using the official StarDist library)

Detection Refinement

Filter, refine, split and merge detections.

Implemented detection refiners:

• Detection filtering
  Filter detections based on intensity and size criteria.

• Watershed
  Convert semantic (binary) segmentation into instance segmentation with Watershed
  Can also be applied to instance segmentation to refine the instances found

Linking

Associate detections across frames.

Implemented linkers:

• Nearest-neighbor linking

  • Euclidean
  • Optical flow
  • Kalman filtering (SKT) [9]
  • KOFT (Kalman + Optical Flow Tracker) [9]
  • optional adaptive gating [12]
  • TrackAstra extension to support missed detections

• EMHT [4]
  Wrapper around the Icy implementation

• TrackMate / u-track [7]
  Wrapper around Fiji's TrackMate implementation [6,8]

• TrackAstra [13]
  Wrapper around the TrackAstra implementation

Track Refinement

Post-processing operations applied to tracks:

• Cleaning
  Remove outliers tracks based on length and motion criteria

• Gap Closing / Stitching
  Tracklet Stitching via EMC2 algorithm [5]

• Interpolation
  Replace miss-detection by an interpolated position
  Extrapolate tracks on the full temporal sequence

• Smoothing
  RTS optimal Kalman smoother

Data, Evaluation & Utilities

Data input

ByoTrack supports:

• Most standard video formats via OpenCV
• TIFF stacks
• Folder of images (sorted by name)
• Dedicated Python loading of the video (np.array) and detections (converted as a Detections object)

Note that microscope private formats are not supported but can be converted into TIFF manually using bftools

Optical flow

Optical flow can be used for:

• Linking detections
• Gap closing / Stitching
• Interpolation

Currently provide wrappers around implementations from:

• OpenCV (TVL1, Farneback) (Only 2D)
• Scikit-Image (ILK, TVL1) (2D + 3D)

Datasets

Built-in loaders for common benchmarks:

• Cell Tracking Challenge (CTC) [10]
• SINETRA [11]

Metrics

Evaluation utilities for:

• Segmentation
• Detection
• Tracking

Currently implemented:

• Cell Tracking Challenge metrics

More metrics will be added in future releases.

Cell Tracking Challenge

Our submission (PAST-FR) to the Cell Linking Benchmark of the Cell Tracking Challenge is available in the examples/ctc folder.

Contributing

Contributions are very welcome! Feel free to open an issue or submit a pull request.

Typical contributions could include:

• New detections or linking algorithms
• Dataset loaders
• Evaluation metrics
• New data format
• Track analysis methods

See the contribution guidelines.

Cite us

If you use ByoTrack in your research, please cite:

@article{hanson2024automatic,
  title={Automatic monitoring of neural activity with single-cell resolution in behaving Hydra},
  author={Hanson, Alison and Reme, Raphael and Telerman, Noah and Yamamoto, Wataru and Olivo-Marin, Jean-Christophe and Lagache, Thibault and Yuste, Rafael},
  journal={Scientific Reports},
  volume={14},
  number={1},
  pages={5083},
  year={2024},
  publisher={Nature Publishing Group UK London}
}

References

• [1] F. De Chaumont, S. Dallongeville, N. Chenouard, et al., "Icy: an open bioimage informatics platform for extended reproducible research", Nature methods, 2012.
• [2] J.-C. Olivo-Marin, "Extraction of spots in biological images using multiscale products", Pattern Recognition, 2002.
• [3] U. Schmidt, M. Weigert, C. Broaddus, and G. Myers, "Cell detection with star-convex polygons", MICCAI, 2018.
• [4] N. Chenouard, I. Bloch, and J.-C. Olivo-Marin, "Multiple hypothesis tracking for cluttered biological image sequences", IEEE TPAMI, 2013.
• [5] T. Lagache, A. Hanson, J. Perez-Ortega, et al., "Tracking calcium dynamics from individual neurons in behaving animals", PLoS Computational Biology, 2021.
• [6] J. Schindelin, I. Arganda-Carreras, E. Frise, et al., "Fiji: an open-source platform for biological-image analysis", Nature Methods, 2012.
• [7] K. Jaqaman, D. Loerke, M. Mettlen, et al., "Robust single-particle tracking in live-cell time-lapse sequences.", Nature Methods, 2008.
• [8] J.-Y. Tinevez, N. Perry, J. Schindelin, et al., "TrackMate: An open and extensible platform for single-particle tracking.", Methods, 2017.
• [9] R. Reme, A. Newson, E. Angelini, J.-C. Olivo-Marin and T. Lagache, "Particle tracking in biological images with optical-flow enhanced kalman filtering", IEEE ISBI, 2024.
• [10] M. Maška, V. Ulman, D. Svoboda, P. Matula, et al., "A benchmark for comparison of cell tracking algorithms", in Bioinformatics, 2014.
• [11] R. Reme, A. Newson, E. Angelini, J.-C. Olivo-Marin and T. Lagache, "SINETRA: a Versatile Framework for Evaluating Single Neuron Tracking in Behaving Animals", IEEE ISBI, 2025.
• [12] A. Genovesio, Z. Belhassine, and J.-C. Olivo-Marin, "Adaptive gating in Gaussian Bayesian multi-target tracking", IEEE ICIP, 2004.
• [13] B. Gallusser, and M. Weigert, "Trackastra: Transformer-based cell tracking for live-cell microscopy." ECCV, 2024.
• [14] N. Sofroniew, T. Lambert, G. Bokota, et al., "napari: a multi- dimensional image viewer for Python". Zenodo, 2022.