GitHub  •  LinkedIn  •  X  •  Discord

Telekinesis Trackers

Telekinesis Trackers provides visual tracking runtimes for robotics and computer vision applications within the Telekinesis ecosystem. It runs compatible ONNX model bundles on CPU, with NVIDIA GPU execution available for CUTIE.

It includes:

• Multi-object mask propagation with CUTIE
• Point tracking with TAPIR
• Click-assisted mask initialization with RITM

Supported Trackers

Runtime	Task	Execution
CutieTracker	Multi-object mask propagation	CPU / CUDA
MaskInitializer	RITM click-assisted masks	CPU only
TapirTracker	Offline or causal point tracking	CPU only

Installing the GPU extra does not enable GPU execution for the CPU-only adapters. The table describes implemented runtime paths, not tracking-quality or performance benchmarks. CUDA hardware tests are opt-in; see Tests.

Contributor setup, model exporting, S3 publishing, and tests are documented in DEVELOPMENT.md.

Release Model

Telekinesis Trackers is in active development (pre-1.0). APIs and model bundle contracts may evolve between releases. See CHANGELOG.md for release notes and DEVELOPMENT.md for the development and release workflow.

Installation

Python 3.11 or newer is required. Create an isolated environment before installing the package. For example, with Conda:

conda create -n telekinesis-trackers python=3.11
conda activate telekinesis-trackers

Install the package from PyPI:

pip install telekinesis-trackers

The base package includes CPU ONNX Runtime, NumPy, Pillow, and requests, so it can run inference without an extra. [cpu] remains a compatibility alias.

Switch to NVIDIA GPU

After installing the package, replace CPU ONNX Runtime with the GPU runtime and install its CUDA libraries and CuPy:

pip uninstall -y onnxruntime onnxruntime-gpu
pip install "onnxruntime-gpu[cuda,cudnn]>=1.21,<1.27" "cupy-cuda12x[ctk]>=14,<15"

Uninstalling both runtime distributions first also repairs environments where both were installed, since they share the same Python module directory. pip install "telekinesis-trackers[gpu]" alone installs both runtime packages; if you use that extra, run the replacement commands above afterward.

Because the package declares CPU ONNX Runtime as a dependency, pip check will report onnxruntime as missing after this switch. Pip does not recognize the GPU distribution as its replacement. Upgrading or reinstalling telekinesis-trackers may restore the CPU dependency; repeat the replacement commands afterward.

The distribution is named telekinesis-trackers; import it in Python as telekinesis.trackers. For installation from the GitLab package registry, see the package installation guide.

PyTorch, Transformers, CUTIE, and TAPIR are not runtime dependencies.

GPU execution needs a CUDA 12-compatible NVIDIA driver. The commands above install the user-space CUDA libraries, and the ONNX Runtime upper bound keeps it on CUDA 12. See the ORT requirements and CuPy installation guide.

Verify Installation

Check that the package imports successfully (this does not load models or test ONNX Runtime/CUDA initialization):

from telekinesis import trackers

print("Available CUTIE tracker:", trackers.CutieTracker)

Usage

Automatic Model Loading

CUTIE, RITM, and TAPIR download their pretrained bundles on first use:

from telekinesis.trackers import CutieTracker, MaskInitializer

cutie = CutieTracker()
ritm = MaskInitializer()

Models are fetched from https://assets.telekinesis.ai/trackers/, validated, and extracted under:

~/.cache/telekinesis/trackers/cutie/480x864-dynamic
~/.cache/telekinesis/trackers/ritm/480x864-20
~/.cache/telekinesis/trackers/tapir-online/causal-2
~/.cache/telekinesis/trackers/tapir-offline/offline-2-2frames

The cached bundle is reused without another network request. The asset origin, cache location, progress display, and network settings are fixed. Pass model_dir only to use a local bundle instead.

CUTIE Mask Tracking

Frames are uint8 RGB arrays shaped (H,W,3). Seed labels are integer arrays shaped (H,W) where zero is background and values 1–255 are object IDs.

from telekinesis.trackers import CutieTracker

tracker = CutieTracker()
tracker.seed(seed_labels, first_rgb)
labels, alive_fraction = tracker.execute(next_rgb)

Use CutieTracker(device="cuda") (or "cuda:N") to select a GPU explicitly, and device="cpu" to force CPU execution. The default "auto" selects CUDA when ONNX Runtime advertises it, otherwise CPU. A selected CUDA backend that cannot initialize raises an error; it does not silently run the tracker on CPU. tracker.device reports the selected device.

On CUDA, graph features and recurrent state stay on the GPU through I/O binding; CuPy performs memory attention and output resizing there. Input validation and frame preprocessing remain on CPU; returned labels are NumPy arrays. Existing float32 bundles work without re-exporting. Unsupported graph operators may still use ONNX Runtime's CPU provider.

The pretrained graph has 480x864 padded dimensions and a dynamic object axis. Frames can be reduced with max_internal_size; objects are selected from the IDs present in the seed mask. Use correct(image_rgb, labels) to replace the current mask and memory state, and reset_session() to clear the tracker.

RITM Mask Initialization

MaskInitializer creates one object's mask from positive and negative (x, y) clicks:

from telekinesis.trackers import MaskInitializer

initializer = MaskInitializer()
probability = initializer.predict_proba(
    image_rgb,
    positive_points=[[320, 200]],
)
mask = initializer.predict(
    image_rgb,
    positive_points=[[320, 200]],
    negative_points=[[20, 20]],
    previous_mask=probability,
)

Images and previous-mask probabilities are resized internally, and results are returned at the original image size. Calls are stateless; pass the preceding probability map when refining the same object. The pretrained bundle accepts up to 20 positive and 20 negative clicks.

TAPIR Point Tracking

TAPIR runs on CPU. Online (causal) mode processes frames sequentially and downloads the default bundle on first use:

from telekinesis.trackers import TapirTracker

tracker = TapirTracker(mode="online")  # Also the default for TapirTracker()
tracks = tracker.seed(first_rgb, query_points_xy)
tracks = tracker.step(next_rgb)

Offline mode processes a complete clip together:

tracker = TapirTracker(mode="offline")
tracks = tracker.track(video_rgb, query_points_xy)

Both default bundles require exactly two query points and use 256x256 processing. The default offline bundle requires exactly two frames; the online bundle accepts successive frames through step(). Switching mode does not make these dimensions dynamic.

The bundles are cached under ~/.cache/telekinesis/trackers/tapir-online/causal-2 and ~/.cache/telekinesis/trackers/tapir-offline/offline-2-2frames. Cached bundles are reused. Pass model_dir to use a custom bundle; its mode is inferred unless you explicitly specify one. An explicit mode must match the bundle. "causal" is accepted as an alias for "online"; tracker.mode retains the manifest value "causal" or "offline".

Resources

• Development guide: contributor setup, model exports, tests, and CI/CD
• Changelog: release notes
• Telekinesis Examples: examples across the Telekinesis ecosystem
• Telekinesis Documentation: SDK documentation

Support

For issues and questions:

• Open an issue in this repository with the tracker name, runtime extra, and steps to reproduce.
• Contact the Telekinesis development team at support@telekinesis.ai or on Discord.