Skip to main content

DeepLabCut-live! SDKDLC LIVE!

Code style: black PyPI - Python Version Downloads Downloads Python package GitHub stars GitHub forks

License Image.sc forum Gitter Twitter Follow

This package contains a DeepLabCut inference pipeline for real-time applications that has minimal (software) dependencies. Thus, it is as easy to install as possible (in particular, on atypical systems like NVIDIA Jetson boards).

Performance: If you would like to see estimates on how your model should perform given different video sizes, neural network type, and hardware, please see: https://deeplabcut.github.io/DLC-inferencespeed-benchmark/

If you have different hardware, please consider submitting your results too! https://github.com/DeepLabCut/DLC-inferencespeed-benchmark

What this SDK provides: This package provides a DLCLive class which enables pose estimation online to provide feedback. This object loads and prepares a DeepLabCut network for inference, and will return the predicted pose for single images.

To perform processing on poses (such as predicting the future pose of an animal given it's current pose, or to trigger external hardware like send TTL pulses to a laser for optogenetic stimulation), this object takes in a Processor object. Processor objects must contain two methods: process and save.

  • The process method takes in a pose, performs some processing, and returns processed pose.
  • The save method saves any valuable data created by or used by the processor

For more details and examples, see documentation here.

Note :: alone, this object does not record video or capture images from a camera. This must be done separately, i.e. see our DeepLabCut-live GUI.

News!

Installation:

Please see our instruction manual to install on a Windows or Linux machine or on a NVIDIA Jetson Development Board. Note, this code works with tensorflow (TF) 1 or TF 2 models, but TF requires that whatever version you exported your model with, you must import with the same version (i.e., export with TF1.13, then use TF1.13 with DlC-Live; export with TF2.3, then use TF2.3 with DLC-live).

  • available on pypi as: pip install deeplabcut-live

Note, you can then test your installation by running:

dlc-live-test

If installed properly, this script will i) create a temporary folder ii) download the full_dog model from the DeepLabCut Model Zoo, iii) download a short video clip of a dog, and iv) run inference while displaying keypoints. v) remove the temporary folder.

DLC LIVE TEST

Quick Start: instructions for use:

  1. Initialize Processor (if desired)
  2. Initialize the DLCLive object
  3. Perform pose estimation!
from dlclive import DLCLive, Processor
dlc_proc = Processor()
dlc_live = DLCLive(<path to exported model directory>, processor=dlc_proc)
dlc_live.init_inference(<your image>)
dlc_live.get_pose(<your image>)

DLCLive parameters:

  • path = string; full path to the exported DLC model directory
  • model_type = string; the type of model to use for inference. Types include:
    • base = the base DeepLabCut model
    • tensorrt = apply tensor-rt optimizations to model
    • tflite = use tensorflow lite inference (in progress...)
  • cropping = list of int, optional; cropping parameters in pixel number: [x1, x2, y1, y2]
  • dynamic = tuple, optional; defines parameters for dynamic cropping of images
    • index 0 = use dynamic cropping, bool
    • index 1 = detection threshold, float
    • index 2 = margin (in pixels) around identified points, int
  • resize = float, optional; factor by which to resize image (resize=0.5 downsizes both width and height of image by half). Can be used to downsize large images for faster inference
  • processor = dlc pose processor object, optional
  • display = bool, optional; display processed image with DeepLabCut points? Can be used to troubleshoot cropping and resizing parameters, but is very slow

DLCLive inputs:

  • <path to exported model directory> = path to the folder that has the .pb files that you acquire after running deeplabcut.export_model
  • <your image> = is a numpy array of each frame

Benchmarking/Analyzing your exported DeepLabCut models

DeepLabCut-live offers some analysis tools that allow users to peform the following operations on videos, from python or from the command line:

  1. Test inference speed across a range of image sizes, downsizing images by specifying the resize or pixels parameter. Using the pixels parameter will resize images to the desired number of pixels, without changing the aspect ratio. Results will be saved (along with system info) to a pickle file if you specify an output directory.
python
dlclive.benchmark_videos('/path/to/exported/model', ['/path/to/video1', '/path/to/video2'], output='/path/to/output', resize=[1.0, 0.75, '0.5'])
command line
dlc-live-benchmark /path/to/exported/model /path/to/video1 /path/to/video2 -o /path/to/output -r 1.0 0.75 0.5
  1. Display keypoints to visually inspect the accuracy of exported models on different image sizes (note, this is slow and only for testing purposes):
python
dlclive.benchmark_videos('/path/to/exported/model', '/path/to/video', resize=0.5, display=True, pcutoff=0.5, display_radius=4, cmap='bmy')
command line
dlc-live-benchmark /path/to/exported/model /path/to/video -r 0.5 --display --pcutoff 0.5 --display-radius 4 --cmap bmy
  1. Analyze and create a labeled video using the exported model and desired resize parameters. This option functions similar to deeplabcut.benchmark_videos and deeplabcut.create_labeled_video (note, this is slow and only for testing purposes).
python
dlclive.benchmark_videos('/path/to/exported/model', '/path/to/video', resize=[1.0, 0.75, 0.5], pcutoff=0.5, display_radius=4, cmap='bmy', save_poses=True, save_video=True)
command line
dlc-live-benchmark /path/to/exported/model /path/to/video -r 0.5 --pcutoff 0.5 --display-radius 4 --cmap bmy --save-poses --save-video

License:

This project is licensed under the GNU AGPLv3. Note that the software is provided "as is", without warranty of any kind, express or implied. If you use the code or data, we ask that you please cite us! This software is available for licensing via the EPFL Technology Transfer Office (https://tto.epfl.ch/, info.tto@epfl.ch).

Community Support, Developers, & Help:

This is an actively developed package and we welcome community development and involvement.

Reference:

If you utilize our tool, please cite Kane et al, eLife 2020. The preprint is available here: https://www.biorxiv.org/content/10.1101/2020.08.04.236422v2

@Article{Kane2020dlclive,
  author    = {Kane, Gary and Lopes, Gonçalo and Sanders, Jonny and Mathis, Alexander and Mathis, Mackenzie},
  title     = {Real-time, low-latency closed-loop feedback using markerless posture tracking},
  journal   = {eLife},
  year      = {2020},
}

Metadata

Release files for deeplabcut-live 1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for deeplabcut-live 1.0
File Size Uploaded Provenance Actions
deeplabcut-live-1.0.tar.gz 23.1 kB Without provenance Details for deeplabcut-live-1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for deeplabcut-live 1.0
File Interpreter ABI Platform Actions
deeplabcut_live-1.0-py3-none-any.whl Python 3 none any Details for deeplabcut_live-1.0-py3-none-any.whl

Total release size: 51.3 kB

Release files / deeplabcut-live-1.0.tar.gz

Download URL deeplabcut-live-1.0.tar.gz
Size 23.1 kB
Tags Source
SHA-256 checksum
How to use checksums
d6a4c6018981fcf537dec5850d50a4e38f0eedb77e733a1686af9db1f73bd80c
BLAKE2b-256 checksum
How to use checksums
3cb0ee7360496056cca284645f8688a399664e75054aafbc7457757371db1e50
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.1.1 pkginfo/1.4.2 requests/2.22.0 setuptools/45.2.0 requests-toolbelt/0.8.0 tqdm/4.30.0 CPython/3.8.5

Release files / deeplabcut_live-1.0-py3-none-any.whl

Download URL deeplabcut_live-1.0-py3-none-any.whl
Size 28.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
95c43bc95579e68be0e664457d21ede6c8aea40628b9f44fc6ea0f024db88f7f
BLAKE2b-256 checksum
How to use checksums
27a76cc7243db0f7de663e57f8b96abadd0c460d38d88efaf6ae1a7485f0ca5d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.1.1 pkginfo/1.4.2 requests/2.22.0 setuptools/45.2.0 requests-toolbelt/0.8.0 tqdm/4.30.0 CPython/3.8.5

Security for deeplabcut-live 1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page