senselab

This Python package streamlines, optimizes, and enforces best open-science practices for processing and analyzing behavioral data (primarily voice and speech, but also text and video) using robust reproducible pipelines and utilities.

Quick start

from senselab.audio.data_structures import Audio
from senselab.audio.tasks.preprocessing import resample_audios
from senselab.audio.tasks.features_extraction import extract_features_from_audios
from senselab.audio.tasks.speech_to_text import transcribe_audios

audio = Audio(filepath='path_to_audio_file.wav')
print(audio.sampling_rate)
# ➡️ 44100

[resampled_audio] = resample_audios([audio], resample_rate=16000)
print(resampled_audio.sampling_rate)
# ➡️ 16000

audio_features = extract_features_from_audios([audio])
print(audio_features[0].keys())
# ➡️ dict_keys(['opensmile', 'praat_parselmouth', 'torchaudio', 'torchaudio_squim', ...])

transcript = transcribe_audios([audio])
print(transcript)
# ➡️ "The quick brown fox jumps over the lazy dog."

For more detailed information, check out our Documentation and our Tutorials.

💡 Tip: Many tutorials include Google Colab badges and you can try them instantly without installing anything on your local machine.

Why should you use senselab?

• Modular design: Easily integrate or use standalone transformations for flexible data manipulation.
• Pre-built pipelines: Access pre-configured pipelines to reduce setup time and effort.
• Reproducibility: Ensure consistent and verifiable results with fixed seeds and version-controlled steps.
• Easy integration: Seamlessly fit into existing workflows with minimal configuration.
• Extensible: Modify and contribute custom transformations and pipelines to meet specific research needs.
• Comprehensive documentation: Detailed guides, examples, and documentation for all features and modules.
• Performance optimized: Efficiently process large datasets with optimized code and algorithms.
• Interactive examples: Jupyter notebooks provide practical examples for deriving insights from real-world datasets.
• senselab AI: Interact with your data through an AI-based chatbot. The AI agent generates and runs senselab-based code for you, making exploration easier and giving you both the results and the code used to produce them (perfect for quick experiments or for users who prefer not to code).

Adaptive audio analysis (uncertainty-driven)

senselab can analyze a recording with its full task suite (diarization, scene/event tagging, quality metrics, multi-model ASR + forced alignment, speaker embeddings), quantify where the models are uncertain along three temporal axes — presence (is someone speaking?), identity (who?), and utterance (what was said?) — and then act on that uncertainty: a deterministic, budgeted loop re-processes only the uncertain regions (extra ASR models, embedding re-clustering, overlap detection), fuses a consensus transcript/diarization, and explains any residual uncertainty instead of hiding it. Design + results: specs/20260723-225523-dynamic-uncertainty-workflow/.

It runs in two steps:

# Step 1 — analyze: run every model on the recording (results are content-addressably
# cached, so re-runs are cheap). `--enhancement auto` adds a triage round 0 that skips
# the speech-enhancement pass on clean audio and stops early on silent recordings.
uv run python scripts/analyze_audio.py path/to/recording.wav --enhancement auto
# → artifacts/analyze_audio/<name>_<timestamp>/  (per-task JSONs, 9 uncertainty
#   parquets, Label Studio bundle, disagreements.json, timeline.png)

# Step 2 — adapt: run the uncertainty-driven loop over that run directory.
uv run python scripts/adaptive_loop.py artifacts/analyze_audio/<run_dir> \
    --cache-dir artifacts/analyze_audio_cache \
    --ground-truth path/to/labelstudio_export.json   # optional: scores vs human labels

The loop writes, under the run directory (or --out):

• final/transcript.json — consensus word-level transcript (family-weighted voting across all ASR models) with speaker attribution, per-word confidence, and alternates where models disagree;
• final/diarization.json + final/presence.parquet — refined speaker segments (embedding change-point + re-clustering repair) and the fused speech-presence track;
• final/convergence.json + final/iterations.json — what the loop did and why: every intervention (fired / deferred / blocked) with trigger values and measured uncertainty deltas, budget accounting, and regions marked converged / irreducible (with a machine-readable reason);
• final/timeline.png — ground truth (if given) vs presence / identity / utterance uncertainty per round, interventions, and the confidence-colored fused words;
• final/labelstudio_{tasks,config}.{json,xml} + disagreements_resolved.json — the original Label Studio bundle with final__* consensus tracks added, and the round-1 disagreements annotated with their resolutions.

Useful knobs (all thresholds/budgets/models live in a versioned policy file — src/senselab/audio/workflows/audio_analysis/adaptive/policy/default.yaml — overridable via --policy my_policy.yaml): --max-rounds, --aggregator, per-run intervention budgets, ASR reserve/escalation pools, and identity-repair parameters. Runs are deterministic: identical inputs + policy produce byte-identical decision logs. HF_TOKEN enables the gated pyannote/segmentation-3.0 overlap detector; without it the loop degrades gracefully and records the skipped intervention in convergence.json → next_actions.

⚠️ System Requirements

1. If on macOS, this package requires an ARM64 architecture due to PyTorch 2.2.2+ dropping support for x86-64 on macOS.

   ❌ Unsupported systems include:

   • macOS (Intel x86-64)
   • Other platforms where dependencies are unavailable

   To check your system compatibility, please run this command:

   python -c "import platform; print(platform.machine())"
   

   If the output is:

   • arm64 → ✅ Your system is compatible.
   • x86_64 → ❌ Your system is not supported.

   If you attempt to install this package on an unsupported system, the installation or execution will fail.

2. FFmpeg is required by some audio and video dependencies (e.g., torchaudio). Please make sure you have FFmpeg properly installed on your machine before installing and using senselab (see here for detailed platform-dependent instructions).

3. CUDA libraries matching the CUDA version expected by the PyTorch wheels (e.g., the latest pytorch 2.8 expects cuda-12.8). To install those with conda, please do:

• conda config --add channels nvidia

• conda install -y nvidia/label/cuda-12.8.1::cuda-libraries-dev

  Hosts with newer system CUDA (e.g., CUDA 12.9): the subprocess-venv backends (nemo-canary-qwen, nemo, qwen-asr) auto-detect the host's CUDA version via nvidia-smi and route their torch/torchaudio installs through the matching PyTorch wheel index (cu128 / cu126 / cu124 / cu121 / cpu). No manual configuration needed.

  Internal mirrors / unsupported CUDA / CPU fallback: set the SENSELAB_TORCH_INDEX_URL environment variable to override the chosen index. Common values:

  # Force CPU wheels (e.g. testing CPU path on a GPU host, or unsupported CUDA)
  export SENSELAB_TORCH_INDEX_URL=https://download.pytorch.org/whl/cpu
  
  # Internal PyPI mirror that proxies PyTorch wheels
  export SENSELAB_TORCH_INDEX_URL=https://pypi.internal.example.com/pytorch/cu128
  

  When no compatible torch+torchaudio binary pair exists for your host (rare; happens in the days after a CUDA major release), installation fails with a named SenselabCudaCompatibilityError that lists the detected host CUDA, the attempted index URL, and the recommended action — no opaque stack traces from inside torchaudio.

4. Docker is required and must be running for some video models (e.g., MediaPipe-based estimators). Please follow the official installation instructions for your platform: Install Docker.
5. Some functionalities rely on HuggingFace models, and increasingly, models require authentication and signed license agreements. Instructions on how to generate a Hugging Face access token can be found here: https://huggingface.co/docs/hub/security-tokens

• You can provide your HuggingFace token either by exporting it in your shell:

  export HF_TOKEN=your_token_here
  

• or by adding it to your .env file (see .env.example for reference).

Installation

Install this package via:

pip install 'senselab[all]'

Or get the newest development version via:

pip install 'git+https://github.com/sensein/senselab.git#egg=senselab[all]'

If you want to install only audio dependencies, you do:

pip install 'senselab'

To install articulatory, video, text, and senselab-ai extras, please do:

pip install 'senselab[articulatory,video,text,senselab-ai]'

senselab AI (our AI-based chatbot)

Development (with uv)

uv sync --extra senselab-ai
uv run senselab-ai

Production (with pip)

pip install 'senselab[senselab-ai]'
senselab-ai

Once started, you can open the provided JupyterLab interface, setup the agent and chat with it, and let it create and execute code for you.

For a walkthrough, see: tutorials/senselab-ai/senselab_ai_intro.ipynb.

Contributing

We welcome contributions from the community! Before proceeding with that, please review our CONTRIBUTING.md.

Funding

senselab is mostly supported by the following organizations and initiatives:

• McGovern Institute ICON Fellowship
• NIH Bridge2AI Precision Public Health (OT2OD032720)
• Child Mind Institute
• ReadNet Project
• Chris and Lann Woehrle Psychiatric Fund

Acknowledgments

senselab builds on the work of many open-source projects. We gratefully acknowledge the developers and maintainers of the following key dependencies:

• PyTorch, Torchvision, Torchaudio deep learning framework and audio/vision extensions
• Transformers, Datasets, Accelerate, Huggingface Hub training and inference utilities plus (pre-)trained models and datasets
• Scikit-learn, UMAP-learn machine learning utilities
• Matplotlib visualization toolkit
• Praat-Parselmouth, OpenSMILE, SpeechBrain, SPARC, Pyannote-audio, Coqui-TTS, NVIDIA NeMo, Vocos, Audiomentations, Torch-audiomentations speech and audio processing tools
• NLTK, Sentence-Transformers, Pylangacq, Jiwer text and language processing tools
• OpenCV, Ultralytics, mediapipe, Python-ffmpeg, AV computer vision and pose estimation
• Pydantic, Iso639, PyCountry, Nest-asyncio validation, and utilities
• Ipywidgets, IpKernel, Nbformat, Nbss-upload, Notebook-intelligence Jupyter and notebook-related tools

We are thankful to the open-source community for enabling this project! 🙏