Skip to main content

Marimba

Marimba: typed, calibrated decisions about short and streaming audio windows (choice / yes-no / score), built on a frozen Qwen3-ASR-0.6B encoder.

An open-source project by charcot.ai. Source code.

Documentation and release scaffold

Start with the documentation index, executed notebooks, model card, or Gradio Space. Decoder context above eight seconds is unvalidated. No real-time guarantee is claimed; see benchmarks for measured Mac CPU latency. Converted head/readout weights are Apache-2.0; see the model card.

Development interface

Install with uv sync --extra datasets. Core installation does not install an inference engine. Use the deterministic synthetic model to build integrations without downloads:

import numpy as np
from marimba import Question
from marimba.testing import FakeMarimba

model = FakeMarimba()
questions = [Question.noul("Is someone speaking?")]
decisions = model(np.zeros(16000, dtype=np.float32), questions, sampling_rate=16000)
stream = model.stream(questions)
updates = stream.push(np.zeros(4000, dtype=np.float32))
updates += stream.flush()

Question.score("Has the speaker finished?") and Question.choice("What language?", ["English", "French"]) select other result kinds. Pass task= for custom wording. Quoted terms are exposed as question.term. Real language inference takes ISO codes or full English names. An Other option collects the softmax mass of all supported languages not explicitly listed: Question.choice("What language is being spoken?", ["English", "Portuguese", "Spanish", "Other"], task="language"). Decision.to_dict() and StreamUpdate.to_dict() return JSON-safe dictionaries.

model.batch(audios, questions, batch_size=16) accepts decoded audio mappings or files. dataset.map(model.map_fn(questions), batched=True) adds marimba_0, marimba_1, etc., one structured decision column per question. Arrays require their sampling rate; files and lazy audio decoders carry it. Two-dimensional arrays are samples by channels; lazy decoder tensors are channels by samples. Audio is mixed to mono and resampled to 16 kHz.

Streaming accepts mono int16 or floating-point samples at 16 kHz, without re-decoding. Each complete hop emits one update; flush() emits a final partial hop once, with no extra silence added to the clock. Windows retain the latest eight seconds, left-padding short inputs with a validity mask. Non-16-kHz stream input is rejected: resample before ingest. latency_ms measures synchronized inference for all questions on one assembled window, excluding capture, decoding, and resampling. Batch currently uses sequential inference; batch_size is reserved for backend batching.

Run marimba doctor for optional runtime discovery, or marimba bench --fake --questions 4 --hops 100 --warmup 10 for p50/p95 per-hop latency. Fake results are synthetic, not calibrated predictions.

Pretrained torch lanes

Install marimba-audio[torch] (or uv sync --extra torch) and authenticate with Hugging Face using HF_TOKEN or hf auth login if your model repository requires access:

from marimba import Marimba, Question

model = Marimba.from_pretrained(backend="torch", device="cpu", dtype="float32")
questions = [
    Question.score("Has the speaker finished?"),
    Question.choice("What language?", ["en", "pt", "es", "fr", "de"]),
    Question.noul('Did someone say "hello"?'),
]

The default manifest revision is pinned in marimba.weights.DEFAULT_MODEL_REVISION. An explicit revision= can be a commit, tag, or branch; it is resolved once to an immutable commit before fetching the manifest and SHA256-verified per-lane files. The frozen Qwen3-ASR-0.6B encoder revision is pinned separately. Only torch is supported for these lanes; selecting ONNX fails explicitly. Heavy imports and downloads happen only at load time.

Each window runs the audio tower once. Its native L18 frames, quantized to float16 as in the readout extraction pipeline, feed valid-frame tail pooling over the last two seconds, a standardized linear readout, and Platt calibration. Language choices use the sum of native language-name token log probabilities, softmaxed within the supplied options. Unsupported options and duplicate aliases raise an error before inference. Without Other, probabilities are normalized within that question's choices; with Other, all 30 supported languages are scored before aggregation.

All term questions share greedy decoding plus four beam hypotheses. Unicode normalization removes accents and punctuation, retains non-Latin letters, and matches exact token sequences. Term probabilities are currently uncalibrated raw fractions of five hypotheses, not Platt-calibrated probabilities; their capability is validated and states this limitation. No calibration is fitted at inference time.

See Calibrate on your domain to fit and persist a per-question Platt map from your own binary labels. This does not train model weights.

Voice activity uses a frozen question-conditioned head, validated with Portuguese prompts and held-out Arabic, Hindi, Korean and Russian prompts; recognized questions use the canonical prompt; custom wording passes through unchanged. It uses L9 features interpolated to 100 frames from the same encoder pass as turn-end. The head uses the physical valid-frame mask, a frozen Whisper question embedding, a small text tower, grapheme-encoded Portuguese sim, and sigmoid on the head logit. Probabilities are temperature-scaled on validation; the default threshold is 0.225. The validated two-second tail option is documented in capabilities.

Three L12 encoder readouts share that same pass:

Task Evidence / limitation
same_speaker AMI AUROC 0.998, VoxConverse 0.996; both 4 s halves single-speaker; on par with ECAPA
speaker_change dominant speaker in last 2 s vs previous 6 s; AMI AUROC 0.966, VoxConverse 0.980
speaker_count 0/1/2/3+ speakers; accuracy AMI 0.778, VoxConverse 0.859; undercounts 3+

Use Question.noul("Is this the same speaker?"), Question.score("Did the speaker change?"), or Question.choice("How many speakers?", ["0", "1", "2", "3+"]). Both binary lanes support noul/score; count requires exactly those four options in any order. Diarization requires eight valid seconds: partial or masked windows return None for value, probability and probabilities. Audio-class is unavailable and does not receive fallback predictions. PII is experimental when a caller supplies a text detector and unavailable otherwise; the package does not include a detector. See PII questions. Empty real-model audio raises an error; silence with valid samples is supported.

marimba bench --per-question --audio public_speech.wav --json bench.json measures the four fixed benchmark questions (turn_end, voice_activity, language, and term), each question alone, and the encoder. The live demo also shows speaker_change and speaker_count cards. Warmup and model loading are excluded. Language options are en, pt, and es.

See lane scheduling and long context for immutable snapshots, reusable encoded windows, independent encoder/decoder locks, decoder cadence, and contexts up to MAX_DECODER_SECONDS. Direct lane APIs report lane-only decision latency; model() and stream updates retain total-window latency. Real-model tests require authentication/downloads and are excluded by default; run them explicitly with uv run pytest -m real_model.

Streaming on slow devices

stream.ingest(chunk) -> None only updates the ring buffer. Call stream.decide() -> StreamUpdate | None to score the latest window on demand; it returns None until audio arrives. A decision snapshots the audio, valid mask, and timestamps under a lock, then releases the lock before inference, so another thread can keep ingesting. The returned timestamps always describe the snapshot, not audio received during inference. Repeated decide() calls can score the same window; each call restarts hop accounting.

model.stream(questions, coalesce=True) makes push() emit at most one update: the latest due hop in that chunk. Trailing samples remain buffered for the next hop or flush(). Skipped hops never clear the context. The default coalesce=False still emits every hop. ingest() does not queue historical inference; switching back to push() resumes hop accounting from the retained partial hop. These APIs also work with FakeMarimba. For Pipecat, set UserTurnStrategies(stop=[TurnAnalyzerUserTurnStopStrategy(turn_analyzer=MarimbaTurnAnalyzer())]); see Pipecat turn analyzer. For LiveKit Agents, set AgentSession(turn_handling={"turn_detection": MarimbaTurnDetector()}); the detector warms the model in the background and defaults to a 2-second prediction_timeout. See LiveKit turn detector.

Metadata

Release files for marimba-audio 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for marimba-audio 0.1.1
File Size Uploaded Provenance Actions
marimba_audio-0.1.1.tar.gz 2.0 MB With provenance Details for marimba_audio-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for marimba-audio 0.1.1
File Interpreter ABI Platform Actions
marimba_audio-0.1.1-py3-none-any.whl Python 3 none any Details for marimba_audio-0.1.1-py3-none-any.whl

Total release size: 2.1 MB

Release files / marimba_audio-0.1.1.tar.gz

Download URL marimba_audio-0.1.1.tar.gz
Size 2.0 MB
Tags Source
SHA-256 checksum
How to use checksums
ea53608aee3b106b06d31ef788f35f75d518dc708c592355693fd6608fc5264d
BLAKE2b-256 checksum
How to use checksums
83e4aeb36eaed403b4089993ceb8cb94657b2ba97dafd1f794ffc241d228c5a6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Release files / marimba_audio-0.1.1-py3-none-any.whl

Download URL marimba_audio-0.1.1-py3-none-any.whl
Size 78.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4edf76a732b7b6c1ad46747e8c740ccbbede9b457813393f35916fdcafa6b17d
BLAKE2b-256 checksum
How to use checksums
fe7b3889573b0dfde9913485856f13976ac04bc57eaa1d7e53574d959291b821
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Security for marimba-audio 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for marimba_audio-0.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page