Skip to main content

whensaid

Search what was said in your local video and audio files, from your AI agent.

Point it at folders of recordings (talks, courses, meetings, interviews, podcasts) or at a YouTube playlist. Your agent (Claude Code, Cursor, ...) calls search("when did they talk about X") and gets back the file, the exact timestamp and the passage; for web videos, a link that opens the video at that moment. Everything runs on your machine, and nothing is uploaded.

Status: pre-alpha. Tested on Linux (CI also runs on macOS). Runs on an NVIDIA GPU or on CPU. English only.

A worked example

terminal demo: two searches that find the passage and one off-topic question that returns nothing

The recording above is real whensaid output (rendered by scripts/make_demo.py), run in a terminal rather than from an agent session. Text version:

The index here is 198 public-domain recordings (65 hours of audiobooks and old radio). The question is paraphrased, not a quote:

$ whensaid search "a mouse tribe claiming rights over everyone else" -n 1
strong  the_consolation_of_philosophy_..._boethius_64kb.mp3  0:01:38  and over whom ye exercise your fancied
lordship suppose now that in the mouse tribe there should rise up one claiming rights and powers for himself above the rest ...

Through MCP the agent gets the same thing as JSON (file, path, start, at, text, match, relevance, and link for web videos) and can call get_transcript to read around the hit. About half a second per search on CPU.

Use it

pip install "whensaid[url]"                 # add cuda on an NVIDIA GPU: "whensaid[cuda,url]"  (or: uv tool install "whensaid[url]")
whensaid doctor                             # checks the GPU, models, yt-dlp and the index, says how to fix problems
whensaid add ~/Videos/talks                 # register a folder of recordings
whensaid add-url https://www.youtube.com/playlist?list=...   # or a video, playlist or channel
whensaid index                              # transcribe and index what is new (models download on first use)
whensaid index --watch                      # or keep running and pick up new files every 30 s
whensaid index --subtitles                  # use a .srt/.vtt next to a recording instead of transcribing it
whensaid remove ~/Videos/old                # stop watching a folder, playlist or video and drop it from the index
whensaid status

The models (about 1.6 GB for the default set) are downloaded the first time they are needed. whensaid models pull fetches them ahead of time, and WHENSAID_OFFLINE=1 forbids any download.

Then give your agent the MCP server (stdio):

claude mcp add whensaid -- whensaid serve

Tools the agent gets (all read-only): search, get_transcript, list_sources, index_status. Indexing is only ever done by the whensaid index command, never by the agent.

How good is it

Measured on the 65-hour public-domain index above, with questions written as paraphrases (the question sets are in tests/, the scripts in scripts/):

  • The right passage is the top result for 83-97% of questions and in the top 5 for 96-100%, depending on the question set.
  • Off-topic questions: 33 of 40 return nothing, and 6 of the other 7 come back marked weak.
  • Against an agent that greps plain transcript files (whensaid export), with Claude Sonnet agents answering 36 questions: grep got 33 right and whensaid 31. That is a tie within noise for one run each, so whensaid is not more accurate than grep at this size. What it adds is speed (an agent finished a batch of 8 questions in about a minute instead of about nine), no export step, timestamp links for web videos, and a library that persists between sessions. One run per arm, questions written by someone who had read the transcripts. An earlier run looked much worse (25 of 36) only because the CLI cut each passage to 200 characters; that is fixed.
  • Not measured: word error rate against human transcripts, noisy or multi-speaker recordings beyond old radio, and any hardware other than one Linux laptop (12 cores, RTX 3050).

Limits

  • English only. Videos in other languages, and recordings the model is unsure of, are skipped (see below).
  • Linux is the tested platform; macOS runs in CI but nobody has used it daily. Windows is untried.
  • Speech only. It does not look at the picture.
  • Transcription is the slow part: about 14x realtime on the GPU it was tested on, about 6x on 12 CPU cores with the default model. WHENSAID_WHISPER_MODEL=small.en trades accuracy for speed.
  • Web videos are for videos you are allowed to download; each site's terms still apply and are your responsibility.

Details

whensaid index also drops files that were deleted or moved (unless their folder is missing or holds no media at all, which looks like an unplugged drive). With --watch, files are indexed once they have stopped changing for 15 s, and a file that failed is retried only after it changes.

Web videos use their captions when there are any (seconds, no GPU) and are transcribed from the audio otherwise. Playlists and channels are checked for new videos at most once an hour.

Things that will never work are skipped once and not asked for again: live streams, private or removed videos, videos that are not in English, and recordings whose transcript the model itself was unsure of (usually another language, or music). whensaid status lists them with the reason; whensaid index --retry-skipped tries them again, and WHENSAID_MIN_CONFIDENCE=0 turns the transcript check off.

Set WHENSAID_HOME to keep the index somewhere other than ~/.whensaid.

How it works

ffmpeg-free decode (PyAV) -> faster-whisper with VAD and word timestamps -> ~45 s overlapping chunks -> bge-small embeddings -> one SQLite file (FTS5 keyword index + vectors). Search fuses keyword and vector ranks (reciprocal rank fusion), then a small cross-encoder reranks the top 20 (WHENSAID_RERANK=0 skips it). Passages it scores as unrelated are dropped, so off-topic questions usually return nothing, and borderline hits come back marked weak (WHENSAID_MIN_RELEVANCE tunes this, off disables).

Develop

uv pip install -e ".[dev]" && pytest
WHENSAID_HOME=<index with tests/data indexed> python scripts/eval.py --verbose

Test media is public domain; see tests/data/README.md.

License: MIT

Metadata

Release files for whensaid 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for whensaid 0.1.0
File Size Uploaded Provenance Actions
whensaid-0.1.0.tar.gz 604.9 kB With provenance Details for whensaid-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for whensaid 0.1.0
File Interpreter ABI Platform Actions
whensaid-0.1.0-py3-none-any.whl Python 3 none any Details for whensaid-0.1.0-py3-none-any.whl

Total release size: 648.1 kB

Release files / whensaid-0.1.0.tar.gz

Download URL whensaid-0.1.0.tar.gz
Size 604.9 kB
Tags Source
SHA-256 checksum
How to use checksums
c4a384be41eb6057b44161304b78a6ebf21d5e1e412de9f1cc22bcc8a28b32e0
BLAKE2b-256 checksum
How to use checksums
4089a35accabf3d4285f9252d7bef86b2c5a8ce15fb02dc22082bff6d0df8b02
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Release files / whensaid-0.1.0-py3-none-any.whl

Download URL whensaid-0.1.0-py3-none-any.whl
Size 43.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
bd678aff9dd0f25b846d89db911fca15c665e5697da0f6375f5aaecdee0aaf41
BLAKE2b-256 checksum
How to use checksums
3fb43582e02f0c1af3ca98b8435547c8c993541814f0ce5cbdfec511c5a4adcc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Security for whensaid 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for whensaid-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page