talkthrough-mcp 0.1.0

Local-first MCP server that turns narrated screen recordings into agent-ready structured data: timestamped transcript, scene keyframes, OCR text, and wall-clock anchoring.

talkthrough-mcp

Quickstart · Tools · FAQ · Troubleshooting · Changelog · Contributing

Feedback ingestion for AI agents. Record your screen and talk; your agent does the rest — files the bugs, writes the spec, builds the backlog.

talkthrough-mcp is a local-first MCP server that turns a narrated screen recording (or any video/audio file) into agent-ready structured data: timestamped transcript segments, scene-change keyframes, OCR'd on-screen text, and wall-clock anchoring. Everything is served through lazy retrieval tools, so a 30-minute recording never floods the model context — the agent pulls exactly the transcript slice, moment bundle, or frame it needs.

There is no LLM inside the server and no cloud anywhere in the path: ffmpeg, faster-whisper, and RapidOCR run on your machine, and the calling agent brings the intelligence. What makes it different from screen-recorder SaaS and video-analyzer MCPs: it works on arbitrary local files, it ships the agent workflows (server prompts + example agents), and it anchors every timestamp to wall-clock time — so "the moment I said the checkout hung" maps straight to the right window of your server logs.

Quickstart

One command, no system dependencies: ffmpeg falls back to a bundled build, OCR is pip-only, and whisper models download themselves on first use.

Claude Code

claude mcp add -s user talkthrough -- uvx talkthrough-mcp

Or install the full plugin (server + the five workflow commands + the triage agent + an agent skill):

/plugin marketplace add korovin-aa97/talkthrough-mcp
/plugin install talkthrough@talkthrough

Every other MCP client

Claude Desktop

claude_desktop_config.json:

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}

More: integrations/claude-desktop/

Cursor

~/.cursor/mcp.json (or project .cursor/mcp.json):

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}

More: integrations/cursor/

OpenAI Codex CLI

~/.codex/config.toml (or project-scoped .codex/config.toml in trusted projects):

[mcp_servers.talkthrough]
command = "uvx"
args = ["talkthrough-mcp"]

More: integrations/codex/

Gemini CLI

~/.gemini/settings.json:

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}

More: integrations/gemini-cli/

Cline / Roo Code

cline_mcp_settings.json (via MCP Servers UI):

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}

More: integrations/cline/

OpenClaw

~/.openclaw/openclaw.json:

{
  "mcp": {
    "servers": {
      "talkthrough": {
        "command": "uvx",
        "args": [
          "talkthrough-mcp"
        ]
      }
    }
  }
}

More: integrations/openclaw/

OpenCode

opencode.json (project) or ~/.config/opencode/opencode.json:

{
  "mcp": {
    "talkthrough": {
      "type": "local",
      "command": [
        "uvx",
        "talkthrough-mcp"
      ],
      "enabled": true
    }
  }
}

More: integrations/opencode/

Goose

~/.config/goose/config.yaml:

extensions:
  talkthrough:
    enabled: true
    type: stdio
    cmd: uvx
    args: ["talkthrough-mcp"]

More: integrations/goose/

GitHub Copilot CLI

~/.copilot/mcp-config.json:

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}

More: integrations/copilot-cli/

Windsurf

~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "talkthrough": {
      "command": "uvx",
      "args": [
        "talkthrough-mcp"
      ]
    }
  }
}

More: integrations/windsurf/

Zed

settings.json (Zed):

{
  "context_servers": {
    "talkthrough": {
      "source": "custom",
      "command": {
        "path": "uvx",
        "args": [
          "talkthrough-mcp"
        ]
      }
    }
  }
}

More: integrations/zed/

Any other MCP stdio client uses the same server command: uvx talkthrough-mcp. Per-engine folders with exactly these snippets plus verification steps live in integrations/; agents can self-install via llms-install.md.

Local checkout (development)

git clone https://github.com/korovin-aa97/talkthrough-mcp
claude mcp add talkthrough -- uv run --directory /path/to/talkthrough-mcp talkthrough-mcp

Then, in your agent:

Process ~/Desktop/recording.mov and triage it — or just invoke the triage-recording server prompt.

Tools

Tool	What it does
process_media(path, recorded_at?, vocabulary?, language?, model?, force?)	Ingest a video/audio file: local STT, keyframes, OCR, wall-clock. Returns a compact summary. Idempotent by content hash — re-calls are instant.
get_transcript(job_id, start_ms?, end_ms?, format?)	Paginated transcript as segments, text, or srt; truncation returns next_start_ms.
get_frames(job_id, at_ms? | start_ms?+end_ms?, max_frames?, include_duplicates?)	Keyframe images nearest a timestamp or evenly thinned across a range (unique frames by default, max 6/call).
get_moment(job_id, start_ms, end_ms)	The "one remark" bundle: transcript slice + up to 3 frames + their OCR text + wall-clock range.
search(job_id, query)	Substring search over the transcript AND on-screen OCR text; hits carry t_ms/t_wall and frame refs.
extract_frame(job_id, at_ms, crop?)	Exact-timestamp full-resolution re-extract from the source video (optional crop) when keyframes miss the instant.
list_jobs()	Recent processed recordings with durations, wall-clock starts, and counts.

Every tool description ships 10+ usage examples, so agents pick the right tool without extra prompting.

Server prompts (slash commands in MCP clients)

Prompt	Workflow
triage-recording	Narrated screencast → precise findings JSON (bug/feature/question routing, frame evidence)
spec-from-workshop	Recorded workshop → structured spec with quoted decisions and open questions
backlog-from-demo	Product demo → prioritized backlog with timestamped evidence
meeting-actions	Meeting audio → action items, decisions, open questions
correlate-with-logs	Recording remarks ↔ system logs via wall-clock windows

The same prompts live as plain files in examples/prompts/ if your client doesn't surface MCP prompts. The findings contract used by triage-recording is examples/output-contract.schema.json.

Works as a skill too (no MCP required)

The same workflow ships as a cross-engine Agent Skill at .agents/skills/talkthrough/ — Claude Code, Codex CLI ($talkthrough), Cursor, Copilot, Gemini CLI, Goose and other SKILL.md-compatible tools read it. Agents without MCP wiring can drive the CLI directly: talkthrough-mcp process recording.mov --json prints the same summary the MCP tool returns, and the job store is shared either way.

Wall-clock anchoring

Every timestamped result carries both t_ms (video-relative) and t_wall (ISO 8601 real time) once the recording start is known. Resolution ladder:

1. recorded_at parameter (agent/user override) → confidence exact
2. QuickTime com.apple.quicktime.creationdate tag, carries the local timezone (QuickTime Player recordings; ⌘⇧5 wrote it before macOS 26) → high
3. Container creation_time tag (UTC) → medium — macOS 26+ ⌘⇧5/ReplayKit screen recordings land here (no creationdate tag anymore); pass recorded_at= when local-tz t_wall matters
4. File mtime minus duration (recorders finalize files at recording END) → low
5. Nothing → tools still work with relative t_ms only

Why it matters: "the upload spinner froze here" becomes a ±30 s grep window in your server logs.

Privacy

Everything runs locally: your recordings never leave your machine, speech is transcribed by a local whisper model, OCR is local ONNX inference, and there is no telemetry. The only network access is one-time tool/model downloads (ffmpeg build, whisper model, OCR models).

Languages

Narration in any of Whisper's ~99 languages works: the language is auto-detected per recording, and the summary reports both language and language_probability so agents can tell a confident detection from a shaky one (silence or music at the start can fool the detector — pin it with language="ru" and force=true when that happens).

Pick the model for your languages — per call (model= parameter, agents do this themselves when a transcript comes back garbled) or as the server default (TALKTHROUGH_WHISPER_MODEL):

Model	Size	Best for
small (default)	464 MB	English and major-language narration on CPU
large-v3-turbo	~1.5 GB	recommended for non-English — near-large quality at near-small speed
medium	~1.5 GB	conservative alternative to turbo
tiny / base	75–145 MB	quick drafts, CI
*.en variants	—	English-only, slightly faster/better for EN

Tips that work in every language: pass product names via vocabulary="Term1, Term2" (biases the decoder so jargon survives), and note that the workflow prompts instruct agents to write digests in the narrator's language while keeping quotes verbatim — the server never translates (exact quotes are evidence; translation is the agent's job).

On-screen text (OCR) defaults to RapidOCR's Latin + Chinese models. For other scripts set TALKTHROUGH_OCR_LANG to your language — ru/uk (→ the eslav pack), ja, ko, ar, hi, el, th, or any RapidOCR pack name like cyrillic — and reprocess with force=true; the matching recognition model downloads once. Spoken-language support is unaffected either way.

Configuration

Env var	Default	Meaning
TALKTHROUGH_WHISPER_MODEL	small	default whisper model (tiny/base/small/medium/large-v3/large-v3-turbo); the model tool param overrides per call
TALKTHROUGH_OCR	on	set off to skip OCR
TALKTHROUGH_OCR_LANG	Latin+Chinese	recognition script for on-screen text: a language code (ru, ja, ko, ar, hi, …) or a RapidOCR pack name (eslav, cyrillic, latin, …); the model downloads once
TALKTHROUGH_OCR_PARAMS	—	advanced: JSON object of raw RapidOCR params merged over the derived ones, e.g. {"Rec.lang_type": "cyrillic"}
TALKTHROUGH_MAX_SECONDS	7200	max media duration
TALKTHROUGH_MAX_FRAMES	600	keyframe cap per job
TALKTHROUGH_HOME	~/.talkthrough	job store root

CLI

The pipeline is also a CLI — useful for pre-processing long recordings outside an agent session (the store is content-addressed, so the agent then queries the same job instantly):

talkthrough-mcp process ~/Videos/long-session.mov   # prints the summary
talkthrough-mcp process demo.mov --json             # machine-readable
talkthrough-mcp gc --keep-days 30                   # clean the job store
talkthrough-mcp serve                               # stdio MCP server (default)

First run notes: missing system ffmpeg triggers a one-time static-ffmpeg download; the first transcription downloads the whisper model (~460 MB for small); both are cached. After that, expect roughly 3× faster than real time on an Apple-Silicon CPU with the default model, OCR included (a 2-minute clip processes in ~40 s) — and instant re-runs on the same file. Progress streams as MCP progress notifications, and the CLI prints stage lines. More: docs/TROUBLESHOOTING.md.

Windows (best-effort)

CI runs lint, the unit suite, and a full CLI smoke on windows-latest (static-ffmpeg Windows build, whisper tiny transcription, OCR, and the instant idempotent re-run). Notes: the per-job lock is POSIX fcntl and degrades to a no-op on Windows — fine for a single-user machine; quote paths with spaces (uv run talkthrough-mcp process "C:\Videos\Screen Recording.mp4"). Windows is not a release gate — if something breaks, please open an issue.

Supported inputs

Video: .mov .mp4 .webm .mkv — audio-only: .m4a .mp3 .wav .ogg .flac (transcript tools only; frame tools explain why they're unavailable). Local files only.

Limitations

Honest edges, so you can decide fast:

• One speaker stream. No diarization yet — "who said it" isn't tracked (#4).
• Local files only. No URL/YouTube ingestion (#5) — download first.
• Keyframes + transcript, not motion analysis. A glitch between scene changes can be invisible in the frame set; extract_frame re-checks any instant, but frame-by-frame motion reasoning is your multimodal model's job.
• STT quality tracks the model you pick. The default small favors speed; non-English narration wants model="large-v3-turbo" (see Languages).
• OCR reads crisp UI text well; tiny or low-contrast print is best-effort.
• Wall-clock confidence depends on recorder metadata — worst case pass recorded_at= (see the ladder above).
• Windows is best-effort (see above).

How it compares

	talkthrough	cloud recorder SaaS	meeting notetakers	typical video-analyzer MCPs
Runs fully locally	✅	❌	❌	varies
Any local video/audio file	✅	browser/app captures	meetings only	✅
Wall-clock anchoring (log correlation)	✅	❌	❌	❌
Ships agent workflows (prompts, skill, findings contract)	✅	❌	❌	❌
OCR of on-screen text, searchable	✅	some	❌	rare

FAQ

Why not just upload the video to a multimodal model (e.g. Gemini)? For a short, non-sensitive clip — do that. The trade-offs appear with length and sensitivity: an hour of screen recording costs on the order of a million tokens per question, the file leaves your machine, and you still can't map a remark to 14:32:07 UTC to grep your server logs. talkthrough indexes once, locally, then answers any number of follow-ups from the index.

Why not screenpipe? Different job. screenpipe is an always-on recorder of your machine going forward (commercial license). It can't open the .mov a teammate or customer just sent you. talkthrough analyzes any file it's handed — the two compose fine.

There are agent skills that "watch" videos. Why a server with an index? Watch-style skills push a budgeted frame dump into the context window (and go sparse on long videos), often call cloud STT for the audio, and keep nothing. talkthrough builds a persistent local index — transcript + OCR, full-text searchable — retrieves exact frames lazily, anchors everything to wall-clock time, and answers the next question without reprocessing.

I use Jam for bug reports — do I need this? Keep Jam for browser bugs: console+network captured at record time is great evidence. talkthrough covers what a browser extension can't — desktop apps, mobile screencasts, ops incidents, meetings, any file — with no account, and correlates with server-side logs via wall-clock time.

Can't I just script ffmpeg + whisper myself? Yes — that's exactly this pipeline. What you'd be rebuilding: scene-change detection with perceptual dedup, OCR, transcript+OCR search, the wall-clock ladder, MCP tools with embedded usage examples, five workflow prompts, and a findings contract. One uvx command instead of an afternoon of glue.

Is it really local? What leaves my machine? Nothing at runtime. The network is used only for one-time downloads (ffmpeg build, whisper/OCR models). No telemetry. See Privacy — and SECURITY.md treats a violation of this promise as a vulnerability.

For agents & tooling

Machine-readable entry points, so AI agents can install and use this server without a human reading docs:

• llms-install.md — step-by-step install instructions for agents
• llms.txt — index of the documentation
• .agents/skills/talkthrough/SKILL.md — an Agent Skill teaching the tool workflow; discovered automatically inside a checkout by Codex CLI ($talkthrough) and readable by Claude Code, Cursor, Copilot, Gemini CLI and other SKILL.md-compatible tools
• AGENTS.md — instructions for coding agents contributing to this repo
• server.json — MCP registry manifest
• integrations/ — per-engine adapters, all generated from one source of truth and drift-tested (incl. the Claude Code plugin under integrations/claude-code/)

Roadmap (not in v1)

URL/YouTube ingestion · speaker diarization · cloud STT · embeddings/semantic search · hosted/remote mode · .mcpb bundle · whisper.cpp backend

License

MIT