youty-mcp

Local MCP server that exposes the Youty vault's vector index to any MCP-compatible AI (Claude Desktop, Claude Code, Cursor).

What it does

Seven tools, hybrid dense + BM25 retrieval over your captured YouTube / Instagram / TikTok videos, plus joint text → frame retrieval via Google's SigLIP-Base-Patch16-224 (Apache-2.0). Queries land in ~300 ms for text, ~32 ms warm for frames on Apple Silicon.

The loop: search finds the relevant moments → get_transcript pulls the words into context → view_frames loads the matching frames into the model's vision. search / search_frames also return raw frame paths, but only Claude Code can open a path itself — view_frames returns the images inline, so the visual half of the loop works in Claude Desktop, Cursor, and Claude Code alike.

On-screen text. search covers both what a video said and what it showed: each result's chunk.type is body/description/header (spoken + metadata) or frame_text — text recognized on-screen via on-device OCR (slides, code, terminal output, labels). A frame_text hit means the answer was visible in the video; pair it with view_frames to see that moment.

Install

cd youty-mcp
uv sync                       # creates .venv, installs deps

Dependencies: mcp, cryptography, sqlite-vec, httpx, numpy, tokenizers, huggingface-hub, sentencepiece, protobuf, and coremltools (macOS only). Python ≥ 3.11. No PyTorch or Transformers: existing tokenizer engines prepare queries directly, and all inference runs through Core ML. Tokenizer assets use fixed revisions to preserve compatibility with the existing vault index.

Text + frame search: 100% on-device — no key, zero config

The server embeds each query on-device with the same Core ML models the index was built with — Google's EmbeddingGemma (text) and the SigLIP-Base text tower (frames) — so query and document vectors share one space. Inference is CPU-only via coremltools to match the int8-quantized indexer. No key, no provider option, no cloud call of any kind.

The models are not a ~1.6 GB HuggingFace download. They come from Youty's own release asset (youty-models-<ver>.tar.gz, a few hundred MB of Core ML), fetched once and verified by SHA-256, then cached under:

~/.cache/youty/coreml-models/<version>/

One-time per machine; every query after that is fully offline. Hot-path embed is ~300 ms for text and ~32 ms for frames on Apple Silicon. (Set YOUTY_COREML_MODELS_DIR to point at a local .mlpackage tree in dev/CI.)

Wiring it into your AI client

One command wires every MCP client on this Mac:

uvx youty-mcp@latest install

It detects Claude Code, Claude Desktop, Cursor, Codex, Gemini CLI, Windsurf, Continue, Cline, Antigravity, and Grok CLI, then merges a youty entry into each client's config — preserving your other settings, idempotent (safe to re-run), and reversible with uvx youty-mcp@latest uninstall. The entry points at the absolute path of uvx so GUI apps (which launch with a stripped-down PATH) reliably find it. Useful variants:

uvx youty-mcp@latest install --list      # every supported client + its config path
uvx youty-mcp@latest install cursor      # wire just one client
uvx youty-mcp@latest uninstall           # remove Youty from all detected clients

Why a command and not a button in the Youty app? The Mac app is sandboxed, so it can't edit other apps' config files or run their CLIs. This installer ships in the (non-sandboxed) youty-mcp package, which can. Bare uvx youty-mcp@latest still runs the MCP server itself — that's what the clients launch.

Editors whose config allows comments (Zed → a context_servers entry in ~/.config/zed/settings.json, VS Code → an mcp.servers entry) aren't auto-wired — rewriting them could strip your comments — so add Youty there by hand using the command/args shown below.

Prefer to wire a client by hand? The exact per-client config follows.

Claude Desktop wiring

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "youty": {
      "command": "uvx",
      "args": ["youty-mcp@latest"]
    }
  }
}

The @latest pin means uvx fetches the newest published youty-mcp on each launch — so a normal restart always loads the current server with no manual uv tool upgrade. (The Mac app auto-updates via Sparkle and the CLI via Homebrew, so the whole stack stays in step on its own.)

Restart Claude Desktop. Then ask: "What are best practices on creating AI influencers, and what tools should I use? Use my Youty vault."

Claude Code wiring

claude mcp add youty -- uvx youty-mcp@latest

Tests

uv run pytest -q
YOUTY_TOKENIZER_PARITY=1 uv run pytest -q tests/test_query_tokenizers.py
YOUTY_COREML_SMOKE=1 uv run pytest -q tests/test_coreml_inputs.py
uv run python tests/smoke_live.py    # one-shot live on-device search smoke

The Core ML smoke check requires macOS and the model assets. It exercises both query encoders repeatedly and waits for deferred native cleanup, catching crashes that token-only tests cannot detect. Set YOUTY_COREML_MODELS_DIR to a local model directory when testing without the normal download cache.

Index location

Default: the Mac app's sandboxed index at ~/Library/Containers/dev.leget.youty/Data/Library/Application Support/Youty/index.db, falling back to ~/Library/Application Support/Youty/index.db if that isn't present. Override either with YOUTY_INDEX_DB=/abs/path.

The Mac app writes here when it saves a video (background, non-blocking). The MCP server reads here and promotes data to sqlite-vec and FTS5 virtual tables at startup.

The index is rebuildable from the vault's video.md files alone — losing it is recoverable, never catastrophic. Use the Mac app's Settings window → "Re-index entire vault", or run headless:

"/path/to/youty.app/Contents/MacOS/youty" --reindex "/path/to/vault"
"/path/to/youty.app/Contents/MacOS/youty" --index-frames "/path/to/vault"

Troubleshooting

• search returns 0 results — the index is empty. Save a video from the Mac app (indexer enabled in Settings) or run --reindex on an existing vault. No key needed — text indexing is on-device by default.
• First search / search_frames is slow — the Core ML models asset downloads once (youty-models-<ver>.tar.gz, a few hundred MB, SHA-verified) into ~/.cache/youty/coreml-models/ and the encoders load lazily. Subsequent queries are ~300 ms (text) / ~32 ms (frames).
• Legacy bundles with 4-digit-second JPEG names (0717.jpg) are silently skipped by frame indexing. The current contract is 8-digit milliseconds (00718000.jpg). Re-saving the video regenerates frames in the new format.
• Vault location unknown error from get_transcript — the indexer records the vault path; if you've changed it, run --reindex once against the new path so index_meta.vault_root updates.