wm-mcp 0.10.0

Run a world model locally, and let any LLM agent call it as a tool. V-JEPA 2 as an MCP server.

wm

Run a world model locally, and let any LLM agent call it as a tool.

Live and unedited: Claude calls the surprise tool; V-JEPA 2 watches the clip and reports where its predictions broke (3.0–4.9s, z≈2 — the test video's splice is at exactly 4.0s). Wait time hidden, output verbatim.

LLMs can describe what happens when a cup gets knocked off a table. A world model can watch it happen and tell you, frame by frame, where reality stopped matching its prediction. wm puts Meta's V-JEPA 2 (MIT-licensed, 0.3B params) behind an MCP server, so Claude, or any MCP client, can consult a video world model the way it consults a database.

uvx wm-mcp serve

That's the whole install. First run downloads PyTorch and ~1.2GB of model weights — minutes of downloading, not a hang.

The demo

Ask your agent:

"Where does this clip stop being predictable? /path/to/video.mp4"

The agent calls surprise(), the world model slides a window over the video, predicts each window's second half from its first, and reports the segment where its prediction failed hardest — z-scored per video, so scores mean something.

The three tools

Tool	What it does
embed_video(source, start_s?, end_s?)	Video, video slice, or still image → latent sequence, stored server-side under an opaque handle. Images become static clips — useful as goals.
predict(handle, horizon_frames, target_handle?)	Predict the trailing frames of a clip from its leading context (masked in-window prediction — horizon capped at clip length, this is not open-ended rollout). With target_handle, also returns cosine similarity between the prediction and a target embedding — "does this video end up looking like this photo?", computed server-side.
surprise(source, window, stride, fps, start_s?, end_s?, max_windows)	Streams over video of any length; per-segment "how wrong was the model" scores with timestamps in seconds + the most surprising segment. Long videos get an explicit compute budget (max_windows, default 120) and a resume_s cursor to continue — never a silent cap.
compare(handle_a, handle_b)	Cosine similarity between any two embeddings. "Are these two clips the same scene?", near-duplicate detection, frame-vs-reference matching.

Latents never cross the wire: tools exchange handle IDs, tensors stay in the server.

Setup with Claude Code

Do it in this order — step 1 downloads PyTorch and the model weights (~4GB total) so your MCP client doesn't time out waiting on a first-run download:

uvx wm-mcp warmup                     # 1. one-time: pull deps + weights (minutes)
claude mcp add wm -- uvx wm-mcp serve # 2. register the server

Then: "embed this video and tell me how surprising the ending is".

Also usable directly, no agent:

uvx wm-mcp surprise clip.mp4
uvx wm-mcp embed clip.mp4
uvx wm-mcp warmup     # pre-download weights so first real call is fast
uvx wm-mcp doctor     # environment diagnostic + timed inference — paste into bug reports
uvx wm-mcp info

HTTP API (non-MCP consumers)

The same three primitives over localhost HTTP — for scripts, notebooks, or anything that doesn't speak MCP:

uvx wm-mcp serve --http          # 127.0.0.1:8642
curl -s localhost:8642/health
curl -s -X POST localhost:8642/surprise -H 'content-type: application/json' \
     -d '{"source": "clip.mp4"}'

Endpoints: GET /health, GET /info, POST /embed, POST /predict, POST /surprise — one-to-one with the MCP tools. Binds to localhost, no auth; don't expose it to the internet.

Things to ask your agent

• "Where does this clip stop being predictable? ~/Downloads/dashcam.mp4"
• "Scan the last 10 minutes of warehouse.mp4 and tell me if anything unexpected happens." (long footage scans in budgeted chunks — the agent resumes automatically via resume_s)
• "Here's a photo of the finished assembly (goal.jpg). Does assembly-run-3.mp4 end up looking like it?"
• "Are clip_a.mp4 and clip_b.mp4 the same scene?"
• "Embed the first 30 seconds and the last 30 seconds of the match and tell me how similar they are."

Hardware & honest numbers

• Apple Silicon (MPS): ≥16GB unified memory. Measured on an M2 Pro / 16GB: surprise on a 6s clip = ~10-18s end to end (fp16, 8fps sampling); wm-mcp doctor reports model load 9.3s, ~1.7s encode + ~1.1s predict per 16-frame window. A full 64-frame embed_video is the heavyweight case — expect minutes, not seconds.
• NVIDIA (≥8GB VRAM): expected to work and to be faster — currently untested by the author. Run wm-mcp doctor and open an issue with the output; first confirmed CUDA report gets a thank-you in this README.
• CPU works; it's just slow.
• The model loads in fp16 on GPU devices (fp32 attention over 8k tokens will swap a 16GB machine — we measured it, you don't want it).
• Bigger machine? WM_MODEL=facebook/vjepa2-vitg-fpc64-256 wm-mcp serve swaps in a larger checkpoint (same V-JEPA 2 family only).

Known limitations

• Handles don't survive server restarts. Embeddings live in process memory; a new MCP session starts empty. Re-embed (it's seconds) rather than storing handles long-term.
• NVIDIA and Windows are untested. Linux CPU is exercised in CI; macOS/MPS is the developed-on path.
• predict is masked in-window prediction, capped at clip length. It is not open-ended future simulation, and we won't pretend otherwise.
• First run downloads ~4GB (PyTorch + weights). wm-mcp warmup front-loads this; don't skip it before registering the MCP server.

What this is (and isn't)

This is a probe — the smallest useful bridge between LLM agents and video world models, shipped to find out whether anyone wants that bridge. It is not a robotics stack, not a leaderboard, not a hosted service, and predict is honest about what V-JEPA 2's predictor actually does (masked latent prediction within a clip), not a marketing claim about simulating the future.

Public kill criteria, decided before launch: if by day 14 this has under 200 stars AND under 10 substantive issues/PRs from strangers AND under 3 unprompted integrations — it gets archived with a public retro. If it resonates, the next step is a protocol, not a feature list.

License

MIT. V-JEPA 2 weights are MIT-licensed by Meta FAIR.