wm-mcp 0.3.0

Run a world model locally, and let any LLM agent call it as a tool. V-JEPA 2 as an MCP server.

wm

Run a world model locally, and let any LLM agent call it as a tool.

Live and unedited: Claude calls the surprise tool; V-JEPA 2 watches the clip and reports where its predictions broke (z=3.1 at frames 56–71 — the exact splice point of the test video). Wait time hidden, output verbatim.

LLMs can describe what happens when a cup gets knocked off a table. A world model can watch it happen and tell you, frame by frame, where reality stopped matching its prediction. wm puts Meta's V-JEPA 2 (MIT-licensed, 0.3B params) behind an MCP server, so Claude, or any MCP client, can consult a video world model the way it consults a database.

uvx wm-mcp serve

That's the whole install. First run downloads PyTorch and ~1.2GB of model weights — minutes of downloading, not a hang.

The demo

Ask your agent:

"Where does this clip stop being predictable? /path/to/video.mp4"

The agent calls surprise(), the world model slides a window over the video, predicts each window's second half from its first, and reports the segment where its prediction failed hardest — z-scored per video, so scores mean something.

The three tools

Tool	What it does
embed_video(source)	Video → latent sequence, stored server-side under an opaque handle.
predict(handle, horizon_frames, target_handle?)	Predict the trailing frames of a clip from its leading context (masked in-window prediction — horizon capped at clip length, this is not open-ended rollout). With target_handle, also returns cosine similarity between the prediction and a target embedding — goal-similarity scoring, computed server-side.
surprise(source, window, stride)	Per-segment "how wrong was the model" scores + the most surprising segment. Anomaly spotting, event detection, "did anything unexpected happen."

Latents never cross the wire: tools exchange handle IDs, tensors stay in the server.

Setup with Claude Code

claude mcp add wm -- uvx wm-mcp serve

Then: "embed this video and tell me how surprising the ending is".

Also usable directly, no agent:

uvx wm-mcp surprise clip.mp4
uvx wm-mcp embed clip.mp4
uvx wm-mcp warmup     # pre-download weights so first real call is fast
uvx wm-mcp info

HTTP API (non-MCP consumers)

The same three primitives over localhost HTTP — for scripts, notebooks, or anything that doesn't speak MCP:

uvx wm-mcp serve --http          # 127.0.0.1:8642
curl -s localhost:8642/health
curl -s -X POST localhost:8642/surprise -H 'content-type: application/json' \
     -d '{"source": "clip.mp4"}'

Endpoints: GET /health, GET /info, POST /embed, POST /predict, POST /surprise — one-to-one with the MCP tools. Binds to localhost, no auth; don't expose it to the internet.

Hardware & honest numbers

• Apple Silicon (MPS): ≥16GB unified memory. Measured on an M2 Pro / 16GB: surprise on a 6s clip = ~18s end to end (11 windows, fp16). A full 64-frame embed_video is the heavyweight case — expect minutes, not seconds.
• NVIDIA: ≥8GB VRAM, substantially faster.
• CPU works; it's just slow.
• The model loads in fp16 on GPU devices (fp32 attention over 8k tokens will swap a 16GB machine — we measured it, you don't want it).

What this is (and isn't)

This is a probe — the smallest useful bridge between LLM agents and video world models, shipped to find out whether anyone wants that bridge. It is not a robotics stack, not a leaderboard, not a hosted service, and predict is honest about what V-JEPA 2's predictor actually does (masked latent prediction within a clip), not a marketing claim about simulating the future.

Public kill criteria, decided before launch: if by day 14 this has under 200 stars AND under 10 substantive issues/PRs from strangers AND under 3 unprompted integrations — it gets archived with a public retro. If it resonates, the next step is a protocol, not a feature list.

License

MIT. V-JEPA 2 weights are MIT-licensed by Meta FAIR.