mantis-agent-sdk

The Claude Agent SDK, for every model. Write to Anthropic's claude-agent-sdk API once; run the same loop against Claude, OpenAI, Gemini, Grok, or any open-weight model — Llama, Qwen, DeepSeek, GLM, Kimi, Phi, Gemma — served through Ollama, vLLM, llama.cpp, TGI, Together, Fireworks, Groq, or OpenRouter. The migration is one import:

# Before
from claude_agent_sdk import query, MantisAgentOptions, tool

# After
from mantis_agent import query, MantisAgentOptions, tool

That's the whole diff. Every canonical Claude SDK example runs verbatim — the surface is Anthropic-shaped, and underneath it speaks Anthropic Messages, OpenAI chat completions, or Ollama, whichever the model you named needs. Five provider families, one env var each:

Two ways in, one pip install: the mantis terminal — a Claude-Code-style coding agent you run in any directory — and the Python library for building your own agents on top of the same engine.

The mantis terminal

mantis is a coding agent that lives in your terminal. Point it at any directory and it reads, writes, edits, greps, and runs shell commands to actually get work done — Claude Code's feel, driving the open model you choose: a local Ollama, your own vLLM box, or a hosted endpoint.

pip install mantis-agent-sdk   # the terminal is included — no extras
mantis setup                   # detects your machine, pulls the best local coding model
mantis                         # start coding

mantis setup reads your RAM/GPU and recommends a model that actually fits — the Qwen2.5-Coder family (the strongest open coding models) plus DeepSeek-R1 for step-by-step code reasoning. Take the recommendation, pick another from the list, or mantis setup --auto to skip the prompt. No GPU needed; it'll pick something snappy for your laptop.

Want it isolated and on your PATH everywhere? uv tool install mantis-agent-sdk or pipx install mantis-agent-sdk.

            ▄▀▄▀
           ▄█▀                Mantis Code v1.5.0
        ▄██▀▀█▀               qwen2.5-7b-instruct  ·  Ollama (local)
    ▄█ ▄███▀▀                 ~/Documents/code/your-project
 ▄▄██▀▀██▀▀▀▀▀
 ▀▀ █  █▀ ▀▄
 ▄▄▀  ▄▀   ▀▄

› build me a fastapi todo app

⚒ Edit app/main.py  +12 -0
   1  + from fastapi import FastAPI
   2  + app = FastAPI()
   3  + todos: list[str] = []
       …

● Done — run it with `uvicorn app.main:app --reload`.

It's built to feel like the real thing. The input stays pinned to the bottom and never disappears — even mid-response — while the conversation scrolls above it. Replies render as Markdown with syntax-highlighted code. When the agent touches a file you get a real diff: line-numbered, syntax-highlighted, on Claude Code's exact green/red — not a wall of text. Tool calls read like ⚒ Edit app/main.py with their result tucked underneath, and a ✻ Undulating… (3s) spinner ticks while it thinks.

A few things worth knowing:

• Switch models mid-conversation — /model qwen2.5:7b, or /models to browse everything you can run locally, self-host, or reach over an API.
• Paste images and files — Ctrl+V drops a copied screenshot or file path straight into the prompt.
• Stay in control — Esc/Ctrl+C interrupts a running reply, Ctrl+D quits, shift+tab cycles the permission mode. Prefer a plain scrolling REPL? MANTIS_CLASSIC=1.

It reads configuration from the same env vars as the library:

mantis --model qwen2.5:7b
MANTIS_AGENT_BASE_URL=https://gpu-box:8000/v1 mantis --model my-model   # your own server

Want to poke at a backend without the full UI? mantis-agent is a zero-dependency diagnostics CLI — mantis-agent probe, list-models, run, chat, setup-local.

Quick start

Building your own agent? Install, set up a local model, and you're a few lines from a tool-calling loop:

pip install mantis-agent-sdk
mantis-agent setup-local         # installs Ollama if missing, pulls qwen2.5:1.5b, verifies

import asyncio
from mantis_agent import query, MantisAgentOptions, tool, AssistantMessage

@tool
async def get_weather(city: str) -> str:
    """Get the current weather for a city."""
    return f"{city}: 67°F"

async def main():
    async for msg in query(
        prompt="What's the weather in SF?",
        options=MantisAgentOptions(
            model="qwen2.5:1.5b",   # routes to local Ollama automatically
            tools=[get_weather],
            max_turns=5,
        ),
    ):
        if isinstance(msg, AssistantMessage):
            for block in msg.content:
                if hasattr(block, "text"):
                    print(block.text)

asyncio.run(main())

Same script against Together AI — change one line:

options = MantisAgentOptions(
    model="Qwen/Qwen2.5-72B-Instruct-Turbo",  # routes to Together automatically (uses $TOGETHER_API_KEY)
    tools=[get_weather],
    max_turns=5,
)

Same script against Fireworks, vLLM, llama.cpp, Groq — just change model. The backend URL is inferred from the model name shape; pass backend= explicitly to override.

Custom backend — point at any OpenAI-compatible server

Auto-routing covers the well-known providers from the model name. For everything else — your own vLLM on a private GPU box, LM Studio on a custom port, a corporate proxy, OpenRouter, Groq, an internal inference cluster — pass backend= explicitly. The URL wins over inference.

# Self-hosted vLLM on a private GPU box
options = MantisAgentOptions(
    model="Qwen/Qwen2.5-72B-Instruct",
    backend="https://gpu-box.internal:8000/v1",
    api_key=os.environ["INTERNAL_KEY"],
    tools=[get_weather],
)

# LM Studio on a non-standard port
options = MantisAgentOptions(
    model="qwen2.5:7b",
    backend="http://localhost:1234/v1",
    tools=[get_weather],
)

# Groq (blazing fast llama / mixtral)
options = MantisAgentOptions(
    model="llama-3.3-70b-versatile",
    backend="https://api.groq.com/openai/v1",
    api_key=os.environ["GROQ_API_KEY"],
)

# OpenRouter aggregator (200+ models behind one API)
options = MantisAgentOptions(
    model="anthropic/claude-3.5-sonnet",  # OpenRouter proxies even Anthropic
    backend="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

Or set it once for the whole process via env:

export MANTIS_AGENT_BASE_URL=https://gpu-box.internal:8000/v1
export MANTIS_AGENT_API_KEY=...
python my_agent.py

Precedence: explicit backend= > $MANTIS_AGENT_BASE_URL > model-name inference > Ollama default.

Models — ranked, picked by where they run

Ranked by current OSS leaderboards (Arena Elo · GPQA · SWE-bench, May 2026). Pick the highest-ranked model that fits your hardware.

CPU-laptop tier (no GPU, ≤ 8 GB RAM) — mantis-agent setup-local picks from this list:

mantis-agent setup-local           # one command — installs Ollama if missing, pulls C1, smoke tests
mantis-agent setup-local --list    # see the catalog
mantis-agent setup-local --model qwen2.5:3b

How to actually call them

Auto-routing reads the model name shape (see mantis_agent/routing.py):

For Groq, Moonshot (Kimi native), DeepSeek native, OpenRouter, Cerebras, DeepInfra, Anyscale, LM Studio, self-hosted vLLM / llama.cpp / TGI — pass backend= explicitly or set MANTIS_AGENT_BASE_URL (see Custom backend above). The pattern is the same: it's an OpenAI-compatible URL plus an API key.

Why this exists

The Claude Agent SDK is the best-designed agent runtime in the open. Streaming tool dispatch, 28-event hook system, permission rules per source, MCP across four transports, sub-agents, sessions with fork/resume, auto-compaction — none of the OSS alternatives ship the whole set. LangGraph is too heavy and skips MCP. smolagents is too small. llama-stack is tightly scoped. The Anthropic and OpenAI agent SDKs are bound to their hosted APIs.

mantis-agent-sdk is the same surface, model-agnostic underneath. You write to Anthropic's design; you run it on whatever you can serve.

Plus the OSS-specific bits the hosted SDKs don't need to think about:

• Universal tool use — Path A (native via OpenAI-compat tools[]) when supported; Path B (prompt-engineered <tool_call> XML) when not; Path C (grammar-constrained JSON) when the server can enforce it. Capability-table-driven, automatic per model.
• Universal thinking — handles inline <think> tags (R1, QwQ, Marco-o1, R1-Distill) and out-of-band thinking blocks. Zero cost when the model doesn't emit thinking.
• Backend agnosticism — same agent code, one env var or one kwarg between Ollama at localhost:11434 and Fireworks at api.fireworks.ai.
• Tracing built in — Agent(tracer=InMemoryTracer()) gives you a full span tree of every run (agent.run → agent.turn → llm.call + tool.call), with token / cost totals on the root span and tool.call spans that record input KEYS but never values. Swap in OTelTracer() to ship the same spans to Datadog / Honeycomb / Tempo / Jaeger with zero extra code. Anthropic's official SDK requires you to wire OpenTelemetry yourself; we ship it.

Workflows — named multi-agent orchestration

A workflow is a named template that fans subagents out across phases, runs in the background, and can be watched, controlled, and resumed. One mental model: workflow › phase › agent, and a workflow is also a background job — so it has an id and an observable status from the moment it starts.

/workflows list                                  # built-ins + yours, with source
/workflows run review target=mantis_agent/agent.py
▶ workflow review · run w2h94j · job #3 · 3 phases
/workflows                                       # live viewer

Workflows 1/2 · review (w2h94j) · job #3
◇ Review    2/3    ❯● ◇ correctness: reading agent.py…   ▶ 12s · ↓4.1k tok
✓ Verify    3/3      ● ✓ edge-cases: three findings…       28s · ↑9.2k tok
↑↓ select · enter/→ inspect · ←/esc back · x stop · p pause/resume · c cancel · k skip · r retry · s save

Enter drills into one agent — its prompt, activity, tokens, cost, result or error. Controls that don't apply say why instead of failing silently. Five templates ship in the box (understand · design · review · research · implement), and you add your own as Markdown:

---
name: review
description: Review a change across dimensions, then verify each finding
---

Shared briefing — prepended to every agent's prompt.

```json
{
  "inputs": [{"name": "target", "required": true}],
  "phases": [
    {"title": "Review", "mode": "parallel", "agents": [
      {"label": "bugs", "agent_type": "explore", "prompt": "Review {target} for bugs."}
    ]},
    {"title": "Verify", "mode": "pipeline", "over": "phase:Review", "stages": [
      {"agent_type": "verify", "prompt": "Try to refute:\n{item}"}
    ]}
  ]
}
```

Drop it in .mantis/workflows/<name>.md (project) or $MANTIS_AGENT_HOME/workflows/<name>.md (user) — project wins, and either overrides a built-in of the same name. Phases run parallel (barrier), sequential (each sees {prev}), or pipeline (one independent chain per item, no barrier).

Every run is persisted to $MANTIS_AGENT_HOME/workflows/runs/ — so /workflows history still lists it next week, and /workflows resume <run-id> replays every agent whose prompt is unchanged for free and re-runs only the rest. Credential-shaped inputs are redacted before anything is written.

Full guide: docs/guides/workflows.md.

Connect a provider, any way you have

Every family accepts more than one route in, and they all work:

mantis-agent auth list claude          # every method, and which is active
mantis-agent auth login claude         # browser sign-in with a subscription
mantis-agent auth use claude bedrock   # or a cloud you already pay for
mantis-agent auth check claude         # probe it, with latency and model ids

Configure several at once; one is active and switching is a click in mantis serve → Models, or one command. Cloud credentials resolve without extra SDKs, and Vertex and Bedrock never activate themselves just because a gcloud or AWS login happens to exist on the machine.

Deploy — bring your own GPU provider

Add a GPU cloud credential once, then deploy any open-weight model as an OpenAI-compatible endpoint from the dashboard, the CLI, or the terminal, and use it immediately. Six providers ship in the box: RunPod, Hugging Face Inference Endpoints, Modal, DeepInfra, Baseten, and Vast.ai.

mantis-agent deploy creds runpod --set RUNPOD_API_KEY=...      # once
mantis-agent deploy models qwen3                              # search the HF Hub, see params · VRAM · vLLM-ok
mantis-agent deploy gpus runpod --min-vram 48                 # what fits, cheapest first
mantis-agent deploy up runpod Qwen/Qwen3-32B --gpu <id>       # deploys, waits, prints the endpoint
mantis-agent deploy connect <id>                              # makes it the current model
mantis                                                        # the terminal now runs on it

Or open mantis serve → Deploy: pick a model, see which GPUs fit and what they cost per hour, click deploy, watch it come up, click Use this model.

Pre-flight reads the model's architecture, parameter count, dtype, licence, and gated flag from the Hub, estimates VRAM (weights plus KV cache), and only offers GPUs that fit. Every deployment remembers its endpoint, the name the endpoint answers to, and which env var authenticates it, so the SDK side is just:

options = MantisAgentOptions(model=dep.served_model_name, backend=dep.endpoint_url)

Scale-to-zero, idle timeouts, logs, and teardown are one call each where the provider supports them. Full guide: docs/guides/deploy.md.

Observability

from mantis_agent import Agent, InMemoryTracer, UserMessage, TextBlock

tracer = InMemoryTracer()
agent = Agent(model="claude-sonnet-4.5", tools=[...], tracer=tracer)
await agent.run([UserMessage(content=[TextBlock(text="...")])])

# Flat list of every finished span, in end-time order.
for sp in tracer.spans:
    print(sp.name, sp.duration_ms, sp.attributes)

# Or the forest, with parent/child links restored.
import json; print(json.dumps(tracer.tree(), indent=2, default=str))

# Or per-span-name aggregates + run totals (turns / tokens / cost_usd).
print(tracer.summary())

# Or ship the trace to disk for offline analysis.
tracer.write_jsonl("trace.jsonl")

To push the same spans into an existing OpenTelemetry pipeline:

from mantis_agent import OTelTracer
tracer = OTelTracer(service_name="my-agent")          # requires opentelemetry-api
agent  = Agent(model="claude-sonnet-4.5", tracer=tracer)

OTelTracer uses your already-configured TracerProvider — point it at Datadog, Honeycomb, Tempo, Jaeger, or anything else that speaks OTLP. We don't ship an exporter; we ship spans that fit your existing one. Spans carry the same attributes whether you use InMemoryTracer or OTelTracer, so dashboards built against one work against both.

Privacy by default. Tool spans carry the sorted list of input keys but never input values — agent traces routinely get shipped to third-party SaaS and showed up in screenshots and tickets, so we made the safe choice the only choice. If you need values too, build your own Tracer impl in ~30 lines.

Live example you can run with no API key:

python -m mantis_agent.examples.with_tracing

Does it actually work?

The bar, met on a fresh machine with no GPU:

pip install mantis-agent-sdk
mantis-agent setup-local
# a 10-line script: two tools, a 5-turn agent task
python my_agent.py   # works on the first try

Change one word — model= — and the same script runs against Together, Fireworks, vLLM, llama.cpp, or Groq. Anthropic's own canonical SDK examples run verbatim against DeepSeek-R1 1.5B on local Ollama. The suite runs across Python 3.9–3.14, and every release is published to PyPI from this same tree.

Roadmap

The full surface, laid out honestly — what's shipped (almost all of it) and what's still in flight.

Drop-in surface (Claude SDK parity)

• query() yielding flat-shape AssistantMessage / UserMessage / SystemMessage / ResultMessage
• MantisAgentOptions with model, backend, tools, system_prompt, max_turns, max_tokens, temperature, hooks, can_use_tool, permissions, mcp_servers, plugins, agents, max_budget_usd, setting_sources, allowed_tools, disallowed_tools, cwd, session_id, persist, stderr
• ClaudeSDKClient — streaming async context manager
• @tool decorator (Claude-shaped positional signature)
• AgentDefinition for sub-agents
• Plugin(tools=, system_prompt_addition=, hooks=) — merges at session start
• PermissionResultAllow(updated_input=...) rewriting tool args before dispatch
• PermissionResultDeny surfacing through ResultMessage.permission_denials
• HookMatcher for 28 hook events (PreToolUse, PostToolUse, SessionStart, SessionEnd, Stop, ...)
• ToolPermissionContext passed to can_use_tool
• create_sdk_mcp_server(name, version, tools=)
• WebFetch / WebSearch built-in tools (Exa-backed)
• CLIConnectionError, ClaudeSDKError
• ToolPermissionContext.signal for cancellation (anyio.Event, fired by Agent.cancel())
• setting_sources actually loading and persisting per source
• Streaming-mode client.query() with mid-stream tool dispatch

Backends

• Ollama (native API + auto-routing from tag form)
• OpenAI-compat (vLLM, Together, Fireworks, Groq, OpenRouter, Cerebras)
• llama.cpp (via --jinja)
• TGI (HuggingFace text-generation-inference)
• OpenAI native (gpt-*, o1/o3/o4)
• Gemini OpenAI-compat endpoint
• Mock provider for tests
• Auto-route from model name shape — no backend= needed
• Modal serverless adapter
• Anthropic native Messages API — claude-* routes to it automatically (API key or subscription OAuth)
• xAI Grok (grok-*) with reasoning effort mapped

Auth

• Claude: API key · subscription OAuth · Vertex · Bedrock · Azure Foundry
• OpenAI: API key · Azure OpenAI · Gemini: API key · Vertex · Grok: API key
• mantis-agent auth list|use|login|check|clear and the dashboard setup surface

Deploy (bring your own GPU)

• DeployProvider contract + store; RunPod, HF Inference Endpoints, Modal, DeepInfra, Baseten, Vast.ai
• Hub pre-flight: architecture, params, dtype, licence, gated, vLLM-ok, VRAM estimate, GPU fit
• mantis-agent deploy CLI · mantis serve Deploy page · /deploy in the terminal
• Together / Fireworks custom-model upload path; Koyeb / Northflank; Lambda raw-VM reuse of the Vast bootstrap

Tool use

• Path A: native via OpenAI-compat tools[]
• Path B: prompt-engineered <tool_call> XML (for Llama 2, Mistral 7B, older Qwens)
• Path C: grammar-constrained JSON
• Capability-table-driven path selection (30+ models)
• Parallel tool dispatch
• Tool result threading
• Streaming tool dispatch (start tool execution mid-stream, not after MessageStop)

Thinking / reasoning

• Inline <think> blocks (DeepSeek-R1, QwQ, Marco-o1, R1-distill family)
• Out-of-band thinking blocks (DeepSeek API)
• ThinkingBlock in AssistantMessage.content

MCP

• In-process MCP server via create_sdk_mcp_server
• stdio transport
• sse transport
• http transport
• Elicitation (server prompts user mid-session)
• Sampling (server calls back into the agent's model)

Sessions + state

• JSONL transcript persistence
• ~/.mantis-agent/ directory + per-session paths
• Memory entries + index
• <system-reminder> + isMeta injection
• Auto-compaction at token threshold
• Session fork
• Session resume from arbitrary checkpoint

Structured output

• response_format={"type": "json_object"} — free-form JSON mode
• response_format={"type": "json_schema", "json_schema": {...}} — schema-constrained
• Per-backend translation (OpenAI envelope / Ollama format / TGI grammar)
• Loud rejection on backends without support (anthropic_passthrough)

Budget

• Per-model pricing table
• max_usd ceiling → BudgetExceededError
• total_cost_usd on ResultMessage
• modelUsage per-model breakdown
• max_turns ceiling

Local install

• mantis-agent setup-local — installs Ollama if missing, pulls a CPU-friendly model, smoke tests
• 12-entry CPU-friendly catalog (135M → 8B params)
• Auto-install of Ollama on Linux/macOS via official script
• Windows installer wrapper
• llama.cpp setup-local alternative for users who prefer it (mantis-agent setup-local-llamacpp)

Examples (run verbatim against DeepSeek-R1 1.5B on local Ollama)

• quickstart.py
• ollama_local.py
• with_thinking.py
• tools_option.py
• mcp_calculator.py
• system_prompt.py
• fireworks_hosted.py runs against live Fireworks
• vllm_self_hosted.py runs against live vLLM (+ MANTIS_AGENT_MOCK=1 offline mode)
• multi_agent_research.py end-to-end with sub-agents

1.0 prerequisites

• Streaming tool dispatch rewrite (iter_completions / wait_one — observe results in completion order, not batched on wait_all)
• Mid-stream cancellation via ToolPermissionContext.signal
• All 18 examples verified against ≥ 3 backends
• Docs site (mkdocs-material)
• PyPI 1.0 release with semver guarantee

Drop-in compatibility — what works today

from mantis_agent import (
    # Core
    query, MantisAgentOptions, ClaudeSDKClient,

    # Messages (flat shape, matches claude_agent_sdk)
    AssistantMessage, UserMessage, SystemMessage, ResultMessage,
    TextBlock, ToolUseBlock, ToolResultBlock, ThinkingBlock,

    # Tools
    tool, Tool, ToolRegistry, create_sdk_mcp_server,

    # Permissions
    PermissionResultAllow, PermissionResultDeny, ToolPermissionContext,

    # Hooks
    HookMatcher, HookInput, HookJSONOutput, HookContext,

    # Sub-agents
    AgentDefinition,

    # Plugins
    Plugin,

    # Built-in tools
    WebFetch, WebSearch,

    # Errors
    ClaudeSDKError, CLIConnectionError,
)

Every name in that import block has a working implementation backed by tests. ClaudeSDKClient is a streaming async context manager. Plugin(tools=..., system_prompt_addition=..., hooks=...) merges into the agent at session start. PermissionResultAllow(updated_input={...}) rewrites tool args before dispatch. ResultMessage.permission_denials carries every rejected call.

License

Apache-2.0. See LICENSE.