🎵 Orchestra

Automated testing for MCP servers — declarative, fast, CI/CD-ready.

The Model Context Protocol (MCP) is becoming the standard way to connect AI models to external tools and data. Orchestra is a CLI tool that lets you write declarative YAML test suites for any MCP server — whether you're building one or integrating one into your stack.

Write tests once. Run them anywhere. Catch regressions before they reach production.

Why Orchestra?

MCP servers are proliferating fast, but tooling for testing them is still early. Orchestra fills that gap:

• No code required — define tests in readable YAML
• Works with any MCP server — STDIO (local subprocess), HTTP (Streamable HTTP), and SSE transports
• Assertions that understand MCP — JSONPath queries + first-class isError detection
• CI/CD native — exit codes, quiet mode, JSON reports

Install

Orchestra is closed-source. The source is proprietary (see LICENSE), but the CLI is distributed as a wheel on PyPI so it can be installed in CI without the desktop app:

uv tool install orchestra-mcp     # or: pipx install orchestra-mcp
orchestra --help

Installing it as a tool rather than into the project environment keeps Orchestra's dependencies from colliding with the MCP server under test.

The desktop app bundles the same CLI; Get Started → Install the orchestra command-line tool links it onto your PATH.

For development in this repo:

uv sync --extra server --extra dev
uv run orchestra --help

Quickstart

New to Orchestra? Use the interactive builder:

orchestra new schemas/my_test.yaml

The wizard guides you through transport, auth, and your first test step. Time to first passing test: ~3 minutes.

Or write YAML directly:

# schemas/memory_test.yaml
version: 1
name: "Memory Server Test"

server:
  transport: "stdio"
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-memory"]

steps:
  - id: create_entity
    type: tool_call
    tool: "create_entities"
    input:
      entities:
        - name: "TestUser"
          entityType: "person"
          observations: ["Loves testing"]
    delay_ms: 1000

  - id: verify_created
    type: assert
    from: "create_entity"
    check:
      op: "jsonpath_exists"
      path: "$.content[0].text"

orchestra run schemas/memory_test.yaml

============================================================
 Running: Memory Server Test
 Server: stdio | Steps: 2
============================================================

📡 Connecting to MCP server...
 ✅ Connected to memory-server v0.6.3

▶ Step: create_entity (tool_call)
  ✅ Success

▶ Step: verify_created (assert)
  Check: jsonpath_exists at $.content[0].text
  ✅ Passed

═══════════════════════════════════════════════════════════
 Status: ✅ PASSED  |  Duration: 1247ms  |  2/2 steps
═══════════════════════════════════════════════════════════
📁 Report saved: reports/abc123.json

Features

CLI Reference

orchestra inspect <name|url|yaml>    # Discover tools, schemas, resources, prompts
orchestra call <name|url> <tool>     # Call one tool and print the result
orchestra run <collection|dir>       # Execute a collection, or every one in a directory
orchestra watch <name> --path src    # Rebuild + re-inspect on change, diff the surface
orchestra validate <collection.yaml> # Validate schema without connecting
orchestra new [output_file]          # Interactive collection builder
orchestra mcp                        # Serve Orchestra itself over MCP, for AI agents
orchestra playground                 # Run the built-in demo MCP server
orchestra info                       # Version and environment info

orchestra servers list                       # Named connections (global + workspace)
orchestra servers add <name> --url <url>     # Save a connection
orchestra servers add <name> --command <cmd> --args "..."
orchestra servers detect <dir>               # Find a running HTTP server from its project folder
orchestra servers import                     # Import from Claude Desktop / Cursor / Windsurf
orchestra servers remove <name>

orchestra auth login <mcp-url>       # OAuth sign-in (saves tokens for run/inspect)
orchestra auth logout <url|profile>  # Remove stored OAuth session

Arguments to call follow the httpie convention: -i key=value is a string, -i key:=value is raw JSON, so -i count:=3 sends a number and -i count=3 sends "3".

Run options

orchestra run tests/mcp --output junit --report-dir reports  # JUnit XML + summary.json for CI
orchestra run tests/mcp --environment staging               # Named env from orchestra.env.yaml
orchestra run tests/mcp --env API_KEY=$SECRET               # Override a variable (repeatable)
orchestra run tests/mcp --fail-on breaking                  # Only fail on breaking contract changes
orchestra run tests/mcp --update-baselines                  # Accept snapshot changes, jest-style
orchestra run tests/mcp --show-responses                    # Print full JSON responses
orchestra run tests/mcp --quiet                             # Errors only
orchestra run tests/mcp --no-report                         # Skip saving report files

Transports

Local (STDIO) — runs the server as a subprocess:

server:
  transport: "stdio"
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-memory"]
  env:
    API_KEY: "{{env.MY_API_KEY}}"

Remote (HTTP) — Streamable HTTP:

server:
  transport: "http"
  url: "https://mcp.deepwiki.com/mcp"

SSE:

server:
  transport: "sse"
  url: "http://localhost:3001"

Authentication

OAuth (MCP HTTP servers with authorization metadata) — run once, then use type: oauth in YAML:

orchestra auth login https://mcp.example.com
# Optional: --profile myapp   # if set, add oauth_profile: "myapp" in YAML
# Optional: --scopes "scope1 scope2"

Tokens are stored under ~/.config/orchestra/oauth_sessions.json (POSIX mode 600). This file-backed store is used by default for both CLI and the desktop GUI so OAuth works consistently across processes. Set ORCHESTRA_OAUTH_PROVIDER=chuk only if you prefer chuk's platform credential store instead (requires the optional extra: pip install "orchestra-mcp[chuk]"). If dynamic client registration fails, set ORCHESTRA_OAUTH_CLIENT_ID (and ORCHESTRA_OAUTH_CLIENT_SECRET if required).

server:
  transport: "http"
  url: "https://mcp.example.com"
  auth:
    type: "oauth"
    # oauth_profile: "myapp"  # only if you used --profile on login

# Bearer token
auth:
  type: "bearer"
  token: "{{env.API_TOKEN}}"

# API key
auth:
  type: "api_key"
  key: "{{env.API_KEY}}"

# Basic auth
auth:
  type: "basic"
  username: "{{env.USERNAME}}"
  password: "{{env.PASSWORD}}"

Assertions

- id: check_result
  type: assert
  from: "my_step"
  check:
    op: "jsonpath_contains"
    path: "$.content[0].text"
    value: "expected string"

jsonpath_gt vs jsonpath_len_gte — the comparison operators compare the value found at the path; the len_ operators compare the length of the array or string found there. $.results with three items satisfies jsonpath_len_gte: 3; $.total holding 42 satisfies jsonpath_gt: 41. Pointing a comparison at a non-number (a string, an object, null) fails the step with "Cannot compare … numerically" rather than guessing, and booleans are never treated as 1/0.

- id: enough_results
  type: assert
  from: "search"
  check:
    op: "jsonpath_gt"
    path: "$.structuredContent.total"
    value: 0

Pattern matching — jsonpath_matches uses re.search, so the pattern is found anywhere in the string, consistent with jsonpath_contains being a substring check. Anchor it with ^...$ when you want a full match. An invalid regex is rejected by orchestra validate before the run starts.

- id: looks_like_an_id
  type: assert
  from: "create"
  check:
    op: "jsonpath_matches"
    path: "$.structuredContent.id"
    value: "^user_[0-9a-f]{8}$"

Negation — add not: true to any check to invert it, instead of a separate jsonpath_not_* operator for every operator:

- id: no_stack_traces
  type: assert
  from: "search"
  check:
    op: "jsonpath_matches"
    path: "$.content[0].text"
    value: "Traceback"
    not: true

not: true flips pass and fail. It does not rescue a check that could not be evaluated at all — an unparseable JSONPath or a string under a numeric comparison stays an error, since reporting "not greater than: pass" for a value that was never a number would hide the mistake behind a green check.

MCP error detection — Orchestra distinguishes between JSON-RPC transport errors and tool-level errors (isError: true in the response body), so you can assert on both:

- id: expect_failure
  type: assert
  from: "bad_call"
  check:
    op: "is_error"   # Passes if the tool itself returned an error

Reports

Every run saves a structured JSON report:

{
  "run_id": "abc123-def456",
  "collection_name": "My Test",
  "status": "passed",
  "duration_ms": 1234,
  "server": { "name": "memory-server", "version": "0.6.3" },
  "steps": [
    {
      "id": "create_entity",
      "type": "tool_call",
      "status": "success",
      "duration_ms": 150
    },
    {
      "id": "verify_created",
      "type": "assert",
      "status": "passed",
      "duration_ms": 5
    }
  ]
}

Reports are saved to reports/ by default. Use --report-dir to customize.

Running a directory of collections also writes reports/summary.json — one entry per collection with its source path, status, step counts, and the severity counts of any contract-snapshot changes. A single exit code cannot say which of N servers moved; that file can.

Examples

version: 1
name: "DeepWiki Test"

server:
  transport: "http"
  url: "https://mcp.deepwiki.com/mcp"

steps:
  - id: ask_about_react
    type: tool_call
    tool: "ask_question"
    input:
      repoName: "facebook/react"
      question: "What are React hooks?"
    delay_ms: 2000

  - id: check_answer
    type: assert
    from: "ask_about_react"
    check:
      op: "jsonpath_contains"
      path: "$.content[0].text"
      value: "hook"

version: 1
name: "Brave Search Test"

env:
  BRAVE_API_KEY: "your-api-key"

server:
  transport: "stdio"
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-brave-search"]
  env:
    BRAVE_API_KEY: "{{env.BRAVE_API_KEY}}"

steps:
  - id: search_python
    type: tool_call
    tool: "brave_web_search"
    input:
      query: "Python programming"
      count: 5
    delay_ms: 2000

  - id: check_results
    type: assert
    from: "search_python"
    check:
      op: "jsonpath_len_gte"
      path: "$.content"
      value: 1

CI/CD Integration

Orchestra exits with code 0 on pass and 1 on failure — drop it into any pipeline:

# GitHub Actions
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v5

# Install the server under test FROM THIS BRANCH, not from a registry.
- run: uv sync --frozen          # or: npm ci && npm run build

# uvx keeps Orchestra's dependencies isolated from the server's.
- run: uvx --from orchestra-mcp orchestra run tests/mcp --output junit --report-dir reports

The npx trap. Every MCP server README documents setup as npx -y @scope/server, because that is the Claude Desktop config. Do not copy that into CI: it installs the published package, so your gate tests the last release instead of the pull request — a green check that proves nothing. Build from source and run the local artifact.

Prefer stdio for the server under test: a subprocess over pipes binds no port, so there is nothing to background and no port to wait on.

Support

Orchestra is proprietary software (see LICENSE); the source is not public. Bug reports, feature requests, and questions are welcome at admin@ahaan.world.

Docs: ahaan.world/orchestra/docs

License

Proprietary — all rights reserved. See LICENSE. Built on the Model Context Protocol by Anthropic.