🎵 Orchestra

Automated testing for MCP servers — declarative, fast, CI/CD-ready.

The Model Context Protocol (MCP) is becoming the standard way to connect AI models to external tools and data. Orchestra is a CLI tool that lets you write declarative YAML test suites for any MCP server — whether you're building one or integrating one into your stack.

Write tests once. Run them anywhere. Catch regressions before they reach production.

Why Orchestra?

MCP servers are proliferating fast, but tooling for testing them is still early. Orchestra fills that gap:

• No code required — define tests in readable YAML
• Works with any MCP server — STDIO (local subprocess), HTTP (Streamable HTTP), and SSE transports
• Assertions that understand MCP — JSONPath queries + first-class isError detection
• CI/CD native — exit codes, quiet mode, JSON reports

Install

Orchestra is closed-source. The source is proprietary (see LICENSE), but the CLI is distributed as a wheel on PyPI so it can be installed in CI without the desktop app:

uv tool install orchestra-mcp     # or: pipx install orchestra-mcp
orchestra --help

Installing it as a tool rather than into the project environment keeps Orchestra's dependencies from colliding with the MCP server under test.

The desktop app bundles the same CLI; Get Started → Install the orchestra command-line tool links it onto your PATH.

For development in this repo:

uv sync --extra server --extra dev
uv run orchestra --help

Quickstart

New to Orchestra? Use the interactive builder:

orchestra new schemas/my_test.yaml

The wizard guides you through transport, auth, and your first test step. Time to first passing test: ~3 minutes.

Or write YAML directly:

# schemas/memory_test.yaml
version: 1
name: "Memory Server Test"

server:
  transport: "stdio"
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-memory"]

steps:
  - id: create_entity
    type: tool_call
    tool: "create_entities"
    input:
      entities:
        - name: "TestUser"
          entityType: "person"
          observations: ["Loves testing"]
    save: "$"
    delay_ms: 1000

  - id: verify_created
    type: assert
    from: "create_entity"
    check:
      op: "jsonpath_exists"
      path: "$.content[0].text"

orchestra run schemas/memory_test.yaml

============================================================
 Running: Memory Server Test
 Server: stdio | Steps: 2
============================================================

📡 Connecting to MCP server...
 ✅ Connected to memory-server v0.6.3

▶ Step: create_entity (tool_call)
  ✅ Success

▶ Step: verify_created (assert)
  Check: jsonpath_exists at $.content[0].text
  ✅ Passed

═══════════════════════════════════════════════════════════
 Status: ✅ PASSED  |  Duration: 1247ms  |  2/2 steps
═══════════════════════════════════════════════════════════
📁 Report saved: reports/abc123.json

Features

CLI Reference

orchestra new [output_file]          # Interactive collection builder
orchestra inspect <server.yaml>      # Discover tools and schemas
orchestra run <collection.yaml>      # Execute a test collection
orchestra validate <collection.yaml> # Validate schema without running
orchestra info                       # Version and environment info
orchestra auth login <mcp-url>       # OAuth sign-in (saves tokens for run/inspect)
orchestra auth logout <url|profile>  # Remove stored OAuth session

Run options

orchestra run schemas/my_test.yaml --show-responses   # Print full JSON responses
orchestra run schemas/my_test.yaml --quiet            # Errors only
orchestra run schemas/my_test.yaml --output json      # Machine-readable output
orchestra run schemas/my_test.yaml --no-report        # Skip saving report file

Transports

Local (STDIO) — runs the server as a subprocess:

server:
  transport: "stdio"
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-memory"]
  env:
    API_KEY: "{{env.MY_API_KEY}}"

Remote (HTTP) — Streamable HTTP:

server:
  transport: "http"
  url: "https://mcp.deepwiki.com/mcp"

SSE:

server:
  transport: "sse"
  url: "http://localhost:3001"

Authentication

OAuth (MCP HTTP servers with authorization metadata) — run once, then use type: oauth in YAML:

orchestra auth login https://mcp.example.com
# Optional: --profile myapp   # if set, add oauth_profile: "myapp" in YAML
# Optional: --scopes "scope1 scope2"

Tokens are stored under ~/.config/orchestra/oauth_sessions.json (POSIX mode 600). This file-backed store is used by default for both CLI and the desktop GUI so OAuth works consistently across processes. Set ORCHESTRA_OAUTH_PROVIDER=chuk only if you prefer chuk's platform credential store instead (requires the optional extra: pip install "orchestra-mcp[chuk]"). If dynamic client registration fails, set ORCHESTRA_OAUTH_CLIENT_ID (and ORCHESTRA_OAUTH_CLIENT_SECRET if required).

server:
  transport: "http"
  url: "https://mcp.example.com"
  auth:
    type: "oauth"
    # oauth_profile: "myapp"  # only if you used --profile on login

# Bearer token
auth:
  type: "bearer"
  token: "{{env.API_TOKEN}}"

# API key
auth:
  type: "api_key"
  key: "{{env.API_KEY}}"

# Basic auth
auth:
  type: "basic"
  username: "{{env.USERNAME}}"
  password: "{{env.PASSWORD}}"

Assertions

- id: check_result
  type: assert
  from: "my_step"
  check:
    op: "jsonpath_contains"
    path: "$.content[0].text"
    value: "expected string"

jsonpath_gt vs jsonpath_len_gte — the comparison operators compare the value found at the path; the len_ operators compare the length of the array or string found there. $.results with three items satisfies jsonpath_len_gte: 3; $.total holding 42 satisfies jsonpath_gt: 41. Pointing a comparison at a non-number (a string, an object, null) fails the step with "Cannot compare … numerically" rather than guessing, and booleans are never treated as 1/0.

- id: enough_results
  type: assert
  from: "search"
  check:
    op: "jsonpath_gt"
    path: "$.structuredContent.total"
    value: 0

Pattern matching — jsonpath_matches uses re.search, so the pattern is found anywhere in the string, consistent with jsonpath_contains being a substring check. Anchor it with ^...$ when you want a full match. An invalid regex is rejected by orchestra validate before the run starts.

- id: looks_like_an_id
  type: assert
  from: "create"
  check:
    op: "jsonpath_matches"
    path: "$.structuredContent.id"
    value: "^user_[0-9a-f]{8}$"

Negation — add not: true to any check to invert it, instead of a separate jsonpath_not_* operator for every operator:

- id: no_stack_traces
  type: assert
  from: "search"
  check:
    op: "jsonpath_matches"
    path: "$.content[0].text"
    value: "Traceback"
    not: true

not: true flips pass and fail. It does not rescue a check that could not be evaluated at all — an unparseable JSONPath or a string under a numeric comparison stays an error, since reporting "not greater than: pass" for a value that was never a number would hide the mistake behind a green check.

MCP error detection — Orchestra distinguishes between JSON-RPC transport errors and tool-level errors (isError: true in the response body), so you can assert on both:

- id: expect_failure
  type: assert
  from: "bad_call"
  check:
    op: "is_error"   # Passes if the tool itself returned an error

Reports

Every run saves a structured JSON report:

{
  "run_id": "abc123-def456",
  "collection_name": "My Test",
  "status": "passed",
  "duration_ms": 1234,
  "server": { "name": "memory-server", "version": "0.6.3" },
  "steps": [
    {
      "id": "create_entity",
      "type": "tool_call",
      "status": "success",
      "duration_ms": 150
    },
    {
      "id": "verify_created",
      "type": "assert",
      "status": "passed",
      "duration_ms": 5
    }
  ]
}

Reports are saved to reports/ by default. Use --report-dir to customize.

Running a directory of collections also writes reports/summary.json — one entry per collection with its source path, status, step counts, and the severity counts of any contract-snapshot changes. A single exit code cannot say which of N servers moved; that file can.

Examples

version: 1
name: "DeepWiki Test"

server:
  transport: "http"
  url: "https://mcp.deepwiki.com/mcp"

steps:
  - id: ask_about_react
    type: tool_call
    tool: "ask_question"
    input:
      repoName: "facebook/react"
      question: "What are React hooks?"
    save: "$"
    delay_ms: 2000

  - id: check_answer
    type: assert
    from: "ask_about_react"
    check:
      op: "jsonpath_contains"
      path: "$.content[0].text"
      value: "hook"

version: 1
name: "Brave Search Test"

env:
  BRAVE_API_KEY: "your-api-key"

server:
  transport: "stdio"
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-brave-search"]
  env:
    BRAVE_API_KEY: "{{env.BRAVE_API_KEY}}"

steps:
  - id: search_python
    type: tool_call
    tool: "brave_web_search"
    input:
      query: "Python programming"
      count: 5
    save: "$"
    delay_ms: 2000

  - id: check_results
    type: assert
    from: "search_python"
    check:
      op: "jsonpath_len_gte"
      path: "$.content"
      value: 1

CI/CD Integration

Orchestra exits with code 0 on pass and 1 on failure — drop it into any pipeline:

# GitHub Actions
- name: Test MCP server
  run: orchestra run schemas/my_test.yaml --quiet

Contributing

Contributions welcome. Clone the repo and run pip install -e . to develop locally.

License

Proprietary — all rights reserved. See LICENSE. Built on the Model Context Protocol by Anthropic.