Kneepoint

Find where your AI agent breaks. Load testing, cost-per-resolved-task, and chaos engineering for AI agents.

Your agent passes its evals at concurrency 1. Kneepoint answers the three questions that decide whether it survives production:

1. The Knee Point — the concurrency where latency (and quality) stops being flat and starts to cliff. Every agent has one; almost nobody knows theirs.
2. $ / Resolved Task — spend divided by tasks actually solved, not tokens, not requests. Retries and failures make these numbers wildly different.
3. Resilience Score — how much resolution rate survives injected chaos: rate limits, server errors, tool timeouts, malformed tool JSON.

Quickstart (zero API keys)

pip install kneepoint
kneepoint demo          # bundled agent + chaos + full report, ~90 seconds

The demo spins up a deliberately naive agent, injects faults into its LLM calls and its tool calls, and opens a report answering all three questions. Nothing leaves your machine; no keys, no spend.

Against your own agent

kneepoint init                            # starter kneepoint.yaml + prompt corpus
kneepoint run --scenario kneepoint.yaml   # ramp, judge, price, report

Any OpenAI-compatible endpoint works as a target. A scenario is one YAML file:

target:
  url: http://127.0.0.1:8000/v1
  model: my-agent

workload:
  ramp: {from: 1, to: 50, step: 5, hold_seconds: 20}
  conversation:
    turns: {min: 1, max: 3}                       # multi-turn: context grows
    corpus: ./prompts/*.txt                       # real prompt distribution

resolution:
  check: {kind: contains, value: "[RESOLVED"}     # or a regex, or an LLM judge

cost:
  input_per_mtok: 3.00
  output_per_mtok: 15.00
  max_spend: 0.50                                 # hard cap: run stops if crossed

chaos:
  profile: standard                               # 429s, 503s, tool faults

slo:
  min_resolution_rate: 0.90                       # breach -> exit code 1 (CI gate)

Example scenarios: support bot · RAG agent · agent with MCP tools

CI gate

Kneepoint exits non-zero when your agent regresses — wire it into CI like any other test:

- run: pip install kneepoint
- run: kneepoint run --scenario kneepoint.yaml --out reports
- uses: actions/upload-artifact@v4
  if: always()
  with: {name: kneepoint-report, path: reports/}

Exit codes: 0 pass · 1 SLO breach · 2 usage error · 3 budget cap hit. Full CI docs →

How it's different

Different layers, honestly: evals tell you the agent can do the task; observability tells you what happened in prod. Kneepoint tells you where it breaks before prod does. Use all three.

Roadmap

• The Kneepoint Index — published knee/$/resilience benchmarks of popular agent frameworks
• Native MCP target adapter (fault-inject MCP tool calls first-class)
• More injectors: stream_cut, tool_error, tool_stale_data, slow_tokens — good first issues
• N-run variance mode

Contributing

pip install -e ".[dev]", then ruff check . && pytest -q — the whole suite runs against a bundled mock agent, $0 and no keys. See CONTRIBUTING.md; new fault injectors are the perfect first PR.

Built in public — follow along at kneepoint.dev. MIT licensed.