PandaProbe Harness

Self-healing for AI agents. The harness wraps any PandaProbe-instrumented agent in an operational envelope that evaluates every turn, alerts the agent when quality degrades, and lets it diagnose its own failures and write — and prove — its own operating rules. Fully automatic, no human in the healing loop.

📚 Documentation · 📦 PyPI · 💬 Discussions

How it works

1. Evaluate — after each turn, the harness scores newly completed traces through a three-tier ladder and watches their task trajectory for stalls or regressions.
2. Notice — trajectory failures and step-level breaches post a structured diagnostic notice to a workspace mailbox. Nothing is ever injected into the agent's conversation.
3. Heal — guided by a standing protocol in its system prompt, the agent pulls the notice, inspects its own flagged traces, and records a mitigation rule.
4. Validate — the rule enters as a candidate: the harness replays the captured failure (or watches the next live sessions) and promotes it only when it demonstrably helps. Validated rules re-enter the prompt on every future run; a replayable eval-set guards old wins against regressions.

The core has zero runtime dependencies and reaches the platform exclusively through the pandaprobe CLI.

Installation

pip install pandaprobe-harness
# framework adapters are optional extras:
pip install "pandaprobe-harness[langgraph]"     # [langchain] [deepagents] [crewai]
                                                # [claude-agent-sdk] [openai-agents] [all]

You'll also need the pandaprobe CLI installed and authenticated, and an agent traced with the PandaProbe SDK.

Quickstart

from pandaprobe_harness import Harness
from pandaprobe_harness.agent_tools.native import as_anthropic_tools

harness = Harness.create()                              # provisions the workspace

system_prompt = harness.system_context() + MY_PROMPT    # protocol + references + banner
specs, dispatch = as_anthropic_tools(harness.toolset)   # the self-diagnostic tools
tools = my_tools + specs

async def one_turn(session_id: str, user_input: str) -> str:
    # `settle=True` waits for this turn's evaluation and any notice before
    # returning, so a rule learned now is in force for the next turn.
    async with harness.turn(session_id, settle=True):
        return await my_agent_step(system_prompt, tools, user_input)

The system context carries the protocol, the tool list, and an index of the agent's rule files — never the rule bodies. The agent pulls those on demand with harness_rules_read, so the workspace stays its own.

Using a framework? Harness.for_langgraph(), for_langchain(), for_deepagents(), for_crewai(), for_claude_agent_sdk(), and for_openai_agents() wire turn detection for you.

➡ Full guides, concepts, and the configuration reference live in the documentation.

Try it offline

The examples/ directory ships fully-offline, credential-free demos:

make example                                        # the pull loop, end to end
uv run python examples/misc/closed_loop_self_heal.py  # candidate → validate → promote → regression
uv run python examples/misc/calibration_demo.py       # threshold calibration

Operator CLIs

Benchmarks

An A/B study measuring the harness's effect on agent reliability across AppWorld, Terminal-Bench (via Harbor), and τ²-bench lives in benchmarks/ — a self-contained uv project that installs the released harness from PyPI. Run it from the repo root with make bench-setup, make bench-smoke, and make bench-report (see benchmarks/README.md).

Development

make install         # uv sync
make test            # full offline suite — no network, no real CLI
make lint typecheck  # ruff + mypy --strict

See CONTRIBUTING.md for the project invariants and PR process, and CHANGELOG.md for release history.

License

MIT © Chirpz AI