ConeReplay

Pre-deploy regression testing for multi-agent AI. Record traces from your production agent app, propose a change (new prompt, new tool response, new model), and replay only the events causally affected — everything else is served byte-identically from the recording.

Patent pending (U.S. Provisional Application 64/043,722). Product site and pilot details: https://conereplay.com. SemVer commitments live in VERSIONING.md and the on-disk trace contract in docs/trace-format.md in the source repository.

Install

pip install conereplay                   # core + CLI
pip install conereplay[server]           # hosted trace-store server
pip install conereplay[langgraph]        # LangGraph recording adapter
pip install conereplay[providers]        # anthropic / openai oracle backends
pip install conereplay[dev]              # dev tooling (pytest, ruff, mypy, pip-audit)

Requires Python 3.11+. Core has three runtime deps (networkx, pydantic, pydantic-settings).

CLI quickstart

# Single-trace replay → Markdown report
conereplay replay \
  --trace examples/trace.json \
  --modify examples/modify.json \
  --report report.md

# Corpus-mode regression harness → aggregate report (Markdown or HTML)
conereplay corpus \
  --traces examples/traces \
  --modify examples/corpus-modify.json \
  --report corpus.html \
  --format html

# Migration audit: re-run recorded traces on a new model, per-trace
# outcome diff + go/no-go (needs OPENAI_API_KEY or ANTHROPIC_API_KEY)
conereplay migrate \
  --traces ./traces --provider openai --model gpt-4o-mini \
  --compare judge --max-change-rate 0.05 --report migration.md

# Emit a Mermaid or Graphviz diagram of a trace's causal DAG
conereplay diagram \
  --trace examples/trace.json \
  --modify examples/modify.json \
  --format mermaid --out cone.mmd

Run conereplay <command> --help for the full reference (also in docs/cli.md in the source repository).

Library quickstart

from conereplay import (
    load_trace, ModificationSpec, selective_replay, compute_divergence,
    render_markdown_report,
)

trace = load_trace("trace.json")
mod = ModificationSpec(target_event_id="e3", substituted_output=b"POLICY: 5 days")
divergent = selective_replay(trace, mod)
report = compute_divergence(trace, divergent)
print(render_markdown_report(trace, divergent, mod, report))

Architecture

Four layers.

Supporting infrastructure:

• config.py — validated runtime config (pydantic-settings, CONEREPLAY_* env)
• flags.py — feature flags (CONEREPLAY_FLAGS_* env)
• logging_setup.py — structured text/JSON logging
• audit.py — append-only JSONL audit trail for compliance
• core/schemas.py — pydantic contract models for trace + modification JSON

Configuration

All configuration is env-driven with the CONEREPLAY_ prefix. Defaults are safe for local use. Sample overrides:

export CONEREPLAY_LOG_LEVEL=debug
export CONEREPLAY_LOG_FORMAT=json
export CONEREPLAY_DATA_DIR=/var/lib/conereplay
export CONEREPLAY_AUDIT_LOG_PATH=/var/log/conereplay/audit.jsonl
export CONEREPLAY_ENABLE_AUDIT=true
export CONEREPLAY_SERVER_DATABASE_URL=postgresql://...

See conereplay/config.py for the full list.

Testing

make install-dev    # install runtime + dev deps (pytest, ruff, mypy)
make test           # run full pytest suite
make test-fast      # fail-fast + failed-first
make test-cov       # with coverage report (htmlcov/)
make lint           # ruff
make typecheck      # mypy

CI (.github/workflows/) runs tests + lint + typecheck on every push and PR against main. Integration tests that hit real LLM providers are gated behind CONEREPLAY_INTEGRATION=1.

Contributing

See CONTRIBUTING.md in the source repository.

Security

See SECURITY.md in the source repository for supply-chain posture and vulnerability disclosure. Patent-pending proprietary code; see LICENSE in the source repository.

Runbooks

Operational procedures live in runbooks/ in the source repository:

• deployment.md — releases + migrations
• on-call.md — incident triage
• disaster-recovery.md — backups + restore

Documentation

In the source repository: docs/HANDOFF.md, docs/BACKLOG.md, docs/LOCAL_DEVELOPMENT.md, docs/TESTING.md, docs/INFRASTRUCTURE.md, docs/ROADMAP.md.

Evidence and sample

In the source repository:

• Sanitized end-to-end sample: examples/sanitized_migration/README.md
• Reproducible attribution configuration: examples/sanitized_migration/bisect.json
• Replay benchmark methodology: docs/BENCHMARKS.md

The sample is synthetic and labels fixture measurements separately from live-provider evidence.