Runflect

The reliability and security toolkit for AI agents.

Trace executions. Enforce tool-use policies. Benchmark agent behavior. Investigate failures with inspectable evidence.

An agent calls a tool it shouldn't. A workflow fails halfway through. A policy change introduces a regression you didn't anticipate.

When something goes wrong, scattered logs and inconsistent traces make it difficult to understand what happened, identify the failure, and verify whether a fix actually works.

Runflect turns agent execution into actionable evidence.

Record execution traces, control tool access through explicit policies, evaluate security boundaries with repeatable scenarios, and catch regressions through benchmark comparisons—all in one local-first toolkit for developers building AI agents. Observe execution. Enforce boundaries. Test before you trust.

Quickstart · Features · CLI · Architecture · Local dashboard

Project status: Runflect is an early-stage developer preview. The distribution name is runflect, but public PyPI installation is not yet available. Install from a source checkout using the instructions below.

Important: Runflect provides observability and policy enforcement at instrumented boundaries. It does not reveal hidden model reasoning, guarantee agent security, or automatically explain every failure.

---

What Runflect provides

Runflect brings agent execution observability, tool-use controls, evaluation, and local security evidence into one developer workflow.

Capability	What it does
Execution tracing	Records runs, nested spans, timing, status, and bounded typed attributes in a versioned SQLite database.
LangGraph integration	Optional synchronous and asynchronous adapters record observed graph, node, tool, and supported model callback events.
Tool-use policies	ToolGuard supports allow, deny, approval-required, and configurable default-deny decisions around wrapped tool calls.
Policy lifecycle	Validate policies, import immutable revisions, review history, evaluate scenarios offline, and explicitly activate revisions.
Runflect Bench	Run deterministic evaluation scenarios and inspect outcomes, metrics, and host-specific timing observations.
Regression testing	Save versioned baselines, compare compatible reports, apply supported thresholds, and use CI-compatible exit codes.
Security simulation	Execute deterministic offline mock scenarios against the actual guarded execution path and retain evidence-backed reports.
Negative controls	Exercise a deliberately unguarded prohibited mock action to verify that the test harness detects the expected failure.
Supply-chain inventory	Scan supported local manifests and configuration files, build a dependency/configuration evidence graph, and report bounded static findings.
Cost observations	Calculate bounded model-cost observations from explicit token usage and an operator-supplied pricing configuration.
OpenTelemetry export	Explicitly export completed recorded traces through an application-configured OpenTelemetry SpanExporter.
Local web interface	Explore traces and configured policy, Bench, Security, and Supply Chain reports through a read-only local API and dashboard.
CLI 2.0	Use a redesigned terminal experience with a workspace homepage, contextual help, structured output, clearer diagnostics, and improved run inspection.

These capabilities share local execution records, policy evidence, reports, and explicit references. Relationships are shown only when the available evidence supports them.

Who is it for?

• AI agent developers who need to inspect tool calls, execution paths, and failures.
• Framework engineers integrating consistent tracing and policy boundaries into agent applications.
• AI platform teams building applications where tool access needs explicit authorization rules.
• Evaluation engineers who need repeatable scenarios, baseline comparisons, and regression checks.
• Security engineers who want to test specific guarded execution boundaries and review the evidence.
• Developer-tool builders who want local traces, structured reports, and optional OpenTelemetry export.

Runflect is a developer toolkit, not a hosted multi-user service or a replacement for application-specific security controls.

---

CLI 2.0

The terminal is a primary interface to Runflect, not an afterthought.

CLI 2.0 focuses on making execution evidence and evaluation results easier to discover, inspect, and use in local development and CI workflows.

A more useful terminal experience

• Workspace homepage: Running runflect without arguments presents a compact overview of the local workspace, database status, available capabilities, and useful next commands.
• Contextual help: Consistent command descriptions, options, examples, and recovery guidance across command groups.
• Execution inspection: Readable run listings and nested span details, with bounded output and safer metadata handling.
• Clear diagnostics: More actionable errors for invalid arguments, missing data, and operational failures.
• Structured output: JSON output for supported commands, suitable for scripts and automation.
• Terminal-aware rendering: Respect for color settings and non-interactive output, with a plain-text experience for logs and CI.
• Module invocation: Support for python -m runflect alongside the runflect executable.
• Version reporting: Inspect the installed version with runflect --version.

Available command groups

Command	Purpose
runflect	Open the terminal homepage.
runflect --help	Explore commands and global options.
runflect list	List recorded agent executions.
runflect show RUN_ID	Inspect a run and its nested spans.
runflect policy --help	Explore policy validation, history, evaluation, and activation.
runflect bench --help	Run evaluations, manage baselines, and compare reports.
runflect security --help	Validate and execute deterministic security scenarios.
runflect supply-chain --help	Inspect the local repository scanning workflow.

The terminal homepage distinguishes a missing database from available recorded data. It does not manufacture activity or scores to make the workspace look populated.

See the CLI 2.0 guide for command behavior, output formats, exit codes, and the PowerShell acceptance walkthrough.

---

What makes Runflect different?

Agent observability, tool permissions, and evaluation are often treated as separate concerns. Runflect brings them together around explicit execution evidence.

1. Understand execution

Record application-defined runs and nested spans. Inspect timing, statuses, relationships, and safe metadata instead of reconstructing every failure from scattered logs.

2. Control tool access

Wrap selected tool calls with ToolGuard. Evaluate policy decisions, audit guarded execution, and prevent a guarded tool from running when its pre-execution audit cannot be recorded successfully.

3. Evaluate before activation

Validate and import policy revisions, evaluate them against deterministic scenarios, inspect the results, and activate a revision explicitly.

4. Test security boundaries

Run versioned offline scenarios against the actual guarded execution path. Review the evidence behind each verdict, including deliberate negative controls.

5. Detect evaluation regressions

Save compatible Bench baselines, compare new reports, and apply supported regression thresholds in automated workflows.

6. Inspect declared dependencies

Scan supported repository files and connect findings to their source locations. Preserve uncertainty when a file format is unsupported or runtime attribution is missing.

7. Keep evidence accessible

Use the CLI, optional local dashboard, JSON reports, and explicit OpenTelemetry export to work with recorded evidence.

The objective is not to claim that every failure or attack can be prevented. It is to make agent behavior easier to inspect, test, and review.

---

How it works

The recorder observes application execution that developers choose to record. ToolGuard evaluates and audits policy decisions around wrapped tool calls. Policy evaluation can run offline without executing the underlying tools. Bench and security simulations use controlled local fixtures. The optional web API reads existing Runflect data without executing tools or modifying policies.

flowchart LR
    App[Agent application] --> Recorder[Runflect recorder]
    App --> Guard[Optional ToolGuard]
    Guard --> Tool[Wrapped tool]
    Guard --> Audit[Policy audit event]
    Recorder --> DB[(Local SQLite)]
    Audit --> DB

    Policy[Policy and scenarios] --> Eval[Offline policy evaluation]
    Eval --> DB

    Bench[Bench scenarios] --> BenchReport[Versioned JSON report]
    Security[Security simulations] --> SecurityReport[Evidence-backed report]
    Repo[Explicit repository path] --> Scanner[Local static scanner]
    Scanner --> ScanReport[Inventory and findings]

    DB --> CLI[Runflect CLI]
    DB --> API[Optional read-only local API]
    BenchReport --> API
    SecurityReport --> API
    ScanReport --> API
    API --> Web[Local web dashboard]
    DB --> OTel[Explicit OpenTelemetry export]

Runflect is local-first. The core does not require a hosted control plane, external model provider, or network connection for its offline recording, policy, Bench, and security-fixture workflows.

---

Quickstart

The core supports Python 3.10 or newer and has no mandatory runtime dependencies.

The package is not yet available from the public PyPI index. Install from the source checkout.

1. Install Runflect

From the repository root:

python -m pip install -e .

Verify the CLI:

runflect --version
runflect --help

You can also invoke it as a Python module:

python -m runflect --help

2. Record your first execution

Run the bundled deterministic example:

python examples/quickstart.py

Inspect recorded runs:

runflect list --db .runflect/traces.sqlite3

The example prints the run ID. Inspect it with:

runflect show RUN_ID --db .runflect/traces.sqlite3

Replace RUN_ID with the identifier printed by the example.

The example records two nested workflow steps and does not require a model provider or external service. It writes to .runflect/traces.sqlite3 unless RUNFLECT_DB is set.

Its compose-answer span explicitly attributes itself to file:examples/quickstart.py, the scanner's stable source-file identifier when scanning this checkout. This is caller-supplied attribution, not proof that the file caused or supplied a runtime dependency.

3. Use the Python API directly

from runflect import run

with run("support-answer", db_path=".runflect/traces.sqlite3") as execution:
    with execution.span(
        "retrieve-context",
        input={"query": "refund policy"},
    ) as span:
        result = "Refunds are available within 30 days."
        span.set_output({"result": result})

print(execution.run_id)

The recorder stores values only when application code passes them to input or set_output.

Common sensitive mapping keys are redacted by default. Each payload is capped at 64 KiB, and oversized payloads are omitted with a safe span attribute. Error messages are omitted by default.

Review Security and privacy before recording application data.

---

LangGraph integration

Runflect provides optional synchronous and asynchronous LangGraph adapters.

Install the integration:

python -m pip install -e ".[langgraph]"

Run the example:

python examples/langgraph_quickstart.py

The example builds and invokes a local LangGraph with a deterministic tool fixture and prints run IDs for success and failure cases.

The adapter supports:

• graph.invoke()
• await graph.ainvoke()
• Observed graph, node, and tool callback events
• Parent-child relationships between supported spans
• Explicit model/provider labels and provider-supplied integer token usage when supported callbacks provide them

Prompts and model responses are omitted by default.

The asynchronous example is:

python examples/langgraph_async_quickstart.py

The supported optional dependency range is langgraph>=1.2.14,<2.0. Consult the architecture notes for tested integration details and boundaries.

Optional model-cost attribution

Reliability can calculate bounded cost observations when explicit token usage is available and you provide a versioned pricing file through RUNFLECT_PRICING_CONFIG.

Runflect does not invent default prices, infer missing token usage, or treat an unknown cost as zero. Partial pricing coverage remains partial. These observations are estimates, not invoice reconciliation.

See reliability documentation.

---

Policies and scenario evaluation

Run the policy example:

python examples/policy_quickstart.py

Validate a policy:

runflect policy validate examples/policies/support-tools.json --json

Import an immutable policy revision:

runflect policy import examples/policies/support-tools.json --db .runflect/policy.sqlite3 --json

Evaluate the revision against offline scenarios:

runflect policy evaluate support-tools 1.0.0 --scenarios examples/scenarios/support-tools.json --db .runflect/policy.sqlite3 --json

Activate it explicitly after review:

runflect policy activate support-tools 1.0.0 --db .runflect/policy.sqlite3

Inspect its history:

runflect policy history support-tools --db .runflect/policy.sqlite3

ToolGuard supports allow, deny, approval-required, and configurable default-deny decisions.

Importing or evaluating a policy does not activate it. Activation is a separate operation. Scenario evaluation checks policy decisions without executing the underlying example tools.

The guard applies only to calls made through its wrapper. Calls made outside that boundary can bypass it.

---

Security simulation

Run deterministic, versioned offline scenarios against the actual ToolGuard and recorder.

Validate a suite:

runflect security validate examples/security/core-v1.json --json

Execute it with an explicit evidence database and report path:

runflect security run --suite examples/security/core-v1.json --db .runflect/security-simulation.sqlite3 --output .runflect/security-report.json

The bundled core suite uses fixed in-process mock tools to exercise scenarios involving:

• Unauthorized tool calls
• Synthetic secret handling
• Prompt-like untrusted instructions
• Dangerous tool chains
• Harness call budgets
• Approval-required calls

An optional langgraph_fake_model scenario exercises the supported LangGraph callback, model, tool, and ToolGuard path without a model provider, network access, credentials, or external side effects.

A deliberate unguarded negative control can execute a prohibited mock operation. The expected result is a failing verdict with evidence that the operation executed. This tests whether the harness detects that specific unsafe outcome.

Reports distinguish PASS, FAIL, INCONCLUSIVE, ERROR, and SKIPPED.

A passing fixture means only that the tested scenario produced its expected result. It does not establish general prompt-injection resistance or comprehensive agent security.

See Security simulation for verdict rules, report evidence, exit codes, and limitations.

---

Runflect Bench

Run the bundled deterministic evaluation suite and write a versioned report:

New-Item -ItemType Directory -Force .runflect | Out-Null

runflect bench run --output .runflect/bench-report.json

The report includes scenario outcomes, metric numerators and denominators, runtime information, and host-specific timing samples.

Save a baseline

runflect bench baseline save --report .runflect/bench-report.json --output .runflect/bench-baseline.json --label main

Compare against a baseline

runflect bench compare --baseline .runflect/bench-baseline.json --current .runflect/bench-report.json --output .runflect/bench-comparison.json

The comparison workflow supports compatible versioned baselines, configured thresholds, and CI-compatible exit codes.

A failed configured threshold returns a nonzero exit code. A requested timing gate without enough samples is reported as INCONCLUSIVE, with exit code 4.

The local Bench view reads a persisted comparison report; it does not execute benchmarks itself.

Baseline policy bypasses are an explicit comparison control, not an enforced configuration. Timing results depend on the host and workload and are not performance guarantees.

Read the Bench report schema and Bench regression guide before consuming reports programmatically.

---

Local repository and supply-chain scanning

Run a scan against an explicit repository path:

New-Item -ItemType Directory -Force .runflect | Out-Null

runflect supply-chain scan . --output .runflect/supply-chain-report.json

Fail when findings meet the configured severity threshold:

runflect supply-chain scan . --fail-on high

The scanner reads supported Python and npm dependency files, selected Docker and GitHub Actions configuration, MCP server declarations, and literal Python model references.

It builds a versioned inventory/evidence graph and reports bounded static findings such as:

• Non-exact dependency declarations
• GitHub Actions references that are not pinned to commit SHAs
• Other supported configuration patterns

Reports include counts, findings, source locations when available, parse errors, unsupported formats, and coverage limitations.

The scanner does not execute repository code, download vulnerability advisories, establish package exploitability, verify artifact provenance, or prove runtime reachability.

A successful scan does not mean that the repository has no vulnerabilities. It means the scan completed according to the implemented rules and configured failure threshold.

See the Supply Chain guide for exact coverage and limitations.

---

OpenTelemetry export

Runflect supports explicit export of completed recorded traces through an OpenTelemetry SpanExporter supplied by your application.

Install the optional dependencies:

python -m pip install -e ".[opentelemetry]"

Then call the export function:

from runflect.integrations.opentelemetry import export_trace

# Supply an existing run ID and an exporter configured by your application.
export_trace(run_id, exporter, db_path=".runflect/traces.sqlite3")

Replace run_id and exporter with your actual run identifier and configured exporter.

Export is explicit and uses existing recorded traces. Exported telemetry leaves the local database boundary, so review your destination and data-handling requirements first.

See OpenTelemetry export for the mapping, privacy behavior, and failure limits.

---

Run the local web interface

Runflect includes an optional read-only dashboard for inspecting local traces and configured reports.

The web app uses Next.js 16 and requires Node.js 20.9 or newer. The Python API is an optional web extra.

1. Install the API dependencies

From the repository root:

python -m pip install -e ".[web,supply-chain]"

New-Item -ItemType Directory -Force .runflect | Out-Null

$env:RUNFLECT_DB = (Join-Path (Get-Location) ".runflect\traces.sqlite3")
$env:RUNFLECT_BENCH_REPORT = (Join-Path (Get-Location) ".runflect\bench-report.json")
$env:RUNFLECT_SUPPLY_CHAIN_REPORT = (Join-Path (Get-Location) ".runflect\supply-chain-report.json")
$env:RUNFLECT_SECURITY_REPORT = (Join-Path (Get-Location) ".runflect\security-report.json")

python examples/quickstart.py
python examples/policy_quickstart.py

runflect bench run --output .runflect/bench-report.json
runflect supply-chain scan . --output .runflect/supply-chain-report.json

runflect security run --suite examples/security/core-v1.json --db $env:RUNFLECT_DB --output .runflect/security-report.json

runflect-web-api

2. Start the frontend

In a second terminal:

Set-Location web

npm ci

$env:NEXT_PUBLIC_RUNFLECT_API_URL = "http://127.0.0.1:8765"

npm run dev -- --hostname 127.0.0.1

Open:

• Landing page: http://127.0.0.1:3000
• Dashboard: http://127.0.0.1:3000/dashboard

The setup commands generate local data and reports before starting the API.

The dashboard includes:

• Run overview and trace inspection
• Policy history and decisions
• Bench results and persisted comparisons
• Security simulation reports
• Supply-chain inventory and findings
• Evidence-linked navigation where explicit references exist

The API binds to loopback, allows only configured localhost browser origins, serves GET requests only, and does not execute tools or modify policies.

Supply Chain links to spans only when the available evidence contains the exact matching unique component ID. That relationship represents explicit caller attribution, not proven dependency use or causality.

The local API is a read-only convenience, not a hosted service or complete security boundary.

See the web guide, Supply Chain guide, and Security simulation guide for further details.

Optional LangGraph traces in the dashboard

Install the LangGraph extra as well:

python -m pip install -e ".[langgraph,web,supply-chain]"

With RUNFLECT_DB pointing to the same local database, run:

python examples/langgraph_quickstart.py

The example records one successful and one deliberately failed LangGraph execution using a fixed tool and deterministic fake chat model. It does not contact a model provider or external network.

---

Example developer workflow

A typical local workflow looks like this:

1. Record: Run python examples/quickstart.py and inspect executions using runflect list and runflect show.
2. Protect: Wrap sensitive tool calls with ToolGuard, using examples/policy_quickstart.py as a starting point.
3. Evaluate: Import and evaluate a policy revision against offline scenarios, then review it before explicit activation.
4. Benchmark: Run runflect bench run and preserve the versioned report.
5. Compare: Save a baseline and compare subsequent reports to detect supported regressions.
6. Simulate: Run runflect security run and inspect scenario verdicts and recorded mock traces.
7. Inspect dependencies: Run runflect supply-chain scan . to review supported repository evidence.
8. Investigate: Follow exact evidence references across the local CLI and dashboard when those relationships are available.
9. Export: Optionally export a completed trace through your configured OpenTelemetry exporter.

The purpose is to create a repeatable loop:

Record → Protect → Evaluate → Inspect → Improve

This is an evidence-driven developer workflow, not an automatic root-cause engine or a guarantee that every agent failure can be reproduced.

---

Security and privacy

Runflect is designed for local inspection, but local-first does not mean risk-free.

• Explicit capture: The core recorder does not automatically inspect agent state. It stores values explicitly passed to span inputs and outputs.
• Default redaction: Common sensitive mapping keys are recursively replaced with [REDACTED] by default. Setting redact=False disables this protection.
• Payload limits: Recorded payloads are bounded. Oversized payloads are omitted with a safe span attribute.
• Error messages: Error messages are omitted by default. capture_error_messages=True opts into storing messages up to 4,096 characters after key-based redaction.
• LangGraph privacy: Input/output capture is off by default and can be opted into using capture_inputs=True and/or capture_outputs=True. Exception messages are omitted by the adapter.
• Guard boundaries: ToolGuard protects calls made through its wrapper. Direct calls outside that boundary can bypass it.
• Audit behavior: A pre-execution policy audit failure blocks a guarded tool. A later audit or trace write failure cannot undo an external side effect.
• Local storage: SQLite is not encryption or a tamper-proof ledger. Protect the database with host filesystem controls.
• Approvals: The approval authority is in-process, not a distributed approval service.
• Web interface: The API is a local read-only convenience with loopback and Host/CORS checks, not a complete security boundary.
• Scanner scope: Repository scanning is bounded static analysis. It does not execute code or download vulnerability advisories.
• OpenTelemetry: Export is explicit. Exported telemetry leaves the local database boundary and should be handled accordingly.

Runflect is not a sandbox, does not prevent arbitrary code from bypassing wrappers, and does not guarantee secure agent behavior.

Read the threat model, architecture and guarantee boundaries, and release checklist.

---

Current status and limitations

Runflect is an early-stage developer preview. Its implemented capabilities include local Python execution recording and CLI inspection, synchronous and asynchronous LangGraph callbacks, guarded tool policies and audit records, immutable policy history, offline scenario evaluation, deterministic Bench and regression comparisons, security simulation fixtures, explicit OpenTelemetry export, an optional local read-only web interface, and bounded static repository scanning.

The CLI 2.0 redesign adds a terminal homepage, contextual help, module invocation, structured output options, clearer diagnostics, and improved run inspection.

The following limitations remain important:

• Model instrumentation: Provider/model callback spans and token counts are available only when supported callbacks supply the necessary metadata. Streaming-token capture and cross-process checkpoint resume are unsupported.
• Cost attribution: Cost observations require a user-supplied versioned rate file and complete explicit input/output token usage. They are partial estimates, not invoice reconciliation.
• Policy controls: Policy argument constraints, distributed approvals, and durable external approval workflows are unavailable.
• Security coverage: The offline fixtures exercise specific deterministic boundaries. The optional LangGraph fake-model scenario is not a real-provider attack evaluation and does not establish general prompt-injection resistance.
• Runtime enforcement: Calls outside a guarded wrapper are not protected by that wrapper.
• Investigation: Evidence-linked navigation exists where exact references are available. A general causal graph, automatic root-cause analysis, and general-purpose deterministic replay are not provided.
• Supply chain: The scanner does not provide complete SBOM coverage, known-vulnerability advisory matching, model provenance verification, or proven runtime dependency reachability.
• Hosting: Remote ingestion, multi-user hosting, encryption, retention management, and tamper-proof audit storage are unavailable.
• Benchmarking: Functional scenarios are deterministic within their fixtures. Timing measurements depend on the host and workload and are not performance guarantees.
• Distribution: The package has not yet been published to public PyPI. Source-checkout installation is the supported installation path for now.

These are known product boundaries, not capabilities to rely on today.

---

Development and testing

Install the optional integrations used by the full test suite:

python -m pip install -e ".[langgraph,opentelemetry]"

Run the Python tests:

python -m unittest discover -s tests -v

Compile the Python source and examples:

python -m compileall -q src tests examples

Run the release smoke test:

python examples/release_smoke.py

The release smoke test exercises the deterministic Bench suite, security scenarios, policy/audit behavior, and trace export.

Frontend checks

From the web/ directory:

npm ci
npm run typecheck
npm run build

Run browser tests if the repository's Playwright setup is installed and configured:

npm run test:e2e

Useful project documentation

• CLI design, commands, exit codes, and PowerShell acceptance
• Architecture and guarantee boundaries
• Local threat model and tested boundaries
• Windows PowerShell acceptance walkthrough
• Bench report schema
• Bench baselines and regression comparisons
• Custom offline evaluation suite
• Local reliability observations and cost attribution
• OpenTelemetry export
• Local repository scanning
• Local web interface
• Release readiness checklist

---

Contributing and license

Runflect is under active development. Please review the repository's current contribution guidance and license before contributing, reusing, or distributing the code.

The repository checkout used to prepare this README did not contain a LICENSE file or a contribution guide. Add the intended license and contribution instructions before making public release claims.