Find wasted work in AI agents. Verify each change.
Trace calls, inspect findings, and compare performance with your quality checks.

Quickstart · Why AgentLoop · Compare runs · Integrations · Guide

AgentLoop records model calls, tool calls, and retries in an agent workflow. Use its reports to inspect latency, token usage, and estimated model costs, find repeated work, and compare runs before and after a change.

Tracing and analysis run locally. You can work with JSON files without a database or hosted account. The dashboard and HTTP API are optional.

AgentLoop also profiles classifiers, rules and decision pipelines. See AI execution profiling for the trace-to-quality workflow and five offline reference examples in 0.1.0.

The Harbor and Omnigent interoperability workstream includes read-only Harbor ATIF and job/trial/verifier import, source-qualified native reports and pinned offline fixtures. Omnigent telemetry/session import preserves policy observations and coverage gaps. Task-paired cross-harness comparisons and behavior-verified capabilities are included in this release.

The bounded OTLP JSONL importer joins split telemetry traces and keeps duplicate/conflicting records, conversion losses and inferred timing explicit. It works offline without upstream runtimes or OpenTelemetry packages.

The capability registry lists explicit adapter declarations and distinguishes current probe evidence from unknown/skipped capabilities. Inspection does not enable runtime controls.

Run the offline capability bench to check actual wrapped behavior, source counters and declaration drift. Missing SDKs and live-only features remain explicit skips.

The cross-harness cohort workflow compares completed Harbor jobs with exact task pairing and independent verifier criteria, while retaining failed, missing and unpaired attempts in population reports.

Coordination reports preserve distinct child documents, observed interval overlap and usage attribution, with explicit gaps for missing children, caller wait, duplicate-work inputs and review quality.

The offline adoption guide connects the copyable examples, source/quality limits, complete inventories and validation workflow. These integrations are Unreleased checkout capabilities.

Without and with AgentLoop

AgentLoop supplies the tracing and comparison workflow. You choose the change, define what a correct output looks like, and decide which checks must pass.

How AgentLoop fits your stack

Your application runs the agent and chooses its framework and models. AgentLoop's decorators and adapters record the instrumented calls. Exported traces feed analysis, baseline/candidate replay, and reports for review or CI.

When you change the framework, model, prompt, or execution strategy, record the same task again. Compare both runs with the same quality fixtures to check the effect of your change.

Quickstart

Requires Python 3.10 or newer:

python -m pip install agentloop-profiler
agentloop quickstart

The package name is agentloop-profiler; the Python import and CLI command are both agentloop.

quickstart creates runs/agentloop_quickstart.json and prints findings from synthetic data. Use it to check the installation and learn the report format. The demo itself makes no network or model API calls.

Inspect the saved trace and export the complete analysis:

agentloop analyze runs/agentloop_quickstart.json --json-out runs/quickstart_analysis.json

The analysis JSON contains the trace, its metrics, findings, and optimization suggestions. See First useful result for more options.

Add --html runs/report.html to analyze for a single offline report with a timeline and linked finding evidence. See HTML reports.

Record your own run

Save this complete example as example_agent.py. It traces a few local function calls so you can try instrumentation without a model SDK or API key. Replace the function body with your own work when adapting it to an agent.

import agentloop

NOTES = {
    "tracing": "Record the steps in a workflow.",
    "optimization": "Investigate repeated or expensive work.",
    "replay": "Compare saved baseline and candidate traces.",
}


@agentloop.trace_tool(name="lookup_note")
def lookup_note(topic: str) -> str:
    return NOTES[topic]


with agentloop.trace_agent("research_agent") as trace:
    output = {"notes": [lookup_note(topic) for topic in NOTES]}
    trace.metadata["output"] = output

trace.export_json("runs/research_agent.json")
print(output)

Run it, then analyze the file it created:

python example_agent.py
agentloop analyze runs/research_agent.json --json-out runs/research_agent_analysis.json

The decorator records each function call's duration and status. The example also explicitly saves the final output in trace metadata so the quality checks below can read it. Decide which outputs are appropriate to retain when adapting this example to real data.

For model calls, use an SDK integration to capture provider token usage. The integration guide also covers model decorators, context managers, streaming, and cancellation.

Inspect findings

Start with analyze. It combines metrics, findings, and proposed optimizations in one report. AgentLoop looks for patterns such as repeated context, retry loops, repeated model calls, large steps, and recurring tool calls.

On main, reports and optimization use the same versioned finding rules. If a rule fails, its diagnostics remain visible and analysis is marked incomplete. CI requires complete analysis before recommending a merge.

Treat a suggestion as a change to investigate. For example, repeated tool names alone do not establish that calls are safe to run concurrently. The development version on main exposes the evidence and assumptions behind parallelization candidates.

Use individual commands when you need a specific artifact:

Change one thing at a time, such as reducing repeated context or changing a retry policy, so you can attribute the result to that change.

Compare before and after

Run your agent before the change and export runs/baseline.json. Run the changed agent on the same task and export runs/candidate.json. replay compares these saved traces:

agentloop replay --baseline runs/baseline.json --candidate runs/candidate.json --min-latency-improvement-pct 10

This requires at least a 10% latency improvement. The report also compares costs, tokens, retries, and call counts. It writes runs/replay_report.md and exits nonzero when a required gate fails. Use agentloop replay --help to choose other thresholds or a JSON output path.

A result applies to the runs and checks you supplied. Repeat the comparison across representative tasks before drawing broader conclusions. Add output checks when you need evidence that the changed agent still meets its task requirements.

Quality gates

Quality fixtures define checks against the recorded output. For the example above, save this as fixtures.json:

{
  "fixtures": [
    {
      "id": "notes_present",
      "scorer": {"type": "required_fields", "required": ["notes"]}
    }
  ]
}

Run the check against the example trace:

agentloop quality-report fixtures.json --candidate runs/research_agent.json

This checks that the final output has a nonempty notes field. Choose expectations that express correctness for your own task. Fixtures can provide outputs directly or use a trace's metadata.output; the scorer can also use the last model call's recorded output text when available.

Every candidate case must pass. If you set --min-quality-score in replay/ci (or --min-score in quality-report), the average score must also meet that threshold. Invalid suites fail closed.

Custom scorers run Python code and should be used only with trusted local fixtures. The HTTP endpoint rejects them. See the changelog for regex and json_schema migration notes.

Once you have baseline and candidate traces, use the same fixtures in CI:

agentloop ci --baseline runs/baseline.json --candidate runs/candidate.json --quality-fixtures fixtures.json

ci writes a Markdown report suitable for a pull request and exits nonzero when a required gate fails. It evaluates the traces your application supplies.

The repository's required replay check uses synthetic demos to test AgentLoop's CI machinery. For comparisons using your application's artifacts, see CI self-tests and application comparisons.

Understand the results

These notes describe the current main branch. Check Unreleased changes for behavior that has not yet shipped in the published package.

Cost gates can be marked indeterminate when pricing or usable token counts are missing. An indeterminate cost comparison does not fail the default gate; requiring a positive --min-cost-improvement-pct makes unverifiable cost improvement fail. See Pricing and Trace compatibility for the details, including how older traces are handled.

The optimizer avoids double-counting suggestions that affect the same spans. Reports say whether the chosen combination is exact or approximate. See Savings selection. For individual predictions, see Estimator provenance. The intervention ledger links finding predictions to measured candidate results and preserves quality checks in an exportable record. For experiments across tasks and repetitions, use paired study summaries.

Keep your Markdown renderer's safe mode or HTML sanitizer enabled when viewing reports from untrusted traces.

Framework integrations

For existing telemetry, OTLP conformance covers GenAI, OpenInference, and MCP JSON, including operation kinds and evidence metadata. From an unreleased source checkout, standards-first onboarding validates existing telemetry before analysis and provides one explicit SDK instrumentation entrypoint with documented coverage and measured local overhead.

The adapters are included in AgentLoop. Install the framework's own SDK separately when needed; python -m pip install "agentloop-profiler[instrumentation]" includes the OpenAI Python SDK. See Framework integrations for setup and supported behavior.

Dashboard and persistence

The dashboard reads stored traces. From a source checkout, load the synthetic example and start Streamlit:

uv run --frozen --all-extras agentloop quickstart
uv run --frozen --all-extras agentloop store-trace --path runs/agentloop_quickstart.json --project-id default
uv run --frozen --all-extras streamlit run dashboard/app.py

Open the URL printed by Streamlit and leave the sidebar's Project field set to default. To inspect your own data, store your trace file with store-trace.

AgentLoop uses SQLite by default and supports Postgres for shared storage. The optional HTTP API accepts traces and serves project-scoped reports and findings. Application endpoints use /v1. Existing unversioned aliases remain available throughout 0.x; see API versioning for the migration path. See Dashboard setup and Production deployment for database configuration, API authentication, and deployment instructions.

Other installation options

Source checkout

Use a source checkout for the dashboard or the latest unreleased changes. Install uv, then run:

git clone https://github.com/dipeshbabu/agentloop.git
cd agentloop
uv sync --locked --all-extras
uv run --frozen --all-extras agentloop quickstart

Standalone CLI

The GitHub Releases page has executables for Linux x86-64, Windows x86-64, macOS Intel, and macOS Apple silicon. They bundle Python and the core CLI. Use the Python package for SDK instrumentation, the dashboard, API server, or Postgres support.

Use the downloaded executable wherever the examples show agentloop.

set -euo pipefail
version=X.Y.Z
asset="agentloop-v${version}-linux-x86_64"
curl -LO "https://github.com/dipeshbabu/agentloop/releases/download/v${version}/${asset}"
curl -LO "https://github.com/dipeshbabu/agentloop/releases/download/v${version}/SHA256SUMS"
mapfile -t checksum_lines < <(awk -v asset="$asset" '$2 == asset { print }' SHA256SUMS)
if [ "${#checksum_lines[@]}" -ne 1 ]; then
  echo "Expected exactly one checksum for ${asset}" >&2
  exit 1
fi
printf '%s\n' "${checksum_lines[0]}" | sha256sum --check --strict
chmod +x "$asset"
./"$asset" --help

$Version = "X.Y.Z"
$Asset = "agentloop-v$Version-windows-x86_64.exe"
Invoke-WebRequest "https://github.com/dipeshbabu/agentloop/releases/download/v$Version/$Asset" -OutFile agentloop.exe
Invoke-WebRequest "https://github.com/dipeshbabu/agentloop/releases/download/v$Version/SHA256SUMS" -OutFile SHA256SUMS
$ChecksumLines = @(Get-Content SHA256SUMS | Where-Object {
    $_ -match "^[0-9a-fA-F]{64}  $([regex]::Escape($Asset))$"
})
if ($ChecksumLines.Count -ne 1) { throw "Expected exactly one checksum for $Asset" }
$ExpectedHash = ($ChecksumLines[0] -split "  ", 2)[0]
$ActualHash = (Get-FileHash .\agentloop.exe -Algorithm SHA256).Hash
if ($ActualHash -ne $ExpectedHash) { throw "Checksum verification failed for $Asset" }
.\agentloop.exe --help

Research use

For experiments, record one trace per task attempt and condition. Attach metadata such as task ID, seed, model, and source commit, and repeat comparisons across your task set. The research guide explains this workflow. From a source checkout, run the offline paired example with:

uv run --frozen --all-extras python examples/research_experiment_demo.py

Help and contributions

Run agentloop --help for all commands, or agentloop COMMAND --help for a specific command. Use agentloop doctor to inspect your installation and available integrations.

• Support: usage help and troubleshooting.
• Issue forms: bugs and feature requests.
• Contributing: development setup and required checks.
• Roadmap and changelog: project direction and release changes.
• Security: private vulnerability reports.
• Governance and Code of Conduct: project decisions and participation.

AgentLoop is under active pre-1.0 development. Use a tagged release when you need a fixed version, and check the changelog before upgrading. Maintainers can use the launch checklist when preparing a public release.

License

Copyright 2026 Dipesh Tharu Mahato.

AgentLoop is licensed under the Apache License 2.0. Dependencies retain their own terms; see THIRD_PARTY_LICENSES.md.