AI Runtime Profiler (airun)

"What exactly happened during this AI workload, and what did it cost?"

airun is a lightweight, local-first runtime profiler and economic measurement layer for AI agents, multi-step LLM pipelines, and tool-augmented workflows.

---

Key Features

• Zero-Friction Tracing: Instrument functions and async coroutines in seconds with @trace or with trace().
• Accurate Cost Engine: Automatic token-level cost attribution for OpenAI, Anthropic, Gemini, Mistral, and local GPU infrastructure.
• Concurrent Critical Path: Accurately computes critical-path latency across parallel tools using interval DAG scheduling.
• Automated Diagnostic Findings: Heuristic detection of cost concentration, retry storms, token context bloat, and over-provisioned models.
• Outcome-Based Economics: Explicit attribution of Cost per Successful Outcome vs Wasted Cost on failed/aborted executions.
• Regression Diffing (airun compare): Side-by-side run comparison to detect latency, token, cost, and quality regressions.
• The Evaluation Boundary: Attach evaluation scores (set_span_quality()) to prove cost optimizations preserve output quality.
• Privacy by Default: Automatic secret and API key redaction. Prompt and completion contents are never stored without explicit opt-in.
• OpenTelemetry Compatible: Export traces directly as OTLP-compliant JSON.
• Ultra-Low Overhead: Measured in-memory span overhead is $<20\mu\text{s}$ and total SQLite persistence overhead is $<1\text{ms}$ per workflow.

---

3-Minute Quickstart

1. Installation

pip install airun-profiler

2. Environment Health Check

airun doctor

3. Run the Instant Demo (Zero Setup)

Run an offline multi-step agent simulation without external API keys:

airun demo

Output:

+---------- AI Workflow Runtime Summary -----------+
| Trace ID        5664cdc8296e41d2ab0b1feae7f0d481  |
| Workflow Name   multi_agent_coordination_pipeline |
| Final Outcome   [OK] SUCCESS                      |
| Total Cost      $0.0310                           |
| Cost / Success  $0.0310                           |
| Total Duration  411.7ms                           |
| Critical Path   411.7ms                           |
| Total Tokens    12,170 (in: 10,900, out: 1,270)   |
| Model Calls     4                                 |
| External Calls  2                                 |
| Retries         0                                 |
| Failed Steps    0                                 |
+---------------------------------------------------+

+-------------------- Findings & Optimization Insights ---------------------+
| * [CRITICAL] 56% of total cost comes from step 'agent_researcher' ($0.017)|
| * [WARNING] Token bloat: prompt context grew 2.7x from planner to synthesis|
| * [INFO] Concurrent execution: parallel tools saved ~411ms sequential delay|
+---------------------------------------------------------------------------+

                               Top Cost Drivers                                
+-----------------------------------------------------------------------------+
| #   | Span Name        | Kind | Model / Target    | Cost (USD) | % Total | Duration | Tokens |
|-----+------------------+------+-------------------+------------+---------+----------+--------|
| 1   | agent_researcher | llm  | claude-3-5-sonnet |    $0.0174 |   55.8% |  140.1ms |  3,020 |
| 2   | agent_planner    | llm  | gpt-4o            |    $0.0054 |   21.4% |  100.2ms |  1,590 |
| 3   | agent_critic     | llm  | gemini-1.5-pro    |    $0.0049 |   19.5% |   90.4ms |  3,310 |
| 4   | agent_synth      | llm  | gpt-4o-mini       |    $0.0008 |    3.3% |   80.2ms |  4,250 |
+-----------------------------------------------------------------------------+

Execution Hierarchy:
Execution Trace
`-- [OK] [workflow] multi_agent_coordination_pipeline (411.7ms)
    +-- [OK] [agent_step] agent_planner_phase (100.3ms)
    |   `-- [OK] [llm] agent_planner (100.2ms, model: gpt-4o, 1590 tok, $0.0054)
    +-- [OK] [agent_step] agent_researcher_phase (140.3ms)
    |   `-- [OK] [llm] agent_researcher (140.1ms, model: claude-3-5-sonnet, 3020 tok, $0.0174)
    +-- [OK] [agent_step] agent_critic_phase (90.6ms)
    |   `-- [OK] [llm] agent_critic (90.4ms, model: gemini-1.5-pro, 3310 tok, $0.0049)
    `-- [OK] [agent_step] agent_synthesizer_phase (80.4ms)
        `-- [OK] [llm] agent_synthesizer (80.2ms, model: gpt-4o-mini, 4250 tok, $0.0008)

---

Instrumenting Your Code

1. Synchronous Functions

from airun import trace, set_span_tokens, SpanKind

@trace(kind=SpanKind.LLM, model="gpt-4o", provider="openai")
def agent_step(prompt: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}]
    )
    set_span_tokens(
        input_tokens=response.usage.prompt_tokens,
        output_tokens=response.usage.completion_tokens,
    )
    return response.choices[0].message.content

with trace("customer_support_pipeline", kind=SpanKind.WORKFLOW):
    result = agent_step("Summarize ticket #4821")

2. Asynchronous Coroutines & Parallel Tools

import asyncio
from airun import trace, SpanKind

async def run_agent():
    async with trace("travel_agent", kind=SpanKind.WORKFLOW):
        # Parallel tools executed concurrently
        async with trace("parallel_booking", kind=SpanKind.AGENT_STEP):
            flights, hotels = await asyncio.gather(
                fetch_flights(),
                fetch_hotels()
            )

3. Zero-Code-Change CLI Profiling & CI/CD Integration

Prefix any Python execution with airun run and capture trace IDs deterministically:

airun run --trace-id-file .airun/trace_id ./agents/my_agent.py
airun report $(cat .airun/trace_id)

---

Comparing Runs (airun compare)

Benchmark two architectures or model swaps side-by-side:

airun compare previous latest

                    Trace Comparison: 84081c95 vs 6f1fe728                    
+----------------------------------------------------------------------------+
| Metric         | Run A (Baseline) | Run B (Optimized)|       Delta (B - A) |
|----------------+------------------+------------------+---------------------|
| Total Duration |          425.0ms |          177.0ms |   -248.0ms (-58.4%) |
| Total Cost     |            $0.03 |          $0.0007 | $-0.027764 (-97.4%) |
| Tokens         |            8,100 |            3,290 |              -4,810 |
| Retries        |                2 |                0 |                  -2 |
| Failed Steps   |                0 |                0 |                   0 |
| Quality Score  |             0.94 |             0.95 |               +0.01 |
+----------------------------------------------------------------------------+

---

CLI Reference

Command	Description
airun doctor	Verify local workspace, database health, and micro-overhead
airun init	Initialize workspace configuration and trace storage
airun demo	Run built-in simulated agent workflow
airun run <script.py>	Execute Python script with active profiler and instant summary
airun report [latest|<id>]	View runtime summary, diagnostic findings, and execution tree
airun compare [previous|<id1>] [latest|<id2>]	Compare two runs and inspect economic/latency deltas
airun trace list	List stored execution traces
airun trace show [latest|<id>]	Inspect execution tree hierarchy
airun export [latest|<id>] --format [json|otel-json]	Export trace in raw JSON or OpenTelemetry format

---

AI Workload Laboratory

Explore representative runnable agent archetypes in examples/lab/:

python examples/lab/run_all.py

• chat_workload.py — Simple Chatbot
• rag_workload.py — RAG Retrieval & Synthesis
• tool_agent_workload.py — Tool Agent with Parallel Concurrency & Retries
• coding_agent_workload.py — Code Evaluation & Bug Fix Loop
• multi_agent_workload.py — Multi-Agent Coordination Pipeline
• comparison_workload.py — The Same Workload, Two Ways (Cost & Latency Regression Lab)

---

Documentation & Campaign Links

• Developer & AI Agent Guide: agent.md
• Milestone AIRUN-100 Charter: validation/AIRUN-100.md
• Campaign Outreach Playbook: validation/outreach-playbook.md
• Teardown Report Template: validation/teardown-template.md
• External Validation Kit:
  • Validation Invitation
  • Quickstart Checklist
  • Feedback Form
  • Friction Log
  • Pain Ranking Matrix

---

License

Apache 2.0 License. See LICENSE for details.