QAgent Tracer

Lightweight tool execution tracing SDK for evaluating AI agents on QAgent.

qagent-tracer captures, timestamps, isolates, and redacts tool executions within your AI agent backend so the QAgent platform can rigorously evaluate tool calling accuracy, parameter fidelity, hallucination, and task completion.

---

Features

• Zero third-party dependencies (built entirely on Python standard library: contextvars, functools, inspect, re, json, time).
• Thread-safe and asynchronous-safe request isolation via contextvars.
• Automatic PII and credential redaction (API keys, Bearer tokens, passwords, credit cards, emails, phone numbers).
• Safe payload truncation for oversized tool returns (prevents memory spikes without breaking agent code).
• Resilient serialization (handles non-JSON serializable objects like datetime or database models safely without crashing).
• Negligible runtime overhead (< 0.1ms per call).

---

Installation

pip install qagent-tracer

Requirements: Python 3.8 or higher.

---

Quickstart

1. Decorate Your Tools

Wrap any tool or helper function with @trace_tool:

from qagent_tracer import trace_tool
# (or: from qagent_sdk import trace_tool)

@trace_tool
def check_account_status(user_email: str) -> dict:
    # Your database query or external API call
    return {
        "plan": "Solo",
        "monthly_cost": 15,
        "runs_used": 84,
        "runs_total": 200,
        "active": True
    }

Async functions are fully supported:

@trace_tool
async def fetch_user_orders(user_id: str) -> list:
    # Asynchronous external call
    return [{"order_id": "ORD-101", "status": "shipped"}]

2. Return Traces in Your Webhook Response

At the start of your request handler, initialize clean storage with clear_traces(). After your agent generates its answer, retrieve the captured traces with get_traces() and include them under "traces" in your JSON response:

from fastapi import FastAPI
from pydantic import BaseModel
from typing import List, Dict, Any, Optional
from qagent_sdk import clear_traces, get_traces

app = FastAPI()

class ChatRequest(BaseModel):
    messages: List[Dict[str, str]]
    user_id: Optional[str] = None

@app.post("/chat")
async def chat(request: ChatRequest):
    # 1. Reset trace context for this incoming request
    clear_traces()

    # 2. Run your agent logic (LLM invocation + tool execution)
    user_message = request.messages[-1]["content"]
    agent_reply = await run_my_agent(user_message)

    # 3. Retrieve captured tool traces (automatically redacted by default)
    traces = get_traces()

    # 4. Return reply alongside traces
    return {
        "reply": agent_reply,
        "traces": traces
    }

---

Complete FastAPI Example

Here is a complete, runnable agent webhook:

from fastapi import FastAPI
from pydantic import BaseModel
from typing import List, Dict, Any
from qagent_sdk import trace_tool, clear_traces, get_traces, set_current_tool_call_id

app = FastAPI(title="Customer Support Agent")

# Define tools
@trace_tool
def check_account_status(user_email: str) -> dict:
    return {
        "email": user_email,
        "plan": "Solo",
        "runs_used": 84,
        "runs_total": 200,
        "active": True
    }

@trace_tool
def escalate_to_human(user_email: str, reason: str, priority: str = "normal") -> dict:
    return {
        "ticket_id": "TICK-9081",
        "status": "queued",
        "priority": priority
    }

class Message(BaseModel):
    role: str
    content: str

class ChatPayload(BaseModel):
    messages: List[Message]

@app.post("/chat")
async def chat_handler(payload: ChatPayload):
    clear_traces()

    last_user_message = payload.messages[-1].content

    # Example: When the LLM requests a tool call with an ID
    tool_call_id = "call_abc123"
    set_current_tool_call_id(tool_call_id)

    # Execute the tool
    result = check_account_status(user_email="alex@startup.io")

    # Generate response
    reply = (
        f"Your account is on the {result['plan']} plan. "
        f"You have used {result['runs_used']} of {result['runs_total']} test runs."
    )

    return {
        "reply": reply,
        "traces": get_traces()
    }

---

Flask Integration

from flask import Flask, request, jsonify
from qagent_sdk import trace_tool, clear_traces, get_traces

app = Flask(__name__)

@trace_tool
def query_knowledge_base(query: str) -> dict:
    return {"results": ["Sample doc content"]}

@app.route("/chat", methods=["POST"])
def chat():
    clear_traces()
    data = request.get_json()

    # Run agent execution
    reply = "Found relevant docs."
    query_knowledge_base(data["messages"][-1]["content"])

    return jsonify({
        "reply": reply,
        "traces": get_traces()
    })

---

Tool Call ID Correlation

When using OpenAI, Anthropic, or LangChain agents, the model assigns a unique tool_call_id (e.g., "call_98x...""). To link the physical tool execution to the model's intent:

from qagent_sdk import set_current_tool_call_id, trace_tool

# Set before calling the tool function
set_current_tool_call_id(tool_call.id)
result = my_tool(**tool_call.arguments)

The resulting trace will record:

• tool_call_id: The matching ID string.
• correlation_confidence: "HIGH".

If no ID is set, the SDK automatically generates a local identifier with "LOW" correlation confidence.

---

Security & Sensitive Data Redaction

By default, get_traces(redact=True) automatically sanitizes sensitive fields before returning them:

• API Keys (sk-..., gsk_..., Anthropic, AWS tokens)
• Bearer Authorization Headers
• Passwords, secret tokens, and credentials
• Credit card numbers (Luhn 13-19 digit patterns)
• Email addresses and phone numbers

Identifiers like order_id, user_id, ticket_id, and customer_id remain intact so evaluation rubrics can verify parameter values.

To retrieve unredacted traces internally (e.g., for custom debugging):

from qagent_sdk import get_raw_traces

raw = get_raw_traces()

---

Trace JSON Schema

The traces emitted by get_traces() adhere to this structure:

[
  {
    "tool_call_id": "call_abc123",
    "tool_name": "check_account_status",
    "input": {
      "user_email": "alex@startup.io"
    },
    "output": {
      "plan": "Solo",
      "runs_used": 84,
      "runs_total": 200,
      "active": true
    },
    "status": "success",
    "error": null,
    "duration_ms": 14.2,
    "timestamp": 1727725482.102,
    "correlation_confidence": "HIGH"
  }
]

---

License

MIT License. See LICENSE for details.