QAgent Tracer

Lightweight tool execution tracing SDK for evaluating AI agents on QAgent.

qagent-tracer captures, timestamps, isolates, and redacts tool executions within your AI agent backend so the QAgent platform can rigorously evaluate tool calling accuracy, parameter fidelity, hallucination, and task completion.

---

Features

• Zero third-party dependencies (built entirely on Python standard library: contextvars, functools, inspect, re, json, time).
• Thread-safe and asynchronous-safe request isolation via contextvars.
• Automatic PII and credential redaction (API keys, Bearer tokens, passwords, credit cards, emails, phone numbers).
• Safe payload truncation for oversized tool returns (prevents memory spikes without breaking agent code).
• Resilient serialization (handles non-JSON serializable objects like datetime or database models safely without crashing).
• Negligible runtime overhead (< 0.1ms per call).

---

Installation

pip install qagent-tracer

Requirements: Python 3.8 or higher.

---

Quickstart

1. Decorate Your Tools

Wrap any tool or helper function with @trace_tool:

from qagent_tracer import trace_tool
# (or: from qagent_sdk import trace_tool)

@trace_tool
def check_account_status(user_email: str) -> dict:
    # Your database query or external API call
    return {
        "plan": "Solo",
        "monthly_cost": 15,
        "runs_used": 84,
        "runs_total": 200,
        "active": True
    }

Async functions are fully supported:

@trace_tool
async def fetch_user_orders(user_id: str) -> list:
    # Asynchronous external call
    return [{"order_id": "ORD-101", "status": "shipped"}]

2. Return Traces in Your Webhook Response

At the start of your request handler, initialize clean storage with clear_traces(). After your agent generates its answer, retrieve the captured traces with get_traces() and include them under "traces" in your JSON response:

from fastapi import FastAPI
from pydantic import BaseModel
from typing import List, Dict, Any, Optional
from qagent_sdk import clear_traces, get_traces

app = FastAPI()

class ChatRequest(BaseModel):
    messages: List[Dict[str, str]]
    user_id: Optional[str] = None

@app.post("/chat")
async def chat(request: ChatRequest):
    # 1. Reset trace context for this incoming request
    clear_traces()

    # 2. Run your agent logic (LLM invocation + tool execution)
    user_message = request.messages[-1]["content"]
    agent_reply = await run_my_agent(user_message)

    # 3. Retrieve captured tool traces (automatically redacted by default)
    traces = get_traces()

    # 4. Return reply alongside traces
    return {
        "reply": agent_reply,
        "traces": traces
    }

---

Complete FastAPI Example

Here is a complete, runnable agent webhook:

from fastapi import FastAPI
from pydantic import BaseModel
from typing import List, Dict, Any
from qagent_sdk import trace_tool, clear_traces, get_traces, set_current_tool_call_id

app = FastAPI(title="Customer Support Agent")

# Define tools
@trace_tool
def check_account_status(user_email: str) -> dict:
    return {
        "email": user_email,
        "plan": "Solo",
        "runs_used": 84,
        "runs_total": 200,
        "active": True
    }

@trace_tool
def escalate_to_human(user_email: str, reason: str, priority: str = "normal") -> dict:
    return {
        "ticket_id": "TICK-9081",
        "status": "queued",
        "priority": priority
    }

class Message(BaseModel):
    role: str
    content: str

class ChatPayload(BaseModel):
    messages: List[Message]

@app.post("/chat")
async def chat_handler(payload: ChatPayload):
    clear_traces()

    last_user_message = payload.messages[-1].content

    # Example: When the LLM requests a tool call with an ID
    tool_call_id = "call_abc123"
    set_current_tool_call_id(tool_call_id)

    # Execute the tool
    result = check_account_status(user_email="alex@startup.io")

    # Generate response
    reply = (
        f"Your account is on the {result['plan']} plan. "
        f"You have used {result['runs_used']} of {result['runs_total']} test runs."
    )

    return {
        "reply": reply,
        "traces": get_traces()
    }

---

Flask Integration

from flask import Flask, request, jsonify
from qagent_sdk import trace_tool, clear_traces, get_traces

app = Flask(__name__)

@trace_tool
def query_knowledge_base(query: str) -> dict:
    return {"results": ["Sample doc content"]}

@app.route("/chat", methods=["POST"])
def chat():
    clear_traces()
    data = request.get_json()

    # Run agent execution
    reply = "Found relevant docs."
    query_knowledge_base(data["messages"][-1]["content"])

    return jsonify({
        "reply": reply,
        "traces": get_traces()
    })

---

Tool Call ID Correlation

When using OpenAI, Anthropic, or LangChain agents, the model assigns a unique tool_call_id (e.g., "call_98x...""). To link the physical tool execution to the model's intent:

from qagent_sdk import set_current_tool_call_id, trace_tool

# Set before calling the tool function
set_current_tool_call_id(tool_call.id)
result = my_tool(**tool_call.arguments)

The resulting trace will record:

• tool_call_id: The matching ID string.
• correlation_confidence: "HIGH".

If no ID is set, the SDK automatically generates a local identifier with "LOW" correlation confidence.

---

Security & Sensitive Data Redaction

By default, get_traces(redact=True) automatically sanitizes sensitive fields before returning them:

• API Keys (sk-..., gsk_..., Anthropic, AWS tokens)
• Bearer Authorization Headers
• Passwords, secret tokens, and credentials
• Credit card numbers (Luhn 13-19 digit patterns)
• Email addresses and phone numbers

Identifiers like order_id, user_id, ticket_id, and customer_id remain intact so evaluation rubrics can verify parameter values.

To retrieve unredacted traces internally (e.g., for custom debugging):

from qagent_sdk import get_raw_traces

raw = get_raw_traces()

---

Privacy & Masking

qagent-tracer supports three privacy and masking levels:

1. Default Mode (Strict Redaction)

• Behavior: All secrets, credentials, API keys, credit cards, emails, and phone numbers are completely masked (e.g. [REDACTED:email], [REDACTED:phone], [REDACTED:api_key]).
• Configuration: Default behavior. Zero configuration required.

2. Fingerprint Mode (Opt-in Verification)

• Behavior: Secrets and payment cards remain strictly masked ([REDACTED:api_key], [REDACTED:card_number], etc.). Identifiers (emails and phone numbers) are masked with a deterministic HMAC-SHA256 fingerprint:
  • [REDACTED:email:fp=<12 hex chars>]
  • [REDACTED:phone:fp=<12 hex chars>]
• Use Case: Allows the QAgent evaluation engine to verify whether the agent used the expected customer email or phone number across multi-turn interactions without ever exposing the real PII value.
• Configuration:
  • Environment variable: QAGENT_FINGERPRINT_KEY="your-secret-key"
  • Code:

    import qagent_sdk
    
    qagent_sdk.configure(fingerprint_key="your-secret-key")
    

3. Test Mode (Opt-in Staging)

• Behavior: For mock/staging test accounts only. Emails and phone numbers are left unmasked so developers can inspect raw values during testing. Critical secrets (API keys, passwords, bearer tokens, credit cards) remain strictly masked. Logs a one-time safety warning: "qagent-tracer test mode: emails and phone numbers are not masked. Do not use with real customer data."
• Configuration:
  • Environment variable: QAGENT_TEST_MODE=true
  • Code:

    import qagent_sdk
    
    qagent_sdk.configure(test_mode=True)
    

---

Trace JSON Schema

The traces emitted by get_traces() adhere to this structure:

[
  {
    "tool_call_id": "call_abc123",
    "tool_name": "check_account_status",
    "input": {
      "user_email": "alex@startup.io"
    },
    "output": {
      "plan": "Solo",
      "runs_used": 84,
      "runs_total": 200,
      "active": true
    },
    "status": "success",
    "error": null,
    "duration_ms": 14.2,
    "timestamp": 1727725482.102,
    "correlation_confidence": "HIGH"
  }
]

---

Changelog

0.2.0 (2026-10-08)

• Added opt-in HMAC-SHA256 fingerprinting for emails and phone numbers via QAGENT_FINGERPRINT_KEY and configure(fingerprint_key="...").
• Added opt-in test mode via QAGENT_TEST_MODE=true and configure(test_mode=True) with one-time safety warning logging.
• Added configure() exported from both qagent_sdk and qagent_tracer.
• Full backward compatibility: 0.1.0 behavior remains default when no new settings are provided.
• Preserved strict non-negotiable masking for all API keys, bearer tokens, passwords, secrets, credentials, CVVs, and credit cards across all modes.

---

License

MIT License. See LICENSE for details.