AI Trace Auditor

The EU AI Act takes effect August 2, 2026. Your AI system needs compliance evidence. Your observability tools collect traces. Your GRC platform manages policies. Nothing translates traces into compliance evidence. This tool does.

Open-source CLI that audits LLM traces against EU AI Act Articles 11, 12, 13, and 25, plus NIST AI RMF and GDPR Article 30. 301 tests. Zero LLM dependencies. Runs locally.

The Problem

Companies have traces from Langfuse, Arize, or OpenTelemetry, but no automated way to answer: "Do our traces satisfy the regulatory requirements?" 67% of AI teams discover quality regressions from user complaints despite having tracing infrastructure. The gap isn't data collection; it's interpretation.

Consultants charge $30K-$500K per compliance engagement. CEPS estimates EUR 29K per AI model for Annex IV documentation alone. This tool automates the gap analysis for free.

What It Covers

Multi-Agent Compliance (v0.14.0)

Automatically audits multi-agent systems (LangGraph, CrewAI, AutoGen, Google ADK):

• Reconstructs execution DAGs from parent-child span relationships
• Per-agent compliance scores with bottom-up penalty propagation
• Article 25 "value chain accountability" checks
• Liability shift detection (deployer becoming provider)
• Mermaid DAG visualizations

No other open-source tool does multi-agent compliance auditing.

Supported Trace Formats

Auto-detected. Use --format to override.

Install

pip install ai-trace-auditor

Quick Start

# Audit traces against EU AI Act
aitrace audit traces.json -r "EU AI Act" -o report.md

# Full compliance package: Articles 11 + 12 + 13 + GDPR in one run
aitrace scan ./my-ai-project/ --traces traces.json

# Generate Annex IV technical documentation from code
aitrace docs ./my-ai-project/

# Map data flows for Article 13 + GDPR Article 30
aitrace flow ./my-ai-project/

# Multi-agent audit with DAG visualization
aitrace audit multi_agent_traces.json --show-dag

Audit a GitHub repository

aitrace audit-repo https://github.com/owner/repo

The auditor clones the repo (shallowly, 50MB cap, 30s timeout), discovers trace artifacts (OTEL JSON, Langfuse exports, chat JSONL), and runs the governance-doc manifest (MODEL_CARD.md, retention configs, Article 50 disclosure language, ISO/SOC 2 policy docs). The web dashboard also accepts a repo URL at /audit.

Example Output

Real output from auditing 1,522 Claude Code spans:

Overall Compliance Score: 79.3%

| Status    | Count |
|-----------|-------|
| Satisfied |    10 |
| Partial   |     5 |
| Missing   |     3 |

Top gaps:
  1. Not logging: Temperature parameter controlling output randomness
  2. Not logging: Maximum token limit for output generation
  3. Incomplete: Output responses generated by the AI model (31.9% coverage)
  4. Incomplete: Input prompts/messages (4.7% coverage)
  5. Not logging: Operation latency in milliseconds

GitHub Action

- name: Audit AI traces
  uses: BipinRimal314/ai-trace-auditor@v0.14.0
  with:
    path: traces/exported.json
    regulation: "EU AI Act"
    output: compliance-report.md
    fail-on-gaps: "true"

Exit code 0 = all requirements satisfied, 1 = gaps found. CI-friendly.

Library API

from ai_trace_auditor.ingest import ingest_file
from ai_trace_auditor.analysis.engine import ComplianceAnalyzer
from ai_trace_auditor.regulations.registry import RequirementRegistry

traces = ingest_file(Path("traces.json"))
registry = RequirementRegistry()
registry.load()

report = ComplianceAnalyzer(registry).analyze(
    traces=traces,
    regulations=["EU AI Act"],
)

print(f"Score: {report.overall_score:.1%}")

Architecture

ai-trace-auditor/
├── src/ai_trace_auditor/
│   ├── cli.py              # 7 commands: audit, docs, flow, scan, ingest, requirements, health
│   ├── ingest/             # Trace ingestion (OTel, Langfuse, Claude Code, raw JSONL)
│   ├── analysis/           # Gap analysis engine + multi-agent DAG auditing
│   ├── models/             # Pydantic v2 data models
│   ├── regulations/        # YAML requirement definitions (extensible)
│   ├── docs/               # Article 11 Annex IV generator
│   ├── flow/               # Article 13 data flow mapper + GDPR RoPA
│   ├── scan/               # Full compliance package runner
│   ├── evidence/           # Auditor-ready evidence pack generator
│   ├── reports/            # Markdown, JSON, PDF report generation
│   ├── scanner/            # Code scanner (AI framework detection)
│   └── guide_linter/       # Lints compliance guides for accuracy
├── requirements/           # YAML regulatory requirement definitions
│   ├── eu_ai_act/          # Articles 12, 19
│   └── nist_ai_rmf/       # GOVERN, MAP, MEASURE, MANAGE
└── tests/                  # 301 tests

No dependency on any LLM framework. Intentionally framework-agnostic.

Competitive Landscape

The competition is consultants, not software.

Disclaimer

This tool provides automated compliance analysis. It is not legal advice. Risk classification under Annex III requires legal review. Consult qualified counsel for compliance decisions.

License

Apache 2.0