ai-trace-auditor 0.5.0

Audit LLM traces against regulatory compliance requirements (EU AI Act, NIST AI RMF)

AI Trace Auditor

Audit LLM traces against regulatory compliance requirements. Open-source CLI that sits between your observability stack (Langfuse, Arize, OTel) and regulatory frameworks (EU AI Act, NIST AI RMF).

Your observability tools collect traces. Your GRC platform manages policies. Nothing translates traces into compliance evidence. This tool does.

Install

pip install ai-trace-auditor

Or from source:

git clone https://github.com/BipinRimal314/ai-trace-auditor.git
cd ai-trace-auditor
pip install -e .

Quick Start

# Audit traces against all regulations
aitrace audit traces.json

# Audit against a specific regulation
aitrace audit traces.json -r "EU AI Act" -o report.md

# Audit your Claude Code conversation traces
aitrace audit ~/.claude/projects/*/session-id.jsonl

# Inspect what requirements exist
aitrace requirements --show EU-AIA-12.1

# Just ingest and summarize traces
aitrace ingest traces.json --summary

GitHub Action

Add compliance checks to your CI pipeline:

- name: Audit AI traces
  uses: BipinRimal314/ai-trace-auditor@v0.2.0
  with:
    path: traces/exported.json
    regulation: "EU AI Act"
    output: compliance-report.md
    fail-on-gaps: "true"

The action fails if compliance gaps are found. Set fail-on-gaps: "false" to report without blocking.

What It Checks

EU AI Act Article 12 (Record-Keeping):

• Event timestamps, operation identification
• Risk situation logging (errors, failure modes)
• Model version tracking for post-market monitoring
• Resource consumption (tokens, latency)
• Content recording (opt-in)
• Tool/function call audit trails
• Trace linkage for multi-step operations

NIST AI RMF:

• Production monitoring (MEASURE 2.4)
• Transparency documentation (MEASURE 2.8)
• Model explainability (MEASURE 2.9)
• Risk tracking (MEASURE 3.1)
• Post-deployment monitoring (MANAGE 4.1)
• Incident communication (MANAGE 4.3)

Supported Trace Formats

Format	Source
OTel OTLP JSON	OpenTelemetry GenAI semantic conventions
Langfuse JSON	Langfuse trace exports
Claude Code	~/.claude/projects/ conversation traces
Raw JSONL	Any provider's API logs

Auto-detected. Use --format to override.

Example Output

Real output from auditing 1,522 Claude Code spans:

Overall Compliance Score: 79.3%

| Status    | Count |
|-----------|-------|
| Satisfied |    10 |
| Partial   |     5 |
| Missing   |     3 |

Top gaps:
  1. Not logging: Temperature parameter controlling output randomness
  2. Not logging: Maximum token limit for output generation
  3. Incomplete: Output responses generated by the AI model (31.9% coverage)
  4. Incomplete: Input prompts/messages (4.7% coverage)
  5. Not logging: Operation latency in milliseconds

CI Integration

Exit code 0 = all satisfied, 1 = gaps found:

aitrace audit traces.json -r "EU AI Act" || echo "Compliance gaps detected"

Library API

Use programmatically in your own tools:

from ai_trace_auditor.ingest import ingest_file
from ai_trace_auditor.analysis.engine import ComplianceAnalyzer
from ai_trace_auditor.regulations.registry import RequirementRegistry

traces = ingest_file(Path("traces.json"))

registry = RequirementRegistry()
registry.load()

report = ComplianceAnalyzer(registry).analyze(
    traces=traces,
    regulations=["EU AI Act"],
)

print(f"Score: {report.overall_score:.1%}")
for result in report.requirement_results:
    if result.gaps:
        print(f"  {result.requirement.id}: {result.gaps[0].recommendation}")

Disclaimer

This tool provides automated compliance assessments based on its interpretation of regulatory requirements. It is not legal advice. Consult qualified legal counsel for compliance decisions.

License

Apache 2.0