xray-observability 1.0.0

Passive observability layer for distributed AI runtime tracing

Passive Observability Layer for Distributed AI Runtime Tracing

---

What is X-RAY?

X-RAY is a Python library that brings structured observability to distributed systems, AI pipelines, and multi-agent runtimes. It records execution traces, preserves causal relationships, and gives you the tools to audit, replay, and diagnose complex executions — without modifying your application's behavior.

Traditional logging answers "What happened?" X-RAY answers:

"What happened, in what order, through which causal chain, and can this execution be reconstructed?"

---

Quick Start

pip install xray-observability

from aethon.xray import start_trace, start_span, SpanKind

with start_trace("my_operation") as trace:
    with start_span(SpanKind.INTERNAL, "step_1") as span:
        # Your code here
        span.end(status="ok")

That's it. Five seconds to start tracing. No agents, no daemons, no config files.

---

Why X-RAY?

	OpenTelemetry	X-RAY
Focus	Telemetry collection protocol	Execution understanding & reconstruction
Causal analysis	❌ Not built-in	✅ Logical clocks, depth tracking, DAG validation
Audit system	❌	✅ 5 integrity checks (orphans, duplicates, live-vs-disk, ...)
Replay engine	❌	✅ Ordered execution timeline reconstruction
Sanitizer	❌	✅ Scan, repair, cleanup orphan traces
Retention policy	❌	✅ Age/count/size policies with dry-run
Passive by design	❌ Exporter may affect perf	✅ Never modifies application behavior
Dependencies	Heavy (protobuf, gRPC, exporter SDK)	Zero — pure Python stdlib
Deployment	Collector, exporters, config files	Copy the folder or pip install
AI-native	❌ Designed for web requests	✅ Designed for multi-agent, async reasoning

When to use OpenTelemetry: You need standard APM, metrics export to Datadog/Prometheus, and full request tracing across HTTP boundaries.

When to use X-RAY: You need causal reconstruction, execution replay, integrity verification, and lightweight per-process observability — especially in AI/agentic systems where non-deterministic execution makes traditional tracing insufficient.

You can use both. X-RAY complements OpenTelemetry where causal depth and reconstruction matter.

---

Architecture

Application Runtime
        |
        v
+---------------------------+
|         X-RAY             |
|  Passive Observability    |
+---------------------------+
        |
        +-- Trace Collector
        |     start_trace() → span hierarchy → contextvars
        |
        +-- Span Manager
        |     SpanKind (internal/server/client/producer/consumer)
        |     Logical timestamps, causal depth, parent-child
        |
        +-- Trace Store
        |     In-memory + atomic JSON persistence
        |     Index rebuild on restart
        |
        +-- Audit Engine
        |     5 integrity checks: live↔disk, orphans, duplicates, ...
        |
        +-- Sanitizer
        |     Scan → Repair → Cleanup orphan → Quarantine
        |
        +-- Replay Engine
        |     Ordered span timeline → execution reconstruction
        |
        +-- Diagnostics API
        |     Health metrics: active/completed/interrupted traces
        |
        +-- Event System
        |     Structured lifecycle events (trace created, span completed, ...)
        |
        +-- Retention Policy
              Age / count / size limits → archive → quarantine cleanup

---

Core Capabilities

1. Distributed Trace Registration

Record execution traces with parent-child hierarchy, causal dependencies, execution depth, and logical timestamps.

Trace
 |
 +-- Span A
 |    |
 |    +-- Span B
 |    +-- Span C
 |
 +-- Span D (client call → external service)

2. Lifecycle Management

created → active → completed → persisted
            |
       interrupted → recovered

Detect incomplete or corrupted executions. Freeze/unfreeze/terminate trace operations.

3. Persistent Storage

Atomic JSON snapshot writes. Restore full state after process restart. Forensic data preservation.

4. Audit System

Five integrity checks: broken parent relationships, invalid lifecycle states, corrupted metadata, missing references, consistency violations.

5. Sanitizer

Scan, repair, or cleanup orphan/corrupted records. All operations support ?dry_run=true for safe preview.

6. Replay Engine

Ordered span timeline with lifecycle reconstruction and causal analysis.

10:01:01  Request created
10:01:02  Agent started
10:01:05  Database operation (DB-Call)
10:01:07  Response completed

7. Event Streaming

Real-time lifecycle events (SSE): trace created, span started/completed, recovery events, persistence events.

8. Retention Policy

Limit storage by age, count, or size. Dry-run preview before execution. Archive-before-delete protection.

9. HTTP Propagation

Five canonical headers for distributed tracing across services:

Header	Purpose
X-Xray-Trace-Id	Trace identifier
X-Xray-Span-Id	Current span identifier
X-Xray-Parent-Span-Id	Parent span identifier
X-Xray-Logical-Ts	Logical timestamp for causal ordering
X-Xray-Causal-Depth	Execution depth in the causal graph

---

API Overview

Once integrated into your FastAPI application:

Endpoint	Description
GET /health	System availability and diagnostics
GET /xray/traces	List all traces
GET /xray/traces/{id}	Trace details
POST /xray/traces/{id}/freeze	Freeze trace (prevent modification)
POST /xray/traces/{id}/unfreeze	Unfreeze trace
POST /xray/traces/{id}/terminate	Force-terminate a trace
GET /xray/replay/{id}	Execution reconstruction
GET /xray/audit/{id}	Integrity check for single trace
GET /xray/audit	Integrity check for all traces
GET /xray/diagnostics	System health metrics
GET /xray/events/stream	SSE lifecycle event stream
POST /xray/sanitizer/scan	Scan for issues
POST /xray/sanitizer/repair	Repair discovered issues
POST /xray/sanitizer/cleanup-orphan	Remove orphan records
POST /xray/retention/run	Apply retention policy

All mutation operations support ?dry_run=true for safe preview.

---

Integration Examples

1. FastAPI Middleware

from fastapi import FastAPI, Request
from aethon.xray.http_propagation import fastapi_extract_xray

app = FastAPI()

@app.middleware("http")
async def xray_middleware(request: Request, call_next):
    trace_id = fastapi_extract_xray(request)
    response = await call_next(request)
    response.headers["X-Trace-Id"] = trace_id
    return response

2. Custom Spans

from aethon.xray import start_trace, start_span, SpanKind

def process_request():
    trace = start_trace("process_request")
    span = start_span(SpanKind.PROVIDER_CALL, "llm_inference")
    try:
        result = llm.query(prompt)
        span.end(status="ok", metadata={"tokens": result.tokens})
    except Exception as e:
        span.end(status="error", metadata={"error": str(e)})
    trace.end(status="ok")

3. Diagnostics & Audit

from aethon.xray.trace_store import store
from aethon.xray.consistency_audit import run_all_audit_checks

# Health diagnostics
print(store.diagnostics())
# → {"active_traces": 3, "completed_traces": 42, "disk_usage_mb": 1.2}

# Integrity check
result = run_all_audit_checks("trace-123")
print(result.all_passed)
# → True

---

Advanced: Why Distributed AI Needs X-RAY

Modern AI systems break traditional APM assumptions:

User Query
    ↓
Orchestrator
    ↓
Agent A ─────┐
Agent B ─────┼─→ Reasoning Engine → Tool Exec → Memory → Response
Agent C ─────┘

• Non-deterministic execution paths
• Parallel agent branches with shared state
• Async tool calls with callbacks
• Multi-step reasoning with retries and fallbacks

X-RAY captures this complexity through causal spans — not just timing, but the logical dependency chain that led to each outcome.

See the whitepaper for the full architectural rationale (377 pages).

---

Testing

pip install xray-observability[dev]
python -m pytest tests/minimal_integration_test.py -v

Verified scenarios

• Trace lifecycle: create → activate → complete → persist
• Persistence: save to disk → restart recovery → restore state
• Integrity: audit validation, metadata consistency, orphan detection
• Three manual scenarios: normal flow, failure+fallback, parallel chaos

---

Documentation

Document	Description
System Overview	Architecture and design philosophy
API Specification	Full REST API reference
Data Model	Trace, span, event schemas
Operational Guarantees	What X-RAY guarantees and what it doesn't
Runbook	Operational procedures and troubleshooting
Installation Guide	Detailed integration walkthrough
Whitepaper	Full architectural rationale
Changelog	Release history

All documents available in English and Русский.

---

How to Contribute

We welcome contributions from the community.

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Ensure all tests pass: python -m pytest tests/ -v
5. Submit a pull request

See CONTRIBUTING.md for detailed guidelines.

---

What X-RAY Is NOT

Not this	Because
Decision engine	X-RAY does not make decisions, only observes
AI reasoning system	No inference, no ML, no optimization
Autonomous healer	No auto-remediation or execution control
Network mesh	No data-plane involvement
Workflow orchestrator	No execution orchestration
APM replacement	Complements APMs where causal depth matters

X-RAY transforms distributed execution from an opaque process into a measurable and reconstructable system. It does not control intelligence. It provides the infrastructure required to understand it.

---

License

MIT © 2026 X-RAY Contributors

---

_{You cannot reliably improve a system you cannot observe.}