chaosrank-cli 1.0.0

Risk-driven chaos experiment scheduler — rank which microservice to chaos-test next

ChaosRank (Public SDK)

Chaos engineering requires a hypothesis. ChaosRank tells you where to point it.

ChaosRank analyzes your service dependency graph and incident history to rank which service to target next.

This is the Open Core SDK. It works by collecting your system data and sending it to the ChaosRank Engine for secure, high-performance scoring.

ChaosRank (Open Core) analyzes your service dependency graph and incident history to rank which service to target next.

Note: This is the Public SDK. It requires the ChaosRank Engine (private) to perform scoring operations.

---

Open Core Architecture

ChaosRank is split into two components to protect core mathematical IP while allowing public development of adapters:

1. Public SDK (This repo): CLI, trace parsing, and incident collection.
2. Private Engine: Core scoring algorithms (Blast Radius, Fragility, Adaptive).

Choose Your Hosting Model:

• SaaS (Community): Point your SDK to a managed ChaosRank Engine URL.
• Self-Hosted (Enterprise): Run the engine Docker container in your own infrastructure.

---

Results

Evaluated on the DeathStarBench social-network topology (31 services) from the UIUC/FIRM dataset (OSDI 2020). ChaosRank's engine identifies high-risk services by combining structural importance (traces) and history (incidents).

Metric	ChaosRank	Random	Improvement
Mean experiments to first weakness	1.0	9.8	9.8x
Mean experiments to all weaknesses	3.0	23.2	7.8x

ChaosRank found all 3 weaknesses in exactly 3 experiments across all 20 trials. Random selection needed 23.2 experiments on average.

Methodology note: Service topology and incident data are derived from the UIUC/FIRM DeathStarBench dataset (CC0 license). The topology reflects real microservice call graphs. Incident data was extracted by comparing per-service latency in anomaly-injected trace files against the no-interference baseline (~7x degradation → critical severity). This is a simulation benchmark — ChaosRank does not inject faults itself.

---

How It Works

ChaosRank uses a Client-Server model. The SDK (this repo) acts as the "Body," collecting traces and incidents from your environment. These are non-destructively summarized and sent to the ChaosRank Engine (The "Brain") for scoring.

traces.json  ──► [ SDK Parser ] ──► [ EngineClient ] ──► [ Hosted Engine ]
incidents.csv ──► [ SDK Parser ] ──► [ EngineClient ] ──► [ Risk Ranking ]

The engine provides deterministic, principled rankings based on:

• Blast Radius: Transitive impact of failure (Impact).
• Fragility: Load-normalized incident history (Likelihood).
• Adaptive Weights: Self-correcting risk factors based on experiment outcomes.

Blast Radius — Blended Centrality

blast_radius(v) = 0.5 * pagerank(v, G) + 0.5 * in_degree_centrality(v, G)

Built from your Jaeger trace data. PageRank captures transitive influence (how far does failure propagate?). In-degree centrality captures direct dependents (what breaks immediately?). Neither alone is sufficient — the blend surfaces both shallow-wide hubs and deep dependency chains.

Fragility Score — Four-Step Pipeline

1. Traffic-aware burst deduplication — collapses alert storms proportionally to traffic volume, preserving genuine failure cascades
2. Per-incident traffic normalization — each incident evaluated in its own traffic context, preventing high-traffic services from being unfairly penalized
3. Exponential decay — recent incidents weighted more heavily (lambda=0.10 → ~30-day effective memory)
4. Z-score normalization — outlier services score high without collapsing all others toward zero (MinMax rejected for this reason)

Severity weights use a log scale: critical=1.000, high=0.602, medium=0.301, low=0.100.

See docs/algorithm.md for the full mathematical derivation.

Fault Type Suggestion

Dominant Signal	Suggested Fault	Confidence
p99 latency spike	latency-injection	high if purity >70% and n ≥ 5
error rate breach	partial-response	high if purity >70% and n ≥ 5
timeout incident	connection-timeout	medium if purity >50% and n ≥ 3
no history	pod-failure	low (cold start default)
mixed/unclear	pod-failure	low (conservative default)

---

Installation

To ensure you have the latest features and to support our Open Core development model, ChaosRank is distributed via GitHub:

# 1. Clone the repository
git clone https://github.com/Medinz01/chaosrank
cd chaosrank

# 2. Install dependencies locally
pip install -e .

Configuration

ChaosRank requires an engine endpoint to function. Update your chaosrank.yaml:

engine:
  url: "https://api.chaosrank.com"  # Or your self-hosted URL
  api_key: "your-access-key"

Or use environment variables: export CHAOSRANK_API_KEY=your-key

Docker

docker compose build
docker compose run chaosrank

---

Usage

Basic ranking

chaosrank rank \
  --traces ./traces.json \
  --incidents ./incidents.csv

JSON output

chaosrank rank \
  --traces ./traces.json \
  --incidents ./incidents.csv \
  --output json

Pipe directly to LitmusChaos

chaosrank rank \
  --traces ./traces.json \
  --incidents ./incidents.csv \
  --output litmus | kubectl apply -f -

With async topology (Kafka, SQS, RabbitMQ)

# Step 1 — convert your async topology source
chaosrank convert --from kafka --input ./kafka-topics.json --dry-run
chaosrank convert --from kafka --input ./kafka-topics.json --output ./async-deps.yaml

# Step 2 — rank with async deps merged
chaosrank rank --traces ./traces.json --async-deps ./async-deps.yaml

Fetch incidents from alerting system

# From PagerDuty (no manual CSV needed)
chaosrank incidents --from pagerduty --token $PD_TOKEN --window 30d --output incidents.csv
chaosrank rank --traces ./traces.json --incidents incidents.csv

# From Alertmanager
chaosrank incidents --from alertmanager --url http://alertmanager:9093 --window 30d --output incidents.csv

# Dry-run to preview
chaosrank incidents --from pagerduty --token $PD_TOKEN --window 7d --dry-run

OTel OTLP traces

# OTel Collector JSON or Tempo/Jaeger v2 — auto-detected
chaosrank rank --traces ./otlp_traces.json --format otlp

Visualize the dependency graph

# Sync topology only
chaosrank graph --traces ./traces.json --output dot | dot -Tpng > graph.png

# With async edges (shown as dashed lines)
chaosrank graph --traces ./traces.json --async-deps ./async-deps.yaml --output dot | dot -Tpng > graph.png

---

Input Formats

Traces — Jaeger JSON

Standard Jaeger HTTP API export format. Export from your Jaeger instance:

curl "http://jaeger:16686/api/traces?service=frontend&limit=5000" > traces.json

ChaosRank streams files >100MB via ijson to avoid memory issues.

Incidents — CSV

timestamp,service,type,severity,request_volume
2026-02-10T08:00:00Z,payment-service,error,critical,9000
2026-02-14T15:00:00Z,productcatalog-service,latency,high,12000
2026-02-20T11:00:00Z,cart-service,error,medium,5000

Column	Required	Description
timestamp	Yes	ISO 8601
service	Yes	Service name (normalized automatically)
type	Yes	error, latency, timeout
severity	Yes	critical, high, medium, low
request_volume	No	Per-service request count at incident time. Falls back to window average, then skips normalization with warning.

---

Configuration

chaosrank.yaml (place in working directory or pass via --config):

weights:
  blast_radius: 0.6   # alpha — blast radius weight
  fragility: 0.4      # beta  — fragility weight

fragility:
  decay_lambda: 0.10          # recency decay (0.05=60d, 0.10=30d, 0.20=15d)
  burst_window_minutes: 5     # base alert dedup window

graph:
  min_call_frequency: 10      # filter noisy edges

output:
  top_n: 5

# Optional: service name aliases
aliases:
  payments: payment-service
  auth: authentication-service

Tuning alpha and beta

Scenario	Recommendation
New system, no incident history	Increase alpha (blast radius only)
Mature system with rich incident data	Decrease alpha, increase beta
Signal misalignment warning fires	Review — blast radius and fragility are disagreeing. Inspect both signals before tuning.

---

Service Name Normalization

OTel exporters often emit versioned or hashed service names. ChaosRank normalizes automatically:

payment-service-v2-7d9f8b  →  payment-service
payment-service-1.2.3      →  payment-service
Payment-Service-v2-abc12f  →  payment-service

Pipeline: lowercase → strip version patterns → strip pod hash suffixes → apply aliases.

---

Prior Art & Positioning

Tool	Experiment Selection	Gap
LitmusChaos	Manual, declarative CRDs	No ranking or guidance
Chaos Mesh	Manual workflow definition	No risk awareness
Gremlin	UI-driven, some "advice"	Closed source, not graph-based
Steadybit	Reliability hints (rule-based)	No dependency graph, no incidents
ChaosEater	LLM-driven hypotheses	Non-deterministic, not reproducible

ChaosRank does not claim novelty in any individual technique. The contribution is the combination of graph-theoretic blast radius scoring, per-incident traffic-normalized fragility scoring, and their application to chaos experiment prioritization. This combination is an open problem in OSS chaos engineering tooling.

---

Known Limitations

Limitation	Impact	Status
Async propagation semantics	Blast radius overestimated for async-heavy producers	async_weight_factor=0.5 default — configurable via --async-weight-factor
Async deps: source code	No C#/Java/Go event class parsers	See docs/async-deps-guide.md for manual extraction
OTel protobuf	JSON-encoded OTLP only	Protobuf support planned for v0.4
Single-region topology	Misses cross-region blast radius	Future work
Static alpha/beta	Optimal weights vary by system	Future: learned weights
Z-score less stable below 10 services	Directional scores only	Documented
Point-in-time request volume	Requires enriched incident CSV	Falls back gracefully

Async dependency support

ChaosRank builds its dependency graph from synchronous trace spans. Services that produce to Kafka topics, SQS queues, or other async channels do not appear as dependents in trace data without additional input. A Kafka producer with 10 downstream consumers would show zero blast radius from traces alone.

Use --async-deps to describe your async topology:

# From a Kafka topic export
chaosrank convert --from kafka --input ./kafka-topics.json --output ./async-deps.yaml
chaosrank rank --traces ./traces.json --async-deps ./async-deps.yaml

# From AsyncAPI 2.x specs
chaosrank convert --from asyncapi --input ./specs/ --output ./async-deps.yaml
chaosrank rank --traces ./traces.json --async-deps ./async-deps.yaml

Use --dry-run to verify extraction before it affects rankings. See docs/async-deps-guide.md for the manifest format and manual extraction guide.

---

What ChaosRank Is Not

• Does not inject faults → use LitmusChaos, Chaos Mesh, or Gremlin
• Does not derive steady-state → bring your own Prometheus thresholds
• Does not verify results → check your dashboards or Steadybit
• Does not need a running cluster → offline analysis on trace exports
• Does not support OTel protobuf → JSON-encoded OTLP supported; protobuf v0.4 roadmap
• Does not parse source code → see docs/async-deps-guide.md for manual topology extraction

---

Benchmark Reproduction

# Convert DeathStarBench traces to ChaosRank format
python benchmarks/convert_deathstar.py \
  --app social-network \
  --data-dir /path/to/tracing-data \
  --output benchmarks/real_traces/social_network.json

# Extract incidents from anomaly injection files
python benchmarks/extract_incidents.py \
  --app social-network \
  --data-dir /path/to/tracing-data \
  --output benchmarks/real_traces/social_network_incidents.csv

# Run 20-trial comparison
python benchmarks/run_comparison.py

# Generate chart
python benchmarks/plot_results.py

Dataset: Qiu et al., FIRM: An Intelligent Fine-grained Resource Management Framework for SLO-oriented Microservices, OSDI 2020. DOI: 10.13012/B2IDB-6738796_V1 — CC0 license.

---

Development

# Run tests
pytest tests/ -v

# Lint
ruff check chaosrank/

# Run with verbose logging
chaosrank rank --traces traces.json --incidents incidents.csv --verbose

Test coverage

Suite	Tests	What it validates
test_parser.py	53	Normalization round-trip, incident parsing, Jaeger edge extraction
test_fragility.py	21	Burst dedup, per-incident normalization, fragility preservation, z-score, decay
test_blast_radius.py	15	Callee model, chain ordering, blend weights, graph reversal
test_ranker.py	18	Risk math, cold start, combined signal, fault suggestion
test_async_deps.py	19	Manifest parser: edge merging, weight assignment, normalization, conflict handling
test_adapters.py	36	AsyncAPI adapter (edge/topic/binding extraction), Kafka adapter (edge extraction, malformed input)
test_parser_otlp.py	39	OTel Collector JSON, Tempo/Jaeger v2, auto-detection, service name normalization
test_incident_adapters.py	48	PagerDuty, Alertmanager, Grafana OnCall, Opsgenie — all HTTP mocked

The fragility preservation test is load-bearing for the benchmark: it asserts that a medium-traffic service with disproportionately high incident rate ranks above a high-traffic service with proportional incidents — the case that post-hoc normalization gets wrong.

---

Repository Structure

chaosrank/
├── chaosrank/
│   ├── cli.py                    # Typer entrypoint: rank, graph, convert
│   ├── adapters/
│   │   ├── base.py               # AsyncDepsAdapter ABC
│   │   ├── asyncapi.py           # AsyncAPI 2.x → async-deps.yaml
│   │   └── kafka.py              # Kafka topic export → async-deps.yaml
│   ├── incident_adapters/
│   │   ├── base.py               # IncidentAdapter ABC
│   │   ├── pagerduty.py          # PagerDuty REST API v2
│   │   ├── alertmanager.py       # Prometheus Alertmanager API
│   │   ├── grafana_oncall.py     # Grafana OnCall API
│   │   ├── opsgenie.py           # Opsgenie API
│   │   └── csv_export.py         # list[Incident] → ChaosRank CSV
│   ├── parser/
│   │   ├── normalize.py          # Service name normalization
│   │   ├── jaeger.py             # Jaeger JSON trace parser
│   │   ├── otlp.py               # OTel OTLP trace parser (Collector + Tempo)
│   │   ├── incidents.py          # Incident CSV parser
│   │   └── async_deps.py         # async-deps.yaml → graph edges
│   ├── graph/
│   │   ├── builder.py            # NetworkX DiGraph construction
│   │   ├── blast_radius.py       # Blended centrality scoring
│   │   └── visualize.py          # DOT/Graphviz export
│   ├── scorer/
│   │   ├── fragility.py          # Four-step fragility pipeline
│   │   ├── ranker.py             # Risk score combination
│   │   └── suggest.py            # Fault type suggestion
│   └── output/
│       ├── table.py              # Rich table renderer
│       ├── json_out.py           # JSON output with reasoning
│       └── litmus.py             # LitmusChaos ChaosEngine manifest
├── tests/                        # 117 tests
├── docs/
│   ├── OPEN_CORE.md              # Beginner's guide to the architecture
│   ├── algorithm.md              # Full mathematical derivation
│   ├── architecture.md           # Component map and data flow
│   └── future-work.md            # v0.2 roadmap
├── chaosrank.yaml                # Default configuration
├── pyproject.toml
└── Dockerfile

---

Contributing

See CONTRIBUTING.md for setup, testing, and PR guidelines.

Documentation

• docs/algorithm.md — full mathematical derivation
• docs/architecture.md — component map, data flow, ingestion layer design
• docs/async-deps-guide.md — async manifest format and manual extraction guide
• docs/future-work.md — roadmap
• benchmarks/sensitivity/ — hyperparameter stability analysis

Changelog

See CHANGELOG.md for version history.

License

Apache 2.0 — see LICENSE for full text.