evosentinel 0.1.0

Advanced Python Runtime Guard & Self-Healing Engine

evoSentinel

Advanced Python Runtime Guard & Self-Healing Engine

Author: Daksha Dubey

"Failures are signals, not exceptions. evoSentinel reasons about them before acting."

Overview

evoSentinel is a production-grade Python SDK that provides multi-layer runtime defense and self-healing capabilities for mission-critical applications. Unlike simple retry libraries or circuit breakers, evoSentinel uses probabilistic decision-making, adaptive control systems, and behavioral modeling to protect your services from cascading failures.

This SDK is designed for senior-level system reliability engineering and reflects the kind of internal tooling used at companies like Google, Stripe, and Netflix.

---

Why evoSentinel?

This is NOT a Simple Retry Library

Traditional approaches treat failures as binary events:

• ❌ Fixed retry counts
• ❌ Static timeouts
• ❌ Binary circuit breaker states

evoSentinel is different:

• ✅ Continuous risk scoring (0.0 - 1.0)
• ✅ Probabilistic retries based on system health
• ✅ Adaptive quarantine with exponential decay
• ✅ Multi-layer defense with coordinated decision-making
• ✅ Self-healing through behavioral modeling

---

Architecture

Four-Layer Defense Stack

┌─────────────────────────────────────────────────────────┐
│  Layer 4: Decision & Control Plane                      │
│  • Converts risk → actions (ALLOW, THROTTLE, BLOCK)     │
│  • Hysteresis prevents flapping                         │
└─────────────────────────────────────────────────────────┘
                          ▲
┌─────────────────────────────────────────────────────────┐
│  Layer 3: Risk Scoring Engine                           │
│  • Computes continuous risk score (0.0 - 1.0)           │
│  • Factors: baseline deviation, failure momentum        │
└─────────────────────────────────────────────────────────┘
                          ▲
┌─────────────────────────────────────────────────────────┐
│  Layer 2: Behavioral Modeling                           │
│  • EWMA-based statistical profiling                     │
│  • Latency baselines, variance analysis                 │
│  • Adaptive decay for self-healing                      │
└─────────────────────────────────────────────────────────┘
                          ▲
┌─────────────────────────────────────────────────────────┐
│  Layer 1: Signal Capture                                │
│  • Execution time, exception types, call frequency      │
│  • Pure observation (no decision logic)                 │
└─────────────────────────────────────────────────────────┘

Health State Machine

HEALTHY → DEGRADED → CRITICAL → QUARANTINED
   ↑                                 ↓
   └─────────── (recovery) ──────────┘

Transitions are atomic, auditable, and reversible.

---

Installation

pip install evosentinel

Or install from source:

git clone https://github.com/yourusername/evoSentinel.git
cd evoSentinel
pip install -e .

---

Quick Start

Basic Usage

from evosentinel import sentinel

@sentinel.guard("payment.charge")
def charge_card(amount):
    # Your critical business logic
    return payment_gateway.charge(amount)

# evoSentinel will:
# 1. Evaluate risk before execution
# 2. Block if risk is too high
# 3. Retry probabilistically on failure
# 4. Quarantine if failures persist
# 5. Self-heal when stability returns

Custom Configuration

from evosentinel import Sentinel

sentinel = Sentinel(
    observation_window=120,      # Time window for metrics
    risk_decay=0.92,             # Decay rate for risk momentum
    max_risk=0.85,               # Maximum acceptable risk
    quarantine_threshold=0.95,   # Risk level triggering quarantine
    recovery_confidence=0.75     # Confidence needed for recovery
)

@sentinel.guard("critical.operation")
def critical_operation():
    # Protected execution
    pass

Observability Hooks

@sentinel.on_risk_change
def handle_risk(func_id, risk_score):
    logger.info(f"{func_id} risk: {risk_score:.4f}")

@sentinel.on_state_transition
def handle_transition(func_id, old_state, new_state):
    metrics.gauge(f"{func_id}.state", new_state.value)

@sentinel.on_quarantine
def handle_quarantine(func_id):
    alerts.send(f"Function {func_id} quarantined!")

@sentinel.on_recovery
def handle_recovery(func_id):
    alerts.send(f"Function {func_id} recovered!")

---

Core Algorithms

1. Exponentially Weighted Moving Average (EWMA)

Used for tracking latency and failure rate baselines:

V_new = α × V_current + (1 - α) × V_previous

2. Risk Momentum Calculation

Tracks recent failure intensity with exponential decay:

V(t) = V₀ × (decay_rate ^ elapsed_time)

3. Probabilistic Retry Decision

P(retry) = (1 - risk_score)²

Higher risk = lower retry probability. Infinite retry loops are mathematically impossible.

4. Adaptive Backoff

backoff = base_backoff × (1 / (1 - risk_score)) × jitter

Backoff increases with risk, preventing thundering herd.

---

Error Handling

evoSentinel uses typed exceptions that never mask user errors:

from evosentinel.errors import (
    SentinelBlockedError,      # Execution blocked due to high risk
    SentinelQuarantinedError,  # Function is quarantined
    SentinelOverloadError      # System overload detected
)

try:
    result = protected_function()
except SentinelBlockedError:
    # evoSentinel prevented execution
    return fallback_response()
except Exception as e:
    # Your application error (not masked)
    handle_error(e)

---

Concurrency Safety

evoSentinel is fully thread-safe and supports:

• ✅ threading
• ✅ asyncio
• ✅ concurrent.futures

import asyncio
from evosentinel import sentinel

@sentinel.guard("async.operation")
async def async_operation():
    await external_api_call()
    return "success"

# Works seamlessly with async/await
await async_operation()

---

Performance

• O(1) execution path - constant time overhead
• Constant memory per guarded function
• No unbounded data structures
• Zero external dependencies for core functionality

---

Production Tuning Guide

High-Throughput Services

sentinel = Sentinel(
    risk_decay=0.98,           # Slower decay for stability
    max_risk=0.90,             # Higher tolerance
    quarantine_threshold=0.98  # Quarantine only severe cases
)

Latency-Sensitive Services

sentinel = Sentinel(
    risk_decay=0.85,           # Faster decay
    max_risk=0.70,             # Lower tolerance
    quarantine_threshold=0.85  # Quarantine earlier
)

Financial/Critical Systems

sentinel = Sentinel(
    risk_decay=0.95,           # Balanced decay
    max_risk=0.75,             # Conservative threshold
    quarantine_threshold=0.90, # Aggressive quarantine
    recovery_confidence=0.85   # High confidence for recovery
)

---

Design Philosophy

1. Failures as Signals

Traditional systems treat failures as discrete events. evoSentinel models them as continuous signals that evolve over time.

2. Probabilistic Over Deterministic

Instead of "retry 3 times," evoSentinel asks: "Given the current system state, what's the probability this retry will succeed?"

3. Adaptive Control

Static thresholds break under changing conditions. evoSentinel adapts to your system's baseline behavior.

4. Defense in Depth

Multiple independent layers provide redundancy. If one layer fails, others compensate.

---

Comparison

Feature	evoSentinel	Circuit Breaker	Retry Library
Risk Modeling	✅ Continuous (0.0-1.0)	❌ Binary (open/closed)	❌ None
Adaptive Behavior	✅ Yes	⚠️ Limited	❌ No
Probabilistic Retries	✅ Yes	❌ No	❌ Fixed count
Self-Healing	✅ Automatic	⚠️ Timer-based	❌ No
Behavioral Profiling	✅ EWMA baselines	❌ No	❌ No
Quarantine System	✅ Adaptive cooldown	⚠️ Fixed timeout	❌ No

---

Demo

Run the included demo to see self-healing in action:

python demo.py

You'll see:

1. Quarantine when failures spike
2. Risk decay over time
3. State transitions (HEALTHY → DEGRADED → CRITICAL → QUARANTINED)
4. Recovery when stability returns

---

Contributing

Contributions are welcome! Please ensure:

• Code passes all tests
• Algorithms are documented
• Performance characteristics are maintained (O(1) execution)
• No external dependencies added to core

---

License

MIT License - see LICENSE file for details.

---

Acknowledgments

Inspired by production reliability systems at:

• Google's SRE practices
• Netflix's Hystrix (but evolved beyond circuit breakers)
• Stripe's internal fault tolerance tooling

---

Built with ❤️ for production reliability engineers who understand that failure is not an exception—it's a signal.