Resilient Circuit

Overview

Resilient Circuit is a powerful resilience library designed to make your Python applications fault-tolerant and highly available. It's an integral component of the Highway Workflow Engine, providing essential failure handling capabilities for modern distributed systems.

This library implements the Circuit Breaker and Retry patterns, offering elegant solutions for handling failures in networked systems, external service calls, and unreliable dependencies.

For comprehensive documentation, visit our Read the Docs page.

Installation

pip install resilient_circuit

PostgreSQL Storage Support (Optional)

For shared state across multiple instances, you can use PostgreSQL as the storage backend:

pip install resilient_circuit[postgres]

That installs pure-Python psycopg, which talks to the system libpq. It needs libpq present (libpq5 on Debian/Ubuntu, libpq on Homebrew) but no compiler and no wheel, so it works on platforms where neither is available.

If you want the C speedups and your machine has libpq headers plus a compiler (pg_config on PATH):

pip install resilient_circuit[postgres-c]      # or the house-convention alias:
pip install resilient_circuit[c]               # same thing, matches auth[c] / stabilize[c]

psycopg[binary] is deliberately not offered by any extra. It bundles its own libpq, which bypasses the system TLS configuration — including the CA bundle and any client-certificate setup your server requires.

Or install the dependencies separately:

pip install psycopg python-dotenv

Features

• Circuit Breaker Pattern: Prevents cascading failures in distributed systems
• Retry Pattern: Automatically retries failed operations with configurable backoff
• Composable: Chain multiple policies together for sophisticated error handling
• Decorator Support: Clean, easy-to-read syntax with Python decorators
• Fine-grained Control: Configure failure thresholds, cooldown periods, and backoff strategies
• State Monitoring: Track breaker state and execution history
• Shared State Storage: Optional PostgreSQL backend for distributed applications

Quick Start

Basic Circuit Protector

from datetime import timedelta
from fractions import Fraction
from resilient_circuit import CircuitProtectorPolicy

# Create a circuit protector that trips after 3 failures
protector = CircuitProtectorPolicy(
    failure_limit=Fraction(3, 10),  # 3 out of 10 failures
    cooldown=timedelta(seconds=30),  # 30-second cooldown
)


@protector
def unreliable_service_call():
    # Your potentially failing external service call
    import random

    if random.random() < 0.7:
        raise Exception("Service temporarily unavailable")
    return "Success!"

Advanced Retry with Exponential Backoff

from datetime import timedelta
from resilient_circuit import RetryWithBackoffPolicy, ExponentialDelay

# Create an exponential backoff strategy
backoff = ExponentialDelay(
    min_delay=timedelta(seconds=1),
    max_delay=timedelta(seconds=10),
    factor=2,
    jitter=0.1,
)

# Apply retry policy with backoff
retry_policy = RetryWithBackoffPolicy(max_retries=3, backoff=backoff)


@retry_policy
def unreliable_database_operation():
    # Operation that might fail temporarily
    import random

    if random.random() < 0.5:
        raise ConnectionError("Database temporarily unavailable")
    return "Database operation completed"

Combining Circuit Protector and Retry

from resilient_circuit import SafetyNet, CircuitProtectorPolicy, RetryWithBackoffPolicy

# Combine both patterns using SafetyNet
safety_net = SafetyNet(
    policies=(
        RetryWithBackoffPolicy(max_retries=2),
        CircuitProtectorPolicy(failure_limit=Fraction(2, 5)),
    )
)


@safety_net
def resilient_external_api_call():
    # This will first retry, then circuit-protect if needed
    import requests

    response = requests.get("https://external-api.example.com/data")
    return response.json()

Detailed Examples

Circuit Protector Customization

from datetime import timedelta
from fractions import Fraction
from resilient_circuit import CircuitProtectorPolicy, CircuitState


def custom_exception_handler(exc):
    """Only handle specific exceptions"""
    return isinstance(exc, (ConnectionError, TimeoutError))


def status_change_handler(policy, old_status, new_status):
    """Handle status transitions"""
    print(f"Circuit protector changed status: {old_status.name} -> {new_status.name}")


# Fully customized circuit protector
custom_protector = CircuitProtectorPolicy(
    cooldown=timedelta(minutes=1),  # 1-minute cooldown
    failure_limit=Fraction(3, 10),  # Trip after 30% failure rate
    success_limit=Fraction(5, 5),  # Close after 5 consecutive successes
    should_handle=custom_exception_handler,  # Custom exception filter
    on_status_change=status_change_handler,  # Status change listener
)


@custom_protector
def monitored_service_call():
    # Your service call with enhanced monitoring
    pass

Complex Retry Scenarios

from resilient_circuit import RetryWithBackoffPolicy, FixedDelay

# Constant delay between retries
constant_backoff = FixedDelay(delay=timedelta(seconds=2))

retry_with_constant_backoff = RetryWithBackoffPolicy(
    max_retries=5,
    backoff=constant_backoff,
    should_handle=lambda e: isinstance(e, ConnectionError),
)


@retry_with_constant_backoff
def service_with_constant_retry():
    # This will retry every 2 seconds up to 5 times
    pass

Accessing Circuit Protector Status

from resilient_circuit import CircuitProtectorPolicy

protector = CircuitProtectorPolicy(failure_limit=Fraction(2, 5))


@protector
def service_call():
    pass


# Check protector status and execution log
print(f"Current status: {protector.status.name}")
print(f"Execution log: {list(protector.execution_log)}")

# The execution_log buffer maintains success/failure record
if protector.status == CircuitState.OPEN:
    print("Circuit protector is currently open - requests are blocked")
else:
    service_call()  # Execute call if not in OPEN status

PostgreSQL Shared Storage

For distributed applications running across multiple instances, Resilient Circuit supports PostgreSQL as a shared storage backend. This allows circuit breaker state to be synchronized across all instances of your application.

Setting Up PostgreSQL Storage

1. Install PostgreSQL dependencies:

pip install resilient_circuit[postgres]

2. Create the database:

You need to create the database first (the CLI assumes the database exists). Create it using your preferred method:

createdb -h localhost -p 5432 -U postgres resilient_circuit_db

Or using psql:

CREATE DATABASE resilient_circuit_db;

3. Configure environment variables:

Create a .env file in your project root:

RC_DB_HOST=localhost
RC_DB_PORT=5432
RC_DB_NAME=resilient_circuit_db
RC_DB_USER=postgres
RC_DB_PASSWORD=your_password

Both RC_DB_HOST and RC_DB_PASSWORD must be set, or storage silently stays in-memory.

TLS and client-certificate authentication

The variables above build a plaintext connection string. If your server requires TLS — sslmode=verify-full, a client certificate, or both — use RC_DB_DSN instead. It is a complete libpq conninfo or postgresql:// URL, passed to psycopg verbatim, and it takes precedence over the discrete variables:

RC_DB_DSN=postgresql://cb_user:password@db.example:5432/appdb?sslmode=verify-full&sslrootcert=/etc/ssl/cert.pem&sslcert=/etc/tls/client.crt&sslkey=/etc/tls/client.key

The client key file must be mode 0600 or libpq refuses it.

If you use the discrete variables instead, the TLS settings are supplied as four more of them:

RC_DB_HOST=db.example
RC_DB_PASSWORD=password
RC_DB_SSLMODE=verify-full
RC_DB_SSLROOTCERT=/etc/ssl/cert.pem
RC_DB_SSLCERT=/etc/tls/client.crt
RC_DB_SSLKEY=/etc/tls/client.key

The two forms are exclusive, and this is the easy mistake to make. If RC_DB_DSN is set it is used verbatim and the four RC_DB_SSL* variables are ignored — they cannot be merged into a DSN without breaking the guarantee that makes RC_DB_DSN useful. So:

If you pass a DSN, the TLS parameters must be in the DSN. RC_DB_SSL* apply only when RC_DB_DSN is unset.

Setting both logs an ERROR naming the ignored variables when the DSN carries no TLS of its own, because that combination would otherwise connect without encryption while looking configured for it.

When PostgreSQL storage is requested but unreachable, create_storage() logs the error, raises a RuntimeWarning, and falls back to InMemoryStorage — breaker state becomes process-local rather than shared. Treat that warning as a configuration failure, not noise. Set RC_DB_STRICT=1 to make it an error instead: create_storage() then re-raises rather than degrading. A degraded breaker is self-consistent inside each process and wrong only in aggregate, which is the hardest kind of failure to notice.

4. Use the CLI to set up the table:

resilient-circuit-cli pg-setup --grant-to my_app_role

pg-setup is the only thing in resilient-circuit that issues DDL. It reads the same environment as the runtime — including RC_DB_DSN and the RC_DB_SSL* variables, so it reaches a server that mandates TLS or client certificates — then creates or upgrades the table, its indexes, trigger and comments.

Run it as a role that owns (or may create) the table, and name your application role with --grant-to. The application role then needs only SELECT, INSERT, UPDATE and DELETE; it never needs ownership, and PostgresStorage issues no DDL at runtime.

resilient-circuit-cli pg-setup --grant-to my_app_role   # create/upgrade, then grant
resilient-circuit-cli pg-setup --yes                    # skip the confirmation prompt
resilient-circuit-cli pg-setup --dry-run                # show what would be done
resilient-circuit-cli pg-check --role my_app_role       # verify schema + grants

--grant-to defaults to RC_DB_APP_USER when that is set. The grant is verified by catalog read before pg-setup reports success, and pg-setup refuses when the named role already has owner authority over the table — including by membership of the owning role — because in that case there is no privilege split to grant.

Schema readiness at runtime

PostgresStorage verifies the schema on construction with a single read-only catalog query and issues no DDL on any code path. If the table is missing or has drifted it raises SchemaNotReady, naming what is wrong and how to fix it, rather than failing later at the moment a circuit tries to trip.

Set RC_DB_AUTO_CREATE=1 to let the runtime provision the schema itself. That restores the pre-0.8.0 behaviour and requires the application role to own or be able to create the table — which discards the privilege split, so prefer pg-setup.

Using PostgreSQL Storage

Once configured, the circuit breaker will automatically use PostgreSQL storage when environment variables are present:

from datetime import timedelta
from fractions import Fraction
from resilient_circuit import CircuitProtectorPolicy

# This will automatically use PostgreSQL if RC_DB_* env vars are set
circuit_breaker = CircuitProtectorPolicy(
    resource_key="payment_service",
    cooldown=timedelta(seconds=60),
    failure_limit=Fraction(5, 10),  # 50% failure rate
    success_limit=Fraction(3, 3),  # 3 consecutive successes to close
)


@circuit_breaker
def process_payment():
    # Your payment processing logic
    pass

Benefits of PostgreSQL Storage

• Shared State: Circuit breaker state is synchronized across all application instances
• Persistence: State survives application restarts
• Monitoring: Query circuit breaker state directly from the database
• Scalability: Supports high-concurrency applications
• Atomic Operations: Uses PostgreSQL row-level locking for thread-safe updates

Monitoring Circuit Breakers

Query the database to monitor circuit breaker status:

-- View all circuit breakers and their status
SELECT resource_key, state, failure_count, open_until, updated_at
FROM rc_circuit_breakers
ORDER BY updated_at DESC;

-- Find all open circuit breakers
SELECT resource_key, open_until
FROM rc_circuit_breakers
WHERE state = 'OPEN';

-- Check failure rates for specific services
SELECT resource_key, failure_count
FROM rc_circuit_breakers
WHERE state = 'CLOSED';

Fallback to In-Memory Storage

If PostgreSQL is not configured or unavailable, the circuit breaker automatically falls back to in-memory storage:

# No environment variables set - uses in-memory storage
circuit_breaker = CircuitProtectorPolicy(resource_key="my_service")

# Or explicitly specify in-memory storage
from resilient_circuit.storage import InMemoryStorage

circuit_breaker = CircuitProtectorPolicy(
    resource_key="my_service", storage=InMemoryStorage()
)

When PostgreSQL was requested (RC_DB_* set) but is unavailable, a RuntimeWarning is emitted — a distributed deployment must never degrade to per-process isolation invisibly. Check storage.backend_name to confirm the effective backend ("postgres" for shared state, "in-memory" for process-local).

InMemoryStorage is bounded: it keeps at most max_entries resource keys (default 8192, pass None for unbounded) and evicts the least-recently-used entry that is not a live OPEN — a live protection signal is never dropped. Use storage.delete_state(resource_key) to remove a retired circuit (e.g. an OPEN that will never recover).

### Environment Variables Reference

| Variable | Description | Default |
|----------|-------------|---------|
| `RC_DB_HOST` | PostgreSQL host | Required |
| `RC_DB_PORT` | PostgreSQL port | `5432` |
| `RC_DB_NAME` | Database name | `resilient_circuit_db` |
| `RC_DB_USER` | Database user | `postgres` |
| `RC_DB_PASSWORD` | Database password | Required |
| `RC_DB_DSN` | Complete libpq conninfo or URL, used verbatim; wins over the discrete variables | Unset |
| `RC_DB_SSLMODE` | TLS mode appended to the discrete form | Unset |
| `RC_DB_SSLROOTCERT` | CA bundle path | Unset |
| `RC_DB_SSLCERT` | Client certificate path | Unset |
| `RC_DB_SSLKEY` | Client key path (mode 0600) | Unset |
| `RC_DB_APP_USER` | Application role `pg-setup --grant-to` defaults to | Unset |
| `RC_DB_AUTO_CREATE` | Allow the runtime to provision the schema (pre-0.8.0 behaviour) | `0` |
| `RC_DB_STRICT` | Raise instead of falling back to in-memory storage | `0` |
| `RC_NAMESPACE` | Namespace when none is passed to `create_storage()` | `default` |

## Highway Workflow Engine Integration

Resilient Circuit is a core component of the Highway Workflow Engine, designed for building resilient, distributed applications. The Highway Workflow Engine provides:

- **Workflow Orchestration**: Define complex business processes
- **Task Management**: Execute and monitor long-running tasks
- **Resilience Patterns**: Built-in fault tolerance with circuit breakers and retries
- **Monitoring & Observability**: Track workflow execution and identify bottlenecks

Learn more about the complete Highway Workflow Engine at [highway-workflow-engine.readthedocs.io](https://highway-workflow-engine.readthedocs.io).

## API Reference

### CircuitProtectorPolicy

Implements the circuit protector pattern with three statuses: CLOSED, OPEN, HALF_OPEN.

**Parameters:**
- `cooldown` (timedelta): Duration before transitioning from OPEN to HALF_OPEN
- `failure_limit` (Fraction): Failure rate to trip the protector (e.g., Fraction(3, 10) for 3 out of 10)
- `success_limit` (Fraction): Success rate to close the protector in HALF_OPEN status
- `should_handle` (Callable): Predicate to determine which exceptions to count as failures
- `on_status_change` (Callable): Callback when the protector changes status

### RetryWithBackoffPolicy

Implements the retry pattern with configurable backoff strategies.

**Parameters:**
- `backoff` (ExponentialDelay | FixedDelay): Backoff strategy between retries
- `max_retries` (int): Maximum number of retry attempts
- `should_handle` (Callable): Predicate to determine which exceptions to retry

### SafetyNet

Combines multiple policies for comprehensive error handling.

**Parameters:**
- `policies` (tuple): Tuple of policies to apply

### ExponentialDelay Strategies

- `ExponentialDelay`: Exponential backoff with configurable parameters
- `FixedDelay`: Constant delay between attempts

## Best Practices

1. **Configure Appropriate Limits**: Set failure limits based on your service's expected error rate
2. **Use Meaningful Cooldown Periods**: Balance between detecting recovery and avoiding thrashing
3. **Handle Specific Exceptions**: Use the `should_handle` parameter to only respond to expected failures
4. **Monitor Status Changes**: Use `on_status_change` to detect and log circuit protector transitions
5. **Chain Policies Thoughtfully**: Apply retry before circuit protector for optimal resilience

## Contributing

We welcome contributions to Resilient Circuit! See our [contributing guide](CONTRIBUTING.md) for details.

## License

Distributed under the Apache Software License 2.0. See [LICENSE](LICENSE) for more information.

## Support

Need help? Check out our documentation or open an issue on GitHub.

---

*Part of the Highway Workflow Engine family of resilience tools*