circuit-breaker-labs 1.0.8

A client library for accessing Circuit Breaker Labs API

Circuit Breaker Labs Python Client

[!Note] This project was automatically generated by OpenAPI Python Client from this OpenAPI spec.

Installation

Install from PyPi directly:

uv pip install circuit-breaker-labs

Or install using a wheel/sdist distributed with each release.

Nix flake usage

Minimal flake using this package as an input:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
    cbl-lib.url = "github:circuitbreakerlabs/circuitbreakerlabs-python";
  };

  outputs = { nixpkgs, flake-utils, cbl-lib, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ cbl-lib.overlays.${system}.default ];
        };
      in {
        devShells.default = pkgs.mkShell {
          packages = [
            (pkgs.python314.withPackages (ps: [ ps.cblLib ]))
          ];
        };
      });
}

Quick check:

nix develop --command python -c "from circuit_breaker_labs import Client; client = Client(base_url='https://api.circuitbreakerlabs.ai/v1/'); print(type(client).__name__)"

Usage

First, create a client:

from circuit_breaker_labs import Client

client = Client(base_url="https://api.circuitbreakerlabs.ai/v1/")

To see which test case groups your API key can use:

import os

from circuit_breaker_labs.api.test_case_groups import get_test_case_groups_get

with client as client:
    test_case_groups = get_test_case_groups_get.sync(
        client=client,
        cbl_api_key=os.environ["CBL_API_KEY"],
    )

Now build a request and use it when calling an endpoint

import os

from circuit_breaker_labs.api.evaluations import singleturn_evaluate_system_prompt_post
from circuit_breaker_labs.models import SingleTurnEvaluateSystemPromptRequest

with client as client:
    request = SingleTurnEvaluateSystemPromptRequest(
        threshold=0.5,
        variations=3,
        maximum_iteration_layers=2,
        test_case_groups=["suicidal_ideation"],
        system_prompt=os.environ["SYSTEM_PROMPT"],
        openrouter_model_name="anthropic/claude-3.7-sonnet",
    )

    response = singleturn_evaluate_system_prompt_post.sync(
        client=client,
        cbl_api_key=os.environ["CBL_API_KEY"],
        body=request,
    )

Or do the same thing with an async version:

import os

from circuit_breaker_labs.api.evaluations import singleturn_evaluate_system_prompt_post
from circuit_breaker_labs.models import SingleTurnEvaluateSystemPromptRequest

async with client as client:
    request = SingleTurnEvaluateSystemPromptRequest(
        threshold=0.5,
        variations=3,
        maximum_iteration_layers=2,
        test_case_groups=["suicidal_ideation"],
        system_prompt=os.environ["SYSTEM_PROMPT"],
        openrouter_model_name="anthropic/claude-3.7-sonnet",
    )

    response = await singleturn_evaluate_system_prompt_post.asyncio(
        client=client,
        cbl_api_key=os.environ["CBL_API_KEY"],
        body=request,
    )

Want to test multi-turn conversations instead? Use the multi-turn endpoint (async version also available):

import os

from circuit_breaker_labs.api.evaluations import multi_turn_evaluate_system_prompt_post
from circuit_breaker_labs.models import MultiTurnEvaluateSystemPromptRequest

with client as client:
    request = MultiTurnEvaluateSystemPromptRequest(
        threshold=0.6,
        max_turns=6,
        test_case_groups=["suicidal_ideation"],
        system_prompt=os.environ["SYSTEM_PROMPT"],
        openrouter_model_name="anthropic/claude-3.7-sonnet",
    )

    response = multi_turn_evaluate_system_prompt_post.sync(
        client=client,
        cbl_api_key=os.environ["CBL_API_KEY"],
        body=request,
    )

Things to know:

1. Every path/method combo becomes a Python module with four functions:

   1. sync: Blocking request that returns parsed data (if successful) or None
   2. sync_detailed: Blocking request that always returns a Request, optionally with parsed set if the request was successful.
   3. asyncio: Like sync but async instead of blocking
   4. asyncio_detailed: Like sync_detailed but async instead of blocking

2. All path/query params, and bodies become method arguments.

3. If your endpoint had any tags on it, the first tag will be used as a module name for the function (my_tag above)

4. Any endpoint which did not have a tag will be in circuit_breaker_labs.api.default

Advanced customizations

There are more settings on the generated Client class which let you control more runtime behavior, check out the docstring on that class for more info. You can also customize the underlying httpx.Client or httpx.AsyncClient (depending on your use-case):

from circuit_breaker_labs import Client

def log_request(request):
    print(f"Request event hook: {request.method} {request.url} - Waiting for response")

def log_response(response):
    request = response.request
    print(f"Response event hook: {request.method} {request.url} - Status {response.status_code}")

client = Client(
    base_url="https://api.circuitbreakerlabs.ai/v1/",
    httpx_args={"event_hooks": {"request": [log_request], "response": [log_response]}},
)

# Or get the underlying httpx client to modify directly with client.get_httpx_client() or client.get_async_httpx_client()

You can even set the httpx client directly, but beware that this will override any existing settings (e.g., base_url):

import httpx
from circuit_breaker_labs import Client

client = Client(
    base_url="https://api.circuitbreakerlabs.ai/v1/",
)
# Note that base_url needs to be re-set, as would any shared cookies, headers, etc.
client.set_httpx_client(httpx.Client(base_url="https://api.circuitbreakerlabs.ai/v1/", proxies="http://localhost:8030"))