RubricLoop SDK

RubricLoop checks AI agent output with ordinary code, returns exact failures to the agent, and stops when the work passes or the run hits a clear limit.

Deterministic checks never call an LLM. An optional, explicitly configured hybrid judge can score only the rule IDs you declare and cannot override a deterministic failure. Your agent and judge can use any OpenAI-compatible provider.

Native registry packages use the deterministic, signed .rlpack format.

Quick start

Install the SDK and OpenAI client, then run the tested reply example:

pip install rubricloop openai
export OPENAI_API_KEY="sk-..."
python -m rubricloop.examples.verify_reply

Install only the extensions you use:

pip install "rubricloop[judge]"  # OpenAI-compatible LLM judge
pip install "rubricloop[media]"  # image, workbook, and presentation extraction
pip install "rubricloop[judge,media]"

The example deliberately produces a short first draft containing a forbidden promise. RubricLoop measures the failures and passes those diagnostics into the next model call. The process exits successfully only when the reply satisfies all three rules within the iteration and token budgets.

Verify a SQL agent

from rubricloop import AgentRequest, AgentResponse, Budget, verify
from rubricloop.packs import sql_safe
from rubricloop.sandboxes import SQLiteSandbox

sandbox = SQLiteSandbox(
    ddl="""
        CREATE TABLE users (
            id INTEGER PRIMARY KEY,
            name TEXT NOT NULL,
            country TEXT NOT NULL
        );
    """,
    seed_sql="""
        INSERT INTO users (name, country)
        VALUES ('Ada', 'DE'), ('Grace', 'US'), ('Linus', 'FI');
    """,
)

rules = sql_safe(
    schema={"users": ["id", "name", "country"]},
    reference_query="SELECT name FROM users WHERE country = 'DE'",
    columns=["name"],
)

def agent(request: AgentRequest) -> AgentResponse:
    # Replace this with your model call. Feed request.feedback into the next prompt.
    if request.feedback:
        return AgentResponse(
            "SELECT name FROM users WHERE country = 'DE'",
            prompt_tokens=120,
            completion_tokens=14,
        )
    return AgentResponse("SELECT * FROM users", prompt_tokens=80, completion_tokens=4)

run = verify(
    agent,
    "Return the names of users in Germany.",
    rules,
    sandbox=sandbox,
    budget=Budget(max_iterations=3, max_tokens=2_000),
)

assert run.passed
print(run.output)
print(run.to_dict())

Compose a small text rubric

from rubricloop import rubric
from rubricloop.checks import forbidden_terms, required_terms, word_count

rules = rubric(
    "support/reply",
    word_count("reply.length", minimum=40, maximum=70),
    required_terms("reply.required", ["timeline"]),
    forbidden_terms("reply.forbidden", ["guaranteed refund"]),
)

Gate a refund action

Refunds use a decision gate, not a retry loop. The gate returns allow, approval, or deny without changing the proposed amount.

from datetime import datetime, timezone
from rubricloop.packs import RefundOrder, RefundPolicy, check_refund

result = check_refund(
    {
        "action": "issue_refund",
        "order_id": "ORD-100",
        "amount": 80,
        "reason": "damaged",
    },
    policy=RefundPolicy(
        refund_window_days=30,
        max_auto_approve=50,
        deny_above=500,
        allowed_reasons=("damaged", "wrong_item", "not_received"),
    ),
    order=RefundOrder(
        id="ORD-100",
        sku="STANDARD-1",
        paid_amount=120,
        refunded_amount=20,
        purchased_at=datetime(2026, 9, 13, tzinfo=timezone.utc),
    ),
)

assert result.decision == "approval"

Never lower or split an action to make it pass. Send the original request and its failed rules to a reviewer.

Current scope

• Bounded retries with token and iteration limits
• Measured failure-state cycle detection
• Rollback to the candidate with the fewest failed rules
• Rule-level feedback and event callbacks
• Text, JSON, PII, and SQL checks
• Local in-memory SQLite sandbox
• Ready-made engineering/sql-safe pack
• support/refund-policy action gate
• Signed, fresh, release-pinned reference feeds
• Scoped LLM-as-judge rules with token evidence and abstention
• Decisive streaming checks with cancellation or observation modes
• Digest-bound artifacts and deterministic media extraction

Four extension examples

The wheel includes one offline, synthetic example for each extension study. They require no API key or customer data and are exercised by the SDK test suite:

python -m rubricloop.examples.reference_feed_example
python -m rubricloop.examples.llm_judge_example
python -m rubricloop.examples.streaming_example
python -m rubricloop.examples.multimodal_example

The main extension parameters preserve the original text-only behavior:

run = verify(
    agent,
    prompt,
    rules,
    judge=judge_config,       # JudgeConfig; credentials are references
    artifact=artifact,       # Artifact or a sequence of artifacts
    extract=extract_spec,    # ExtractSpec or a sequence of extractors
    streaming="cut",         # False, True/"cut", or "observe"
    constrain=True,          # Pass compiled constraints when supported
)

Extension evidence is emitted only when used through run.references, run.judge, run.artifacts, run.extractions, run.constraints, and the matching keys in run.to_dict().

Registry CLI

The same installation adds the rubricloop command. Start locally:

rubricloop init
rubricloop validate dist/acme-check.rlpack
rubricloop test dist/acme-check.rlpack

When the package is ready to share, create an API key in the RubricLoop dashboard, then connect to the hosted registry:

rubricloop login --api-key "$RUBRICLOOP_API_KEY"
rubricloop create acme/check --visibility private
rubricloop push dist/acme-check.rlpack --tag latest
rubricloop vet acme/check@latest
rubricloop pull acme/check@latest

Public community packages are free. Private repositories are available only to their organization and authorized collaborators. login stores the key in the operating system keychain, never in the project.

Contributing

From the sdk directory:

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pytest