Typed Evals

Evaluate responses. Guard actions.
A Python toolkit for evaluating LLMs, RAG, and Agents using Decision Models with optional calibration against human labels

Quickstart · Agent evaluation · Tool guards · Calibration · Examples & guides

Use Jev by default, or native OpenAI Decisions API, to judge generated responses and recorded agent executions. Check proposed tool calls before they run. Start with a preset, then bring your own metrics, thresholds, or judge backend as your application grows.

Quickstart

Install with Python 3.11+. This quickstart uses the default Jev backend, which requires a TypeSafe API key. To use OpenAI Decisions instead, set OPENAI_API_KEY and follow Use OpenAI Decisions.

python -m pip install typed-evals
export TYPESAFE_API_KEY='your-key' # or export OPENAI_API_KEY='your_key'

Evaluate a response against the evidence used to generate it:

from typed_evals import evaluate

result = evaluate(
    input="What is the refund period?",
    response="You can request a refund within 30 days.",
    contexts=["Refunds are allowed within 30 days of purchase."],
    preset="rag",
)

print("All checks passed:", result.passed)
for name, metric in result.metrics.items():
    print(f"{name}: score={metric.score}, passed={metric.passed}")

That checks faithfulness, answer relevancy, and context relevance in one judge request. The default backend uses jev-1.13.0 through the official typesafe-sdk 0.7.x; live examples make API calls using your account.

Want to explore without an API key? Jump to the offline demo.

Use OpenAI Decisions

Install the optional official SDK and set its normal environment variable:

python -m pip install 'typed-evals[openai]'
export OPENAI_API_KEY='your-key'

from typed_evals import OpenAIDecisionsBackend, evaluate

backend = OpenAIDecisionsBackend(model="gpt-6-luna", timeout=30.0, max_retries=2)
result = evaluate(
    input="What is the refund period?",
    response="You can request a refund within 30 days.",
    contexts=["Refunds are allowed within 30 days of purchase."],
    preset="rag",
    backend=backend,
)
print(result.passed)

This uses POST /v1/decisions through AsyncOpenAI.decisions.create with OpenAI SDK. All active metrics share one request. Samples support text, structured tool evidence, and images. Run examples/openai_decisions.py from a checkout for a complete text example.

Each batch owns and closes one async client, shared by its workers. With OpenAIDecisionsBackend(client=your_async_client), you own the client lifecycle and its timeout/retry settings; evaluate asynchronously in the client's event loop. The SDK handles transport retries.

Decisions maps Noul to predicate, Choice to string-valued choices, and Score to ordered levels. Raw scores keep their existing meaning: positive-condition probability, summed probability of passing options, or normalized expected ordinal level. Choice/Score provider confidence is separate; predicates have no confidence field. Refusals are unavailable judgments: they raise by default, or become individual error results under errors="record", preserving successful sibling metrics and leaving the sample's pass status unavailable. Runtime guards apply their configured error policy.

Evaluate image evidence

Add images to a sample and select them in a custom metric's required_fields:

from typed_evals import ImageInput, Metric, OpenAIDecisionsBackend, evaluate

visual_grounding = Metric(
    name="visual_grounding",
    kind="noul",
    instructions="Does the response accurately describe the supplied image?",
    pass_definition="Every material visual claim is supported by the image.",
    required_fields=("input", "response", "images"),
)
result = evaluate(
    input="Describe this receipt.",
    response="The receipt shows a total of $42.00.",
    images=[ImageInput.from_file("receipt.png", label="receipt")],
    metrics=[visual_grounding],
    backend=OpenAIDecisionsBackend(),
)
print(result.passed)

ImageInput accepts inline PNG, JPEG, WebP, and GIF data URLs, with from_file and from_bytes helpers. Images use native Decisions image parts alongside text; image order and optional labels identify the evidence. Built-in presets select text evidence, so attaching images alone does not include them in the judge request. Jev currently supports text; image-selecting metrics require a backend advertising image support. The sample and metric APIs are shared across providers, so future Jev image support can use the same code with backend=JevBackend().

See the image guide and runnable image example.

What you can build

Choose a preset

Presets use a 0.5 threshold for each metric. Supplying contexts alone does not enable RAG checks—choose preset="rag" explicitly. Missing evidence and judge errors raise by default. Use custom metrics to choose different checks or thresholds.

Evaluate agent execution

Use preset="agent" to check whether a completed run achieved its expected outcome and reported tool results accurately. Supply the final response, observed execution trace, and explicit success criteria:

from typed_evals import evaluate

result = evaluate(
    preset="agent",
    input="Create a support ticket for the export failure.",
    response="I created ticket T-42 for your export failure.",
    expected_outcome="A new support ticket exists for the reported export failure.",
    trace=[
        {
            "name": "create_ticket",
            "arguments": {"title": "Export failure"},
            "status": "error",
            "output": {"created": False},
            "error": "Permission denied",
        }
    ],
)

for name, metric in result.metrics.items():
    print(f"{name}: score={metric.score}, passed={metric.passed}")

• Task completion checks whether execution evidence establishes expected_outcome. Plans, attempts, and claims of success are insufficient.
• Tool grounding checks whether the response's claims match the observed tool results, including failed or unverified executions.

Here, ticket creation failed while the agent claimed success. Both rubrics describe a failure; the returned scores depend on the selected judge's judgment.

Capture trace from actual executions in your integration. The preset covers completion and result reporting; step ordering, efficiency, and recovery quality need separate checks. See the runnable example and the runtime guide for enforcing checks at agent and tool boundaries.

Evaluate a dataset

Save your samples as samples.jsonl, with one JSON object per line:

{"input":"Refund period?","response":"30 days.","contexts":["Refunds are allowed within 30 days of purchase."]}

from typed_evals import evaluate

report = evaluate("samples.jsonl", preset="rag")
print(report.summary)
report.save("evaluation-report.json")

You can also pass a dictionary, an EvaluationSample, or a list/tuple of either. A single sample returns SampleResult; a sequence or file returns EvaluationReport, even for one row. JSON files contain an array of samples.

Each sample's metrics share a judge request. Batches validate inputs first, run up to eight concurrent workers by default, and preserve input order.

In notebooks and async applications, use aevaluate with the same dataset:

from typed_evals import aevaluate

report = await aevaluate("samples.jsonl", preset="rag")

The synchronous API also works in notebooks, but blocks until evaluation finishes.

From the terminal or CI:

typed_evals evaluate samples.jsonl \
  --metrics faithfulness answer_relevancy context_relevance \
  --output evaluation-report.json \
  --fail-on-failure

# Native OpenAI Decisions; the default model is gpt-6-luna:
typed_evals evaluate samples.jsonl --backend openai-decisions \
  --metrics faithfulness answer_relevancy context_relevance \
  --output openai-report.json --fail-on-failure

The CLI returns a nonzero exit code for failed or unavailable evaluations when --fail-on-failure is set. See the evaluation guide for error policies, data formats, and report fields.

Guard tool calls

Add a policy check to a LangChain tool before registering it:

python -m pip install 'typed-evals[langchain]'

from langchain.tools import ToolRuntime, tool
from typed_evals.adapters.langchain import guard_tool


@tool
@guard_tool(
    policy="Only read tickets owned by the authenticated customer.",
    contexts=lambda runtime: runtime.context["authorization_evidence"],
)
def read_ticket(ticket_id: str, runtime: ToolRuntime) -> str:
    """Read the complete text of a support ticket."""
    return ticket_store.read_authorized(runtime.context["customer_id"], ticket_id)

Here, ticket_store is your application's ticket service. Supply the current customer ID and application-owned authorization evidence through the agent's runtime context; the complete LangChain example shows the wiring.

The adapter captures the latest human request, proposed arguments, tool name, and call ID. By default, it checks ToolSafety at a 0.9 threshold and raises GuardrailViolation on a failed or unavailable judgment before the function runs. Successful calls preserve the tool's native return value. Keep deterministic authorization inside the tool, as shown above.

Use your framework

The plain Python helper takes explicit input and contexts, as values or callbacks. The CrewAI adapter creates a native tool; the Microsoft adapter wraps a function under @agent_framework.tool. See the adapter guide for each framework's registration and evidence requirements, and the runtime guide for agent decorators and additional checkpoints.

The currently supported CrewAI release requires OpenAI SDK <3, while Decisions requires >=3.26.0. Use separate environments for the CrewAI extra and native Decisions until those dependency bounds are compatible. The plain Python, LangChain, and Microsoft guard integrations accept a Decisions backend.

Make the checks yours

Choose custom metrics instead of a preset to control the panel and thresholds:

from typed_evals import AnswerRelevancy, Faithfulness, evaluate

result = evaluate(
    input="Refund period?",
    response="30 days.",
    contexts=["Refunds are allowed within 30 days."],
    metrics=[Faithfulness(threshold=0.8), AnswerRelevancy(threshold=0.8)],
)

For reusable configuration, use Evaluator(preset="rag") or Evaluator(metrics=[...]).

from typed_evals import list_metrics

print(list_metrics())  # No API credentials needed.

from typed_evals import Metric, evaluate

completeness = Metric(
    name="completeness",
    kind="score",
    instructions="How completely does `response` cover the details requested in `input`?",
    criteria=(
        "None of the requested details are supplied.",
        "Some details are supplied, but essential requested details are missing.",
        "All essential requested details are supplied.",
    ),
    pass_definition="All essential requested details are supplied.",
    required_fields=("input", "response"),
    threshold=0.8,
)

result = evaluate(
    input="What is the refund period, and how do I request one?",
    response="You have 30 days. Contact support with your order number.",
    metrics=[completeness],
)

Calibration

Make thresholds reflect your own acceptance criteria. A raw judge score of 0.8 does not necessarily mean humans would pass 80% of similar examples.

With typed-evals[calibration], use EvaluationPipeline to fit per-metric isotonic curves against representative human pass/fail labels. Inspect the held-out diagnostics, save the fitted artifact, and load it for later evaluations. Calibration is opt-in; improvements are measured on held-out data rather than assumed.

Read the calibration walkthrough for fitting, validation, and saved artifacts. The included labeled datasets are synthetic examples; replace them with human-reviewed labels for your application.

For Decisions, install typed-evals[openai,calibration] and fit a separate artifact with backend=OpenAIDecisionsBackend() or typed_evals calibrate ... --backend openai-decisions. Calibration uses raw metric scores, never provider confidence. New v2 artifacts record backend/compiler provenance and ordered request configuration; provider or compiler changes require a refit, while threshold changes do not. V1 artifacts remain readable for the established Jev compiler and custom backends without provenance; they cannot establish Decisions compatibility and require a fresh Decisions fit. See artifact compatibility.

metric.score uses the raw score unless calibration is applied. metric.passed compares it with the threshold and is None for unavailable evaluations. Judge results can be wrong; individual checks do not establish a single probability that an entire answer is true.

Benchmark: TRIVIA+

On 645 TRIVIA+ test answers, calibration reduced Jev's expected calibration error (ECE) by 68.1%, from 0.0982 to 0.0313, with no additional judge calls. With cutoffs chosen on validation data, hallucination detection remained similar: F1 moved from 0.5833 to 0.5877, while accuracy stayed around 72%.

The measured gain was in how closely probabilities matched human judgments. Read the full benchmark for the dataset, methodology, results, and limitations.

Try it offline

Clone the repository to get the example scripts and datasets; they are not bundled in the installed package:

git clone https://github.com/TrustifAI/typed_evals.git
cd typed_evals
python -m pip install -e '.[calibration]'
python examples/offline_demo.py

This demo uses a deterministic synthetic judge: no API key and no network calls. It shows how a raw score, a calibrated probability, and a pass/fail decision relate.

From the same checkout, explore runtime enforcement:

python examples/runtime_guardrails.py
python examples/decorated_agent.py

The CrewAI and Microsoft Agent Framework demos also use synthetic judges; install the corresponding framework extra first.

Examples and guides

For notebooks, install the repository into the kernel's environment and run from the repository root. They use python-dotenv and make live requests with TYPESAFE_API_KEY. Calibration needs the calibration extra; the advanced agent sections also need agent-framework, langchain, langchain-google-genai, and Gemini credentials.

Contribute

Found a confusing result, need an integration, or have a useful evaluation example? Open an issue or send a pull request. Include a minimal reproduction for bugs and tests for behavior changes.

From a repository checkout:

python -m pip install -e '.[openai,calibration,dev,langchain,agent-framework]'
pytest -m "not live" --cov=typed_evals --cov-report=term-missing
ruff check .
ruff format --check .
python -m build

CI installs the OpenAI extra and runs both providers' contract tests against mocked HTTP, along with framework integrations using fake judges. CrewAI tests run in a separate environment with .[calibration,dev,crewai] because its SDK bounds currently conflict with Decisions. Normal tests need no credentials or paid calls. Tests for optional frameworks are skipped when those extras are absent. The separate Jev live smoke test requires both TYPED_EVALS_LIVE=1 and TYPESAFE_API_KEY and makes a billable request. Decisions' three-question text smoke test is disabled by default; to opt into its billable request after setting OPENAI_API_KEY, run:

TYPED_EVALS_OPENAI_LIVE=1 pytest tests/test_live.py::test_live_openai_decisions_smoke -m live

It defaults to gpt-6-luna; TYPED_EVALS_OPENAI_MODEL optionally selects another model ID. Live smoke tests check the service contract rather than metric accuracy. Maintainers can follow the publishing guide to cut a release.

Built by Aaryan Verma · MIT licensed
Useful for your next evaluation? Star the repo to keep it close.