verdikt-sdk

Python SDK for Verdikt — a standalone AI evaluation service that decouples evaluation and LLM/human judging from the application being evaluated.

Installation

pip install verdikt-sdk

Usage

from verdikt_sdk import AnswerWithCost, VerdiktClient, EvaluationType, Question
from yalc import LLMModel

client = VerdiktClient(
    base_url="https://your-verdikt-instance.com",
    client_id="your-client-id",
    client_secret="your-client-secret",
)

# The app ("my-app") must already exist and this client must be bound to it,
# created beforehand in the Verdikt admin UI — the SDK does not create apps.

# Sync questions to the dataset (idempotent)
await client.add_questions("my-app", [
    Question(question="What is the capital of France?", human_answer="Paris"),
])

# Your callback returns the answer plus the cost it took your app to produce it.
# `cost` is optional — pass None when you do not track it.
async def my_llm_function(question: str) -> AnswerWithCost:
    answer, cost = await my_app(question)
    return AnswerWithCost(answer=answer, cost=cost)

# Run an evaluation cycle
await client.run_evaluation(
    app_slug="my-app",
    app_version="v1.2.0",
    callback=my_llm_function,
    evaluation_type=EvaluationType.LLM_ONLY,
    llm_judge_models=[LLMModel.gpt_4o_mini],
)

run_evaluation calls your callback concurrently for every question in the dataset, then submits all answers to Verdikt for judgment.

Breaking change in 0.2.0: the callback now returns AnswerWithCost(answer=..., cost=...) instead of a bare str. Callers on 0.1.x must wrap their return value (return AnswerWithCost(answer=ans) is a drop-in equivalent of the old behaviour).

Authentication

The SDK authenticates via the OAuth2 client-credentials grant against any OIDC-compliant provider (Zitadel, Keycloak, Okta, Auth0, ...). The token endpoint is discovered from the provider's /.well-known/openid-configuration, so no provider-specific URL is hardcoded.

Create a machine / service-account client in your IdP and pass its client_id and client_secret to VerdiktClient:

client = VerdiktClient(
    base_url="https://verdikt.mycompany.com",
    client_id="...",
    client_secret="...",
    # Set when the backend verifies a specific OIDC_AUDIENCE and your IdP
    # wouldn't otherwise stamp it onto the token's `aud`:
    audience="<backend OIDC_AUDIENCE>",
)

The backend verifies the token's issuer and audience, so audience must match the service's OIDC_AUDIENCE when audience verification is enabled.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

verdikt_sdk-0.5.0.tar.gz (112.1 kB view details)

Uploaded Jul 4, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

verdikt_sdk-0.5.0-py3-none-any.whl (7.9 kB view details)

Uploaded Jul 4, 2026 Python 3

File details

Details for the file verdikt_sdk-0.5.0.tar.gz.

File metadata

Download URL: verdikt_sdk-0.5.0.tar.gz
Upload date: Jul 4, 2026
Size: 112.1 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for verdikt_sdk-0.5.0.tar.gz
Algorithm	Hash digest
SHA256	`be2edb95247f8ebd28ea6edea352462a704a7e3e46c905945370448868273744`
MD5	`a750ad09e78a9008a4664f97ff97a0c0`
BLAKE2b-256	`303552a8e275e4660a36479ccfe5feb4c1dc7899d170f3a47e5ae8aadb1baecd`

See more details on using hashes here.

Provenance

The following attestation bundles were made for verdikt_sdk-0.5.0.tar.gz:

Publisher: publish.yml on cognitai-labs-dev/verdikt-sdk-python

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: verdikt_sdk-0.5.0.tar.gz
- Subject digest: be2edb95247f8ebd28ea6edea352462a704a7e3e46c905945370448868273744
- Sigstore transparency entry: 2067802177
- Sigstore integration time: Jul 4, 2026
Source repository:
- Permalink: cognitai-labs-dev/verdikt-sdk-python@ad86c42593a7a109daee9a16e2962820e8babe48
- Branch / Tag: refs/tags/v0.5.0
- Owner: https://github.com/cognitai-labs-dev
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: publish.yml@ad86c42593a7a109daee9a16e2962820e8babe48
- Trigger Event: push