confident-ai (Python)

The official Python SDK for the Confident AI platform management API — manage organizations, projects, API keys, members, invitations, roles, and policies.

Looking to run evaluations, upload traces, or pull datasets? Use deepeval. This SDK focuses on platform/administration APIs.

Documentation: Admin SDK docs

Installation

pip install confident-ai

Quickstart

from confident_ai import ConfidentAI

client = ConfidentAI(api_key="confident_org_...")

# The organization the API key is scoped to
organization = client.organization().get()

# All projects in the organization
projects = client.projects.list()

# A scoped project client
project = client.project("project_id")
members = project.members.list()

Authentication

These endpoints require an organization API key. Provide it explicitly or via the environment:

client = ConfidentAI(api_key="confident_org_...")

export CONFIDENT_ORG_API_KEY="confident_org_..."

Use CONFIDENT_ORG_API_KEY, not CONFIDENT_API_KEY. The latter is reserved by deepeval for a project key; the distinct name lets both SDKs run side by side.

from confident_ai import ConfidentAI

client = ConfidentAI()  # reads CONFIDENT_ORG_API_KEY

Configuration

ConfidentAI(api_key=None, base_url=None, timeout=None)

Regional defaults: https://api.confident-ai.com (US), https://eu.api.confident-ai.com (EU).

Async

Every method that hits the API has an a_-prefixed async counterpart on the same client (list / a_list, assign / a_assign, …) — the same style as deepeval's measure / a_measure. There is no separate async client; use one ConfidentAI and await the a_ methods in asyncio code:

import asyncio

from confident_ai import ConfidentAI


async def main():
    client = ConfidentAI(api_key="confident_org_...")

    organization = await client.a_whoami()
    projects = await client.projects.a_list()

    project = client.project("project_id")          # no I/O, no await
    members = await project.members.a_list()          # awaited

    created = await client.projects.a_create("Async App")
    await client.project(created.project.id).a_delete()


asyncio.run(main())

organization() and project(id) just build scoped clients (no network), so they are not awaited; only the a_-prefixed methods that actually hit the API are coroutines. Configuration, regions, and ConfidentApiError behave identically to the sync methods.

Organization example

org_client = client.organization()

# Profile
organization = org_client.get()
org_client.update(name="Acme Inc.")

# API keys (the full value is returned only on create)
created = org_client.api_keys.create(name="CI/CD key")
print(created.value)
org_client.api_keys.update(created.id, valid=False)
org_client.api_keys.delete(created.id)

# Members & invitations
members = org_client.members.list(page=1, page_size=25)
org_client.members.update_role(members[0].id, role_id="role_id")
org_client.invitations.create(["teammate@acme.com"], role_id="role_id")

# IAM: roles, policies & permissions
permissions = org_client.iam.permissions.list()
policy = org_client.iam.policies.create(
    "Billing", permission_ids=[permissions[0].id]
)
org_client.iam.roles.create("Billing Manager", policy_ids=[policy.id])

# Governance: list policies and assign projects to one (great for CI/CD)
governance_policies = org_client.governance.policies.list()
if governance_policies:
    org_client.governance.policies.assign(
        governance_policies[0].id, project_ids=["project_id"]
    )

Project example

# Create a project (returns the project + its first API key)
created = client.projects.create("Production App", description="Main app")
print(created.project.id, created.api_key.value)

# List / fetch
projects = client.projects.list()
project = client.project(created.project.id)
project.update(name="Production")

# Project-scoped resources
project.api_keys.create(name="Production agent key")
project.members.list()
project.invitations.create(["analyst@acme.com"], role_id="project_role_id")

# Project-scoped IAM (roles, policies, permissions)
project.iam.roles.list()
project.iam.policies.list()
project.iam.permissions.list()

# Delete
project.delete()

Error handling

When the API returns an unsuccessful response, the SDK raises ConfidentApiError, which carries the error message and an optional link to relevant docs.

from confident_ai import ConfidentAI, ConfidentApiError

client = ConfidentAI()

try:
    client.project("does-not-exist").get()
except ConfidentApiError as err:
    print("Something went wrong:", err, err.link)

Development

cd python
poetry install
poetry run pytest

Or without Poetry:

cd python
python -m venv .venv && source .venv/bin/activate
pip install requests "pydantic>=2.11" tenacity aiohttp pytest
pytest

The default suite mocks the HTTP layer and never hits a real API, so it runs anywhere (and is the pre-merge CI gate).

Integration tests (optional, real API)

A separate, opt-in suite under tests/integration/ exercises the live API. It is excluded from the default run and auto-skips unless CONFIDENT_ORG_API_KEY is set:

CONFIDENT_ORG_API_KEY="confident_org_..." poetry run pytest -m integration

Most checks are read-only (whoami, projects.list, permissions.list); the project round-trip creates and then deletes a throwaway project. CI runs this suite nightly / on demand via the Integration workflow, never on pull requests.

See ROUTES.md for the full endpoint → method map.