mankinds-sdk 1.0.0

Python SDK for Mankinds API

Mankinds SDK

Evaluate AI system with automated tests.

Register an AI system, optionally attach connectors (logs, databases), import or generate your golden dataset, and run automated evaluations covering privacy, security, performance, fairness, explainability, transparency and accountability.

---

Features

• System Management — Create, update, and configure AI systems with custom API endpoints
• Endpoint Configuration — Support for REST, SSE streaming, and multi-turn conversations
• Dataset Generation — Auto-generate or provide custom test scenarios
• Evaluation — Run evaluations with real-time polling and configurable profiles
• Connectors — Attach data sources (log files, Datadog, SQLite, PostgreSQL)
• Error Handling — Typed exceptions for all error cases

Documentation

• Mankinds Documentation

Requirements

• Python ≥ 3.8

Installation

pip install mankinds-sdk

Usage

The SDK follows a simple 3-step workflow: create a system, generate test data, run an evaluation.

Initialize the Client

from mankinds_sdk import MankindsClient

client = MankindsClient(api_key="mk_...")

Parameter	Type	Required	Default	Description
api_key	str	Yes	—	Your API key
base_url	str	No	https://app.mankinds.io	Custom API base URL
timeout	int	No	120	Request timeout in seconds

Create an AI System

Register your AI system by providing its name, description, and API endpoint. The endpoint defines how your AI is called during evaluation.

system = client.create_system(
    name="Customer Support Bot",
    description="A chatbot that handles order inquiries and returns for an e-commerce platform.",
    endpoint={
        "url": "https://api.example.com/chat",
        "method": "POST",
        "headers": {"Authorization": "Bearer your-token"},
        "body": {"message": "{{input}}"},
        "response": {"answer": "{{output}}"}
    }
)

system_id = system["id"]

Use {{input}} in the request body and {{output}} in the response mapping so test inputs and expected outputs are bound during evaluation.

Endpoint Configuration

The endpoint defines how the API calls your AI system during evaluation. It's a JSON object that describes your API's request/response format.

Field	Type	Required	Description
url	string	Yes	API endpoint URL
method	string	Yes	HTTP method (POST, GET, etc.)
body	object	Yes	Request body with {{input}} placeholder
response	object	Yes	Response mapping with {{output}} placeholder
headers	object	No	HTTP headers
streaming	object	No	SSE streaming configuration
multiturn	object	No	Multi-turn conversation configuration

Placeholders:

• {{input}} in body: replaced with test inputs during evaluation
• {{output}} in response: indicates which field contains the AI response

"body": {"message": "{{input}}"},
"response": {"answer": "{{output}}"}

Streaming (SSE):

endpoint = {
    "url": "https://api.example.com/chat",
    "method": "POST",
    "body": {"message": "{{input}}"},
    "response": {"answer": "{{output}}"},
    "streaming": {
        "enabled": True,
        "format": "openai",  # "openai" | "anthropic" | "custom"
        "content_path": "choices[0].delta.content"
    }
}

Multi-turn conversations:

endpoint = {
    "url": "https://api.example.com/chat",
    "method": "POST",
    "body": {"message": "{{input}}", "session_id": "{{session}}"},
    "response": {"answer": "{{output}}"},
    "multiturn": {
        "type": "session_id",  # "none" | "session_id" | "history"
        "field": "conversation_id",
        "location": "body"
    }
}

Generate Evaluation Dataset

Test scenarios can be auto-generated based on your system description, or you can provide custom scenarios.

Auto-generate scenarios:

dataset = client.generate_dataset(system_id, num_scenarios=20)

Provide custom scenarios:

dataset = client.generate_dataset(
    system_id,
    scenarios=[
        {"input": "Where is my order?", "outputs": ["I can help you track your order."]},
        {"input": "I want a refund", "outputs": ["I'll process your refund request."]}
    ]
)

Refine an existing dataset:

dataset = client.update_dataset(
    system_id,
    orientation="Add more edge cases about payment failures"
)

Note: generate_dataset requires a validated system description. If validation fails, a DescriptionNotValidatedError is raised with recommendations.

Run Evaluation

Start an evaluation to test your AI system. By default, the call blocks until the evaluation completes.

Block until complete (default):

result = client.evaluate(system_id)
print(f"Score: {result['summary']}")

Start without waiting:

run_info = client.evaluate(system_id, wait=False)
run_id = run_info["run_id"]

# Check status later
result = client.get_evaluation(run_id)
print(f"Status: {result['status']}")

With specific thematics:

result = client.evaluate(
    system_id,
    thematics_config={
        "explainability": {"justification": {"nb_tests": 5}},
        "robustness": {"prompt_injection": {"nb_tests": 10}}
    }
)

With evaluation profile:

result = client.evaluate(system_id, profile="extended")

With progress callback:

result = client.evaluate(
    system_id,
    poll_interval=10,
    on_poll=lambda status, elapsed: print(f"  {status} ({elapsed}s)")
)

Connectors

Connectors attach external data sources (logs, databases) to your system for richer evaluation context.

File logs:

from mankinds_sdk.connectors import FileConnector

connector = FileConnector(file_path="/path/to/logs.json")
client.add_connector(system_id, connector)

Datadog logs:

from mankinds_sdk.connectors import DatadogConnector

connector = DatadogConnector(
    api_key="dd-api-key",
    app_key="dd-app-key",
    site="datadoghq.eu",  # default
)
client.add_connector(system_id, connector)

SQLite database:

from mankinds_sdk.connectors import SqliteConnector

connector = SqliteConnector(file_path="/path/to/database.db")
client.add_connector(system_id, connector)

PostgreSQL database:

from mankinds_sdk.connectors import PostgresqlConnector

connector = PostgresqlConnector(
    host="localhost",
    database="mydb",
    user="admin",
    password="secret",
    port=5432,
)
client.add_connector(system_id, connector)

Manage connectors:

# List all connectors
connectors = client.get_connectors(system_id)

# Update a connector
connector = FileConnector(file_path="/path/to/new-logs.json")
client.update_connector(system_id, connector)

# Remove a connector
client.delete_connector(system_id, connector)

Only one connector per category (logs, database) is allowed per system. Adding a duplicate raises ConnectorAlreadyExistsError.

Complete Example

from mankinds_sdk import MankindsClient
from mankinds_sdk.connectors import FileConnector

client = MankindsClient(api_key="mk_...")

# Create system
system = client.create_system(
    name="Support Bot",
    description="A customer support chatbot for order tracking and returns.",
    endpoint={
        "url": "https://api.example.com/chat",
        "method": "POST",
        "body": {"message": "{{input}}"},
        "response": {"answer": "{{output}}"}
    }
)
system_id = system["id"]

# Attach production logs
connector = FileConnector(file_path="./logs/production.json")
client.add_connector(system_id, connector)

# Generate dataset and evaluate
dataset = client.generate_dataset(system_id, num_scenarios=15)
result = client.evaluate(system_id, profile="extended")

print(f"Status: {result['status']}")
print(f"Score: {result['summary']}")

API Reference

MankindsClient

Method	Description
get_system(system_id)	Get system details and configuration
create_system(name, description, endpoint)	Create a new AI system
update_system(system_id, name?, description?, endpoint?)	Update an existing system
generate_dataset(system_id, num_scenarios?, scenarios?)	Generate and validate evaluation scenarios
update_dataset(system_id, orientation?, scenarios?)	Refine or replace dataset scenarios
evaluate(system_id, ...)	Run an evaluation
get_evaluation(run_id)	Get evaluation status and results
add_connector(system_id, connector)	Add a data source connector
get_connectors(system_id)	List all connectors for a system
update_connector(system_id, connector)	Update a connector
delete_connector(system_id, connector)	Remove a connector

Exceptions

Exception	When Raised
CredentialsError	Missing API key
AuthenticationError	Invalid or expired API key (401)
NotFoundError	Resource not found (404)
ValidationError	Request validation failed (422)
RateLimitError	Too many requests (429)
ServerError	Server error (5xx)
InvalidEndpointError	Endpoint missing required fields
EndpointNotConfiguredError	Evaluation without endpoint
DescriptionNotValidatedError	Dataset generation before validation
ConnectorAlreadyExistsError	Duplicate connector category

from mankinds_sdk.exceptions import (
    CredentialsError,
    AuthenticationError,
    InvalidEndpointError,
    DescriptionNotValidatedError,
    ConnectorAlreadyExistsError,
)

try:
    result = client.evaluate(system_id)
except AuthenticationError:
    print("Invalid API key")
except InvalidEndpointError as e:
    print(f"Missing fields: {e.missing_fields}")
except DescriptionNotValidatedError as e:
    print(f"Fix description: {e.recommendations}")

---

License

MIT

© 2026 Mankinds. All rights reserved.