Mantisdk

AI Agent Training and Evaluation Platform

Mantisdk is a comprehensive toolkit for training and evaluating AI agents using reinforcement learning, automatic prompt optimization, and supervised fine-tuning.

Core Features

• Turn your agent into an optimizable beast with minimal code changes
• Build with any agent framework (LangChain, OpenAI Agent SDK, AutoGen, CrewAI, and more)
• Selectively optimize one or more agents in a multi-agent system
• Embraces algorithms like Reinforcement Learning, Automatic Prompt Optimization, Supervised Fine-tuning and more

Installation

pip install mantisdk

For optional dependencies:

# For LLMProxy feature (note: has strict dependency pins for boto3, grpcio, uvicorn)
pip install 'mantisdk[proxy]'

# For APO (Automatic Prompt Optimization)
pip install 'mantisdk[apo]'

# For VERL integration
pip install 'mantisdk[verl]'

# For Weave integration
pip install 'mantisdk[weave]'

# For MongoDB store
pip install 'mantisdk[mongo]'

Quick Start

import mantisdk as msk

# Initialize the client
client = msk.MantisdkClient()

# Your agent code here...

Insight Ingestion Helpers

Use these helpers when you want to send external agent traces/scores directly to an Insight instance.

1) Insight ingestion client

from mantisdk import InsightIngestionClient, InsightIngestionConfig

client = InsightIngestionClient(
    InsightIngestionConfig(
        host="http://localhost:3000",
        public_key="pk-lf-...",
        secret_key="sk-lf-...",
    )
)

# Send trace event
client.ingest_trace_event(
    trace_id="trace-001",
    name="Agent Trace",
    input_data={"prompt": "hello"},
    output_data={"answer": "world"},
    session_id="session-001",
    environment="default",
)

# Send scores efficiently in batches
client.send_scores(
    [
        {"name": "task_success", "value": 0.95, "dataType": "NUMERIC", "traceId": "trace-001"},
        {"name": "safety", "value": 1, "dataType": "BOOLEAN", "traceId": "trace-001"},
    ],
    mode="ingestion-batch",
)

2) Simple API wrapper sample (dependency-free)

For a minimal stdlib-only helper and examples, see: samples/insight-agent-trace-wrapper/.

CLI Usage

# Start the Mantisdk server
msk store serve

# Run with vLLM
msk vllm start

Documentation

For full documentation, visit https://withmetis.github.io/mantis/mantisdk/

License

MIT License - see LICENSE for details.