Patronus Python SDK
The Patronus Python SDK is a Python library for systematic evaluation of Large Language Models (LLMs). Build, test, and improve your LLM applications with customizable tasks, evaluators, and comprehensive experiment tracking.
Note: This library is currently in beta and is not stable. The APIs may change in future releases.
Documentation
For detailed documentation, including API references and advanced usage, please visit our documentation.
Installation
pip install patronus
Quickstart
Experiment
import os
from patronus import Client, Row, TaskResult, evaluator, task
client = Client(
# This is the default and can be omitted
api_key=os.environ.get("PATRONUS_API_KEY"),
)
@task
def my_task(row: Row):
return f"{row.evaluated_model_input} World"
@evaluator
def exact_match(row: Row, task_result: TaskResult):
# exact_match is locally defined and run evaluator
return task_result.evaluated_model_output == row.evaluated_model_gold_answer
# Reference remote Judge Patronus Evaluator with is-concise criteria.
# This evaluator runs remotely on Patronus infrastructure.
is_concise = client.remote_evaluator("judge", "patronus:is-concise")
client.experiment(
"Tutorial Project",
dataset=[
{
"evaluated_model_input": "Hello",
"evaluated_model_gold_answer": "Hello World",
},
],
task=my_task,
evaluators=[exact_match, is_concise],
)
3 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file patronus-0.0.12.tar.gz.
File metadata
- Download URL: patronus-0.0.12.tar.gz
- Upload date:
- Size: 23.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.8.3 CPython/3.12.5 Darwin/23.4.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
31ed4f39e2e0dbdf57e832e14f846ebb9f868c334958391b4cedbf7b557e85ba
|
|
| MD5 |
ce48c371ad029caec3109147cc157a55
|
|
| BLAKE2b-256 |
26e045c5978f50dceb50e3279747c2682f383de70db031ed665b5ff228612975
|
File details
Details for the file patronus-0.0.12-py3-none-any.whl.
File metadata
- Download URL: patronus-0.0.12-py3-none-any.whl
- Upload date:
- Size: 28.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.8.3 CPython/3.12.5 Darwin/23.4.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3e45d2e131fbda84436fe1ab13a13b793a2de2b3cb887364b5bdf07550705703
|
|
| MD5 |
75c88a553c1df8c45e5fc398fdcb76c4
|
|
| BLAKE2b-256 |
41b613bd799b65104121526323a959ac47922abaa14629a4bd4b0781c2444cd2
|
