Agent Behavior Monitoring (ABM)
Track and judge any agent behavior in online and offline setups. Set up Sentry-style alerts and analyze agent behaviors / topic patterns at scale!
[NEW] 🎆 Agent Reinforcement Learning
Train your agents with multi-turn reinforcement learning using judgeval and Fireworks AI! Judgeval's ABM now integrates with Fireworks' Reinforcement Fine-Tuning (RFT) endpoint, supporting gpt-oss, qwen3, Kimi2, DeepSeek, and more.
Judgeval's agent monitoring infra provides a simple harness for integrating GRPO into any Python agent, giving builders a quick method to try RL with minimal code changes to their existing agents!
await trainer.train(
agent_function=your_agent_function, # entry point to your agent
scorers=[RewardScorer()], # Custom scorer you define based on task criteria, acts as reward
prompts=training_prompts # Tasks
)
That's it! Judgeval automatically manages trajectory collection and reward tagging - your agent can learn from production data with minimal code changes.
👉 Check out the Wikipedia Racer notebook, where an agent learns to navigate Wikipedia using RL, to see Judgeval in action.
You can view and monitor training progress for free via the Judgment Dashboard.
Judgeval Overview
Judgeval is an open-source framework for agent behavior monitoring. Judgeval offers a toolkit to track and judge agent behavior in online and offline setups, enabling you to convert interaction data from production/test environments into improved agents. To get started, try running one of the notebooks below or dive deeper in our docs.
Our mission is to unlock the power of production data for agent development, enabling teams to improve their apps by catching real-time failures and optimizing over their users' preferences.
📚 Cookbooks
| Try Out | Notebook | Description |
|---|---|---|
| RL | Wikipedia Racer | Train agents with reinforcement learning |
| Online ABM | Research Agent | Monitor agent behavior in production |
| Custom Scorers | HumanEval | Build custom evaluators for your agents |
| Offline Testing | [Get Started For Free] | Compare how different prompts, models, or agent configs affect performance across ANY metric |
You can access our repo of cookbooks.
You can find a list of video tutorials for Judgeval use cases.
Why Judgeval?
🤖 Simple to run multi-turn RL: Optimize your agents with multi-turn RL without managing compute infrastructure or data pipelines. Just add a few lines of code to your existing agent code and train!
⚙️ Custom Evaluators: No restriction to only monitoring with prefab scorers. Judgeval provides simple abstractions for custom Python scorers, supporting any LLM-as-a-judge rubrics/models and code-based scorers that integrate to our live agent-tracking infrastructure. Learn more
🚨 Production Monitoring: Run any custom scorer in a hosted, virtualized secure container to flag agent behaviors online in production. Get Slack alerts for failures and add custom hooks to address regressions before they impact users. Learn more
📊 Behavior/Topic Grouping: Group agent runs by behavior type or topic for deeper analysis. Drill down into subsets of users, agents, or use cases to reveal patterns of agent behavior.
🧪 Run experiments on your agents: Compare test different prompts, models, or agent configs across customer segments. Measure which changes improve agent performance and decrease bad agent behaviors.
🛠️ Quickstart
Get started with Judgeval by installing our SDK using pip:
pip install judgeval
Ensure you have your JUDGMENT_API_KEY and JUDGMENT_ORG_ID environment variables set to connect to the Judgment Platform.
export JUDGMENT_API_KEY=...
export JUDGMENT_ORG_ID=...
If you don't have keys, create an account for free on the platform!
Start monitoring with Judgeval
from judgeval.tracer import Tracer, wrap
from judgeval.data import Example
from judgeval.scorers import AnswerRelevancyScorer
from openai import OpenAI
judgment = Tracer(project_name="default_project")
client = wrap(OpenAI()) # tracks all LLM calls
@judgment.observe(span_type="tool")
def format_question(question: str) -> str:
# dummy tool
return f"Question : {question}"
@judgment.observe(span_type="function")
def run_agent(prompt: str) -> str:
task = format_question(prompt)
response = client.chat.completions.create(
model="gpt-5-mini",
messages=[{"role": "user", "content": task}]
)
judgment.async_evaluate( # trigger online monitoring
scorer=AnswerRelevancyScorer(threshold=0.5), # swap with any scorer
example=Example(input=task, actual_output=response), # customize to your data
model="gpt-5",
)
return response.choices[0].message.content
run_agent("What is the capital of the United States?")
Running this code will deliver monitoring results to your free platform account and should look like this:
Customizable Scorers Over Agent Behavior
Judgeval's strongest suit is the full customization over the types of scorers you can run online monitoring with. No restrictions to only single-prompt LLM judges or prefab scorers - if you can express your scorer in python code, judgeval can monitor it! Under the hood, judgeval hosts your scorer in a virtualized secure container, enabling online monitoring for any scorer.
First, create a behavior scorer in a file called helpfulness_scorer.py:
from judgeval.data import Example
from judgeval.scorers.example_scorer import ExampleScorer
# Define custom example class
class QuestionAnswer(Example):
question: str
answer: str
# Define a server-hosted custom scorer
class HelpfulnessScorer(ExampleScorer):
name: str = "Helpfulness Scorer"
server_hosted: bool = True # Enable server hosting
async def a_score_example(self, example: QuestionAnswer):
# Custom scoring logic for agent behavior
# Can be an arbitrary combination of code and LLM calls
if len(example.answer) > 10 and "?" not in example.answer:
self.reason = "Answer is detailed and provides helpful information"
return 1.0
else:
self.reason = "Answer is too brief or unclear"
return 0.0
Then deploy your scorer to Judgment's infrastructure:
echo "pydantic" > requirements.txt
uv run judgeval upload_scorer helpfulness_scorer.py requirements.txt
Now you can instrument your agent with monitoring and online evaluation:
from judgeval.tracer import Tracer, wrap
from helpfulness_scorer import HelpfulnessScorer, QuestionAnswer
from openai import OpenAI
judgment = Tracer(project_name="default_project")
client = wrap(OpenAI()) # tracks all LLM calls
@judgment.observe(span_type="tool")
def format_task(question: str) -> str: # replace with your prompt engineering
return f"Please answer the following question: {question}"
@judgment.observe(span_type="tool")
def answer_question(prompt: str) -> str: # replace with your LLM system calls
response = client.chat.completions.create(
model="gpt-5-mini",
messages=[{"role": "user", "content": prompt}]
)
return response.choices[0].message.content
@judgment.observe(span_type="function")
def run_agent(question: str) -> str:
task = format_task(question)
answer = answer_question(task)
# Add online evaluation with server-hosted scorer
judgment.async_evaluate(
scorer=HelpfulnessScorer(),
example=QuestionAnswer(question=question, answer=answer),
sampling_rate=0.9 # Evaluate 90% of agent runs
)
return answer
if __name__ == "__main__":
result = run_agent("What is the capital of the United States?")
print(result)
Congratulations! Your online eval result should look like this:
You can now run any online scorer in a secure Firecracker microVMs with no latency impact on your applications.
Judgeval is created and maintained by Judgment Labs.
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file judgeval-0.32.2.tar.gz.
File metadata
- Download URL: judgeval-0.32.2.tar.gz
- Upload date:
- Size: 23.2 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.11.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4e02d7cbce3de52736645d591f60cb5899d310c6708cd2f219dc417b3b11b5ac
|
|
| MD5 |
1ab5def70475ae6dc944ef2b0cef0e8c
|
|
| BLAKE2b-256 |
9fa01303e9e376ffa92997cf2b15c8ee6e8d1472c950d400d28f4d72aac264ea
|
File details
Details for the file judgeval-0.32.2-py3-none-any.whl.
File metadata
- Download URL: judgeval-0.32.2-py3-none-any.whl
- Upload date:
- Size: 131.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.11.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d0f725a0758966b74e47b2e845fa3850acd1a0dc32f6103776586966be924fa6
|
|
| MD5 |
81f6afaf2ac6f7fab25fbdc042d70136
|
|
| BLAKE2b-256 |
68d89ad82a093ec31268e83a99e4cba178ddde53bcf2c71c412be4e1a2f2c249
|
