arize-phoenix-evals
Phoenix Evals provides lightweight, composable building blocks for writing and running evaluations on LLM applications, including tools to determine relevance, toxicity, hallucination detection, and much more.
Features
- Works with your preferred model SDKs via adapters (OpenAI, LiteLLM, LangChain)
- Powerful input mapping and binding for working with complex data structures
- Several pre-built metrics for common evaluation tasks like hallucination detection
- Evaluators are natively instrumented via OpenTelemetry tracing for observability and dataset curation
- Blazing fast performance - achieve up to 20x speedup with built-in concurrency and batching
- Tons of convenience features to improve the developer experience!
Installation
Install Phoenix Evals 2.0 using pip:
pip install 'arize-phoenix-evals>=2.0.0' openai
Quick Start
from phoenix.evals import create_classifier
from phoenix.evals.llm import LLM
# Create an LLM instance
llm = LLM(provider="openai", model="gpt-4o")
# Create an evaluator
evaluator = create_classifier(
name="helpfulness",
prompt_template="Rate the response to the user query as helpful or not:\n\nQuery: {input}\nResponse: {output}",
llm=llm,
choices={"helpful": 1.0, "not_helpful": 0.0},
)
# Simple evaluation
scores = evaluator.evaluate({"input": "How do I reset?", "output": "Go to settings > reset."})
scores[0].pretty_print()
# With input mapping for nested data
scores = evaluator.evaluate(
{"data": {"query": "How do I reset?", "response": "Go to settings > reset."}},
input_mapping={"input": "data.query", "output": "data.response"},
)
scores[0].pretty_print()
Pre-Built Evaluators
The phoenix.evals.metrics module provides ready-to-use evaluators for common tasks:
| Evaluator | Class | Description |
|---|---|---|
| Faithfulness | FaithfulnessEvaluator |
Detects hallucinations — checks if output is grounded in context |
| Completeness | CompletenessEvaluator |
Checks whether every active user request in a conversation was actually completed |
| Conciseness | ConcisenessEvaluator |
Evaluates whether the response is appropriately concise |
| Correctness | CorrectnessEvaluator |
Checks if the output is factually correct |
| Retrieval Relevance | RetrievalRelevanceEvaluator |
Measures how relevant retrieved information is to a request |
| Refusal | RefusalEvaluator |
Detects whether the model refused to answer |
| Tool Invocation | ToolInvocationEvaluator |
Checks whether the correct tool was called with the right arguments |
| Tool Selection | ToolSelectionEvaluator |
Evaluates whether the right tool was selected for the task |
| Tool Response Handling | ToolResponseHandlingEvaluator |
Evaluates how well the model uses a tool's response |
| User Friction | UserFrictionEvaluator |
Detects expressed corrections, retries, frustration, and challenges |
| PII Detection | PiiDetectionEvaluator |
Screens a conversation record for personally identifiable information |
| Exact Match | exact_match |
Checks for exact string equality between output and expected |
| Regex Match | MatchesRegex |
Checks whether the output matches a regular expression |
| Precision/Recall | PrecisionRecallFScore |
Computes precision, recall, and F-score for classification tasks |
from phoenix.evals.llm import LLM
from phoenix.evals.metrics import FaithfulnessEvaluator, exact_match, MatchesRegex
llm = LLM(provider="openai", model="gpt-4o")
# LLM-powered faithfulness evaluator
faithfulness = FaithfulnessEvaluator(llm=llm)
scores = faithfulness.evaluate(
{
"input": "What is the capital of France?",
"context": "Paris is the capital of France.",
"output": "The capital of France is Berlin.",
}
)
scores[0].pretty_print()
# Score(name='faithfulness', score=0.0, label='unfaithful', explanation='...')
# Code-based exact match
match_result = exact_match({"output": "Paris", "expected": "Paris"})
# Regex match
regex_result = MatchesRegex(pattern=r"^\d{4}-\d{2}-\d{2}$").evaluate({"output": "2024-03-15"})
LLM Providers
The LLM class supports multiple AI providers:
from phoenix.evals.llm import LLM
# OpenAI
llm = LLM(provider="openai", model="gpt-4o")
# Anthropic
llm = LLM(provider="anthropic", model="claude-3-5-sonnet-20241022")
# Google Gemini
llm = LLM(provider="google", model="gemini-1.5-pro")
# LiteLLM (unified interface for 100+ providers)
llm = LLM(provider="litellm", model="gpt-4o")
Evaluating Dataframes
import pandas as pd
from phoenix.evals import create_classifier, evaluate_dataframe, async_evaluate_dataframe
from phoenix.evals.llm import LLM
# Create an LLM instance
llm = LLM(provider="openai", model="gpt-4o")
# Create multiple evaluators
relevance_evaluator = create_classifier(
name="relevance",
prompt_template="Is the response relevant to the query?\n\nQuery: {input}\nResponse: {output}",
llm=llm,
choices={"relevant": 1.0, "irrelevant": 0.0},
)
helpfulness_evaluator = create_classifier(
name="helpfulness",
prompt_template="Is the response helpful?\n\nQuery: {input}\nResponse: {output}",
llm=llm,
choices={"helpful": 1.0, "not_helpful": 0.0},
)
# Prepare your dataframe
df = pd.DataFrame(
[
{
"input": "How do I reset my password?",
"output": "Go to settings > account > reset password.",
},
{"input": "What's the weather like?", "output": "I can help you with password resets."},
]
)
# Synchronous evaluation
results_df = evaluate_dataframe(
dataframe=df,
evaluators=[relevance_evaluator, helpfulness_evaluator],
)
print(results_df.head())
# Async evaluation (up to 20x faster with large dataframes)
import asyncio
results_df = asyncio.run(
async_evaluate_dataframe(
dataframe=df,
evaluators=[relevance_evaluator, helpfulness_evaluator],
)
)
Documentation
- Full Documentation - Complete API reference and guides
- Phoenix Docs - Detailed use-cases and examples
- OpenInference - Auto-instrumentation libraries for frameworks
Community
Join our community to connect with thousands of AI builders:
- 🌍 Join our Slack community.
- 📚 Read the Phoenix documentation.
- 💡 Ask questions and provide feedback in the #phoenix-support channel.
- 🌟 Leave a star on our GitHub.
- 🐞 Report bugs with GitHub Issues.
- 𝕏 Follow us on 𝕏.
- 💼 Follow us on LinkedIn.
- 🗺️ Check out our roadmap to see where we're heading next.
1 maintainer
Metadata
1 maintainer
Release files for arize-phoenix-evals 3.8.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| arize_phoenix_evals-3.8.0.tar.gz | 109.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| arize_phoenix_evals-3.8.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 254.6 kB
Release files / arize_phoenix_evals-3.8.0.tar.gz
| Download URL | arize_phoenix_evals-3.8.0.tar.gz |
|---|---|
| Size | 109.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b605d88852323805821a99961387eb2fa0fa1cc9f5c798c991578e305380ab04
|
|
BLAKE2b-256 checksum How to use checksums |
0cea3acbe860fc701a537c811179080ea09136df7f15ecfed8a453bf9336c64e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 11, 2026.
Transparency logRelease files / arize_phoenix_evals-3.8.0-py3-none-any.whl
| Download URL | arize_phoenix_evals-3.8.0-py3-none-any.whl |
|---|---|
| Size | 144.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e25b39afa03f9c26ea872898bee13b9610e902643c04fa6d7b175e40d6fbef78
|
|
BLAKE2b-256 checksum How to use checksums |
68f750ea514a5e393bfcb0b8586d17f58b5d6b7819c92ba015fee642aef990f6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 11, 2026.
Transparency log
