llm-behave
Behavioral testing for LLM applications. A pytest plugin.
No judge LLM. No API cost. Offline models. Works with any provider.
def test_rag_answer():
context = retriever.get_docs("refund policy")
answer = my_rag_app("How long do I have to return something?")
assert_behavior(answer) \
.grounded_in(context) \ # every claim entailed by context
.no_hallucination(context) \ # flags unsupported claims
.tone("empathetic") \
.not_mentions("competitor")
Why llm-behave?
Most LLM testing tools either:
- Use another LLM to judge the output — Ragas, DeepEval, etc. need an API key and call a judge LLM (expensive, slow, circular)
- Only do exact string matching (misses semantic meaning)
- Don't support multi-turn conversations at all
llm-behave uses small offline transformer models (80MB embeddings + a small NLI model, both CPU) to understand meaning — no API calls, no API key, no cost, no internet required during tests. That includes RAG grounding and hallucination detection: grounded_in() / no_hallucination() check every claim against retrieved context using the same offline NLI model, no judge LLM required.
Install
# Core (tool call assertions, structure tests — no ML deps)
pip install llm-behave
# Full (semantic assertions: mentions, tone, intent, drift)
pip install llm-behave[semantic]
Features at a glance
| Feature | What it does |
|---|---|
grounded_in() |
Assert every claim is entailed by the retrieved context (offline RAG eval) |
no_hallucination() |
Assert the output adds no claims unsupported by the context |
cites() |
Assert the output references a given source id / URL / marker |
mentions() |
Semantic similarity — "money back" matches "refund" |
not_mentions() |
Assert a topic is NOT brought up |
tone() |
Detect empathetic / professional / rude / helpful etc. |
intent() |
Does the response intend to help? refuse? apologize? |
calls_tool() |
Assert which tool the LLM called |
contradicts() |
Assert the output contradicts a reference statement (NLI) |
returns_valid_json() |
Assert the output parses as JSON (tolerates ```json fences) |
matches_schema() |
Validate output JSON against a Pydantic model |
has_keys() |
Assert the output JSON object contains required keys |
ConversationTest |
Multi-turn testing with memory, contradiction detection |
DriftTest |
Save baseline behavior, detect regressions in CI |
Usage
Basic assertions
from llm_behave import assert_behavior
output = my_llm("I want a refund")
# Semantic match — not exact string
assert_behavior(output).mentions("refund policy")
# Tone detection
assert_behavior(output).tone("empathetic")
assert_behavior(output).tone("professional", threshold=0.6)
# Intent
assert_behavior(output).intent("offering to help the customer")
# Negative assertions
assert_behavior(output).not_mentions("competitor")
# Fluent chaining
assert_behavior(output) \
.mentions("refund") \
.tone("empathetic") \
.not_mentions("competitor")
When mentions(), not_mentions(), or intent() fail, the error shows the top-3 closest sentences with their similarity scores — so you can see whether it was a near-miss (0.41) or nowhere close (0.05), instead of guessing from a single number.
Contradiction assertions
Assert that an output contradicts a previous statement — useful for detecting policy reversals across conversation turns.
# Turn 1 said refunds are available. Does turn 3 contradict that?
assert_behavior(turn_3_response).contradicts("Refunds are always available within 30 days.")
RAG / grounding assertions
Verify a RAG answer is actually supported by the retrieved context — fully offline, no judge LLM, no API cost. Uses the same NLI model as contradicts().
context = retriever.get_docs("refund policy") # your retrieved chunks
answer = my_rag_app("How long do I have to return something?")
# Every claim in the answer must be entailed by the context
assert_behavior(answer).grounded_in(context)
# No claim may go beyond the context (catches hallucinations)
assert_behavior(answer).no_hallucination(context)
# The answer must cite the expected source
assert_behavior(answer).cites("doc-42")
On failure, no_hallucination() lists the specific unsupported sentences and their entailment scores, so a green check is trustworthy.
Structured-output assertions
For LLMs that emit JSON. Pure Python — no model download needed.
from pydantic import BaseModel
class Order(BaseModel):
order_id: str
total: float
shipped: bool
output = my_llm("Return the order as JSON")
assert_behavior(output).returns_valid_json()
assert_behavior(output).has_keys(["order_id", "total"])
assert_behavior(output).matches_schema(Order)
Tool call assertions
text, tool_calls = my_llm.chat_with_tools(messages, tools=my_tools)
assert_behavior(text, tool_calls) \
.calls_tool("lookup_order") \
.mentions("order")
Multi-turn conversation testing
from llm_behave import ConversationTest, MockProvider
conv = ConversationTest(agent=my_agent)
conv.say("Hi, my name is Alex")
conv.say("I placed order #5678 last week")
response = conv.say("When will it arrive?")
# Does it remember context from earlier turns?
assert response.recalls("order")
assert response.recalls("Alex")
# Did it reverse what it said in turn 2? (returns a bool -- wrap in assert)
assert not response.contradicts_turn(2)
# Is tone consistent across the whole conversation?
# Note: currently compares overall meaning, so topic changes also lower the score.
assert response.consistent_tone_across_turns(threshold=0.6)
Drift detection (for CI)
Catch silent regressions when you update your model or prompts.
from llm_behave import DriftTest
# First run: save baseline
@DriftTest.baseline(save_as="support_refund_flow")
def get_baseline_output():
return my_llm("I need a refund")
# Every CI run: compare against baseline
result = DriftTest.compare("support_refund_flow", current_output)
assert result.passed, f"Behavior drift detected: {result.details}"
pytest fixtures (auto-registered)
# These fixtures are available in any test file automatically
def test_with_mock(mock_provider, assert_llm):
provider = mock_provider(responses=["I'll help with your refund right away."])
output = provider.chat([{"role": "user", "content": "refund please"}])
assert_llm(output).mentions("refund").tone("helpful")
def test_conversation(conversation):
conv = conversation(responses=["Hello!", "Sure, I can help with that."])
conv.say("Hi")
response = conv.say("I need help")
assert "help" in response.text.lower()
Providers
Built-in adapters for all major LLM providers:
from llm_behave.providers.openai_adapter import OpenAIProvider
from llm_behave.providers.anthropic_adapter import AnthropicProvider
from llm_behave.providers.ollama_adapter import OllamaProvider
from llm_behave import MockProvider # for tests, no API calls
# All providers have the same interface
provider = OpenAIProvider(model="gpt-4o-mini")
provider = AnthropicProvider(model="claude-haiku-4-5")
provider = OllamaProvider(model="llama3")
output = provider.chat([{"role": "user", "content": "Hello"}])
text, tool_calls = provider.chat_with_tools(messages, tools=my_tools)
Bring your own provider by subclassing LLMProvider:
from llm_behave.providers.base import LLMProvider
class MyProvider(LLMProvider):
def chat(self, messages, **kwargs):
...
def chat_with_tools(self, messages, tools, **kwargs):
...
How it works
llm-behave uses all-MiniLM-L6-v2 — an 80MB sentence-transformer model that runs fully offline on CPU.
mentions()/not_mentions()— splits text into sentences, computes max cosine similarity between any sentence and your concepttone()— batch-encodes input text against example sentences for each tone, returns max similarityintent()— semantic similarity between output and your intent descriptioncontradicts()— NLI (Natural Language Inference) model detects if the output contradicts a reference statementcontradicts_turn()— same NLI model, applied across conversation turnsgrounded_in()/no_hallucination()— the same NLI model, run the other direction: each output sentence must be entailed by some context sentence, else it's flagged as ungroundedreturns_valid_json()/matches_schema()/has_keys()— pure Python + Pydantic, no model needed
Models load lazily on first use and are cached for the rest of the test session. Import time stays fast.
Performance
Measured after model warmup (model loads once per test session):
| Assertion | Time |
|---|---|
mentions() |
~32ms |
tone() |
~40ms |
intent() |
~32ms |
| 4-assertion chain | ~350ms |
pytest markers
import pytest
@pytest.mark.behavioral
def test_refund_flow():
...
@pytest.mark.drift
def test_no_regression():
...
Run only behavioral tests:
pytest -m behavioral
pytest -m drift
Full install options
pip install llm-behave # core only
pip install llm-behave[semantic] # + sentence-transformers + torch
pip install llm-behave[openai] # + openai SDK
pip install llm-behave[anthropic] # + anthropic SDK
pip install llm-behave[ollama] # + ollama SDK
pip install llm-behave[all] # everything
Requirements
- Python 3.10+
- pytest 7.0+
- For semantic assertions:
pip install llm-behave[semantic]
License
MIT — free to use in personal and commercial projects.
Author
Built by Swanand Potnis — Pune, India.
Metadata
Release files for llm-behave 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llm_behave-0.2.1.tar.gz | 40.2 kB | Without provenance | Details for llm_behave-0.2.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llm_behave-0.2.1-py3-none-any.whl | Python 3 | none | any | Details for llm_behave-0.2.1-py3-none-any.whl |
Total release size: 68.0 kB
Release files / llm_behave-0.2.1.tar.gz
| Download URL | llm_behave-0.2.1.tar.gz |
|---|---|
| Size | 40.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7adcee90fd216e3a04ea18e176f4fef14c2316528ba65effae448f6d8e769522
|
|
BLAKE2b-256 checksum How to use checksums |
84ab9bbab2460c561615f9e2f5ce6db647f664f4d28716325cf17b579618306d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.15
|
Release files / llm_behave-0.2.1-py3-none-any.whl
| Download URL | llm_behave-0.2.1-py3-none-any.whl |
|---|---|
| Size | 27.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4ccb1e74c2884f43e66e7922c646541c90970ef15e42781b8cb55c11113ad69b
|
|
BLAKE2b-256 checksum How to use checksums |
eeb37037a59af284092279bfd6ae1bb615b90e7f8fbb10057e215db51fd60697
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.15
|
Security for llm-behave 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.