Skip to main content

llm-behave

PyPI version Python versions License: MIT Tests

Behavioral testing for LLM applications. A pytest plugin.

No judge LLM. No API cost. Offline models. Works with any provider.

def test_rag_answer():
    context = retriever.get_docs("refund policy")
    answer = my_rag_app("How long do I have to return something?")

    assert_behavior(answer) \
        .grounded_in(context) \          # every claim entailed by context
        .no_hallucination(context) \     # flags unsupported claims
        .tone("empathetic") \
        .not_mentions("competitor")

Why llm-behave?

Most LLM testing tools either:

  • Use another LLM to judge the output — Ragas, DeepEval, etc. need an API key and call a judge LLM (expensive, slow, circular)
  • Only do exact string matching (misses semantic meaning)
  • Don't support multi-turn conversations at all

llm-behave uses small offline transformer models (80MB embeddings + a small NLI model, both CPU) to understand meaning — no API calls, no API key, no cost, no internet required during tests. That includes RAG grounding and hallucination detection: grounded_in() / no_hallucination() check every claim against retrieved context using the same offline NLI model, no judge LLM required.


Install

# Core (tool call assertions, structure tests — no ML deps)
pip install llm-behave

# Full (semantic assertions: mentions, tone, intent, drift)
pip install llm-behave[semantic]

Features at a glance

Feature What it does
grounded_in() Assert every claim is entailed by the retrieved context (offline RAG eval)
no_hallucination() Assert the output adds no claims unsupported by the context
cites() Assert the output references a given source id / URL / marker
mentions() Semantic similarity — "money back" matches "refund"
not_mentions() Assert a topic is NOT brought up
tone() Detect empathetic / professional / rude / helpful etc.
intent() Does the response intend to help? refuse? apologize?
calls_tool() Assert which tool the LLM called
contradicts() Assert the output contradicts a reference statement (NLI)
returns_valid_json() Assert the output parses as JSON (tolerates ```json fences)
matches_schema() Validate output JSON against a Pydantic model
has_keys() Assert the output JSON object contains required keys
ConversationTest Multi-turn testing with memory, contradiction detection
DriftTest Save baseline behavior, detect regressions in CI

Usage

Basic assertions

from llm_behave import assert_behavior

output = my_llm("I want a refund")

# Semantic match — not exact string
assert_behavior(output).mentions("refund policy")

# Tone detection
assert_behavior(output).tone("empathetic")
assert_behavior(output).tone("professional", threshold=0.6)

# Intent
assert_behavior(output).intent("offering to help the customer")

# Negative assertions
assert_behavior(output).not_mentions("competitor")

# Fluent chaining
assert_behavior(output) \
    .mentions("refund") \
    .tone("empathetic") \
    .not_mentions("competitor")

When mentions(), not_mentions(), or intent() fail, the error shows the top-3 closest sentences with their similarity scores — so you can see whether it was a near-miss (0.41) or nowhere close (0.05), instead of guessing from a single number.

Contradiction assertions

Assert that an output contradicts a previous statement — useful for detecting policy reversals across conversation turns.

# Turn 1 said refunds are available. Does turn 3 contradict that?
assert_behavior(turn_3_response).contradicts("Refunds are always available within 30 days.")

RAG / grounding assertions

Verify a RAG answer is actually supported by the retrieved context — fully offline, no judge LLM, no API cost. Uses the same NLI model as contradicts().

context = retriever.get_docs("refund policy")   # your retrieved chunks
answer = my_rag_app("How long do I have to return something?")

# Every claim in the answer must be entailed by the context
assert_behavior(answer).grounded_in(context)

# No claim may go beyond the context (catches hallucinations)
assert_behavior(answer).no_hallucination(context)

# The answer must cite the expected source
assert_behavior(answer).cites("doc-42")

On failure, no_hallucination() lists the specific unsupported sentences and their entailment scores, so a green check is trustworthy.

Structured-output assertions

For LLMs that emit JSON. Pure Python — no model download needed.

from pydantic import BaseModel

class Order(BaseModel):
    order_id: str
    total: float
    shipped: bool

output = my_llm("Return the order as JSON")

assert_behavior(output).returns_valid_json()
assert_behavior(output).has_keys(["order_id", "total"])
assert_behavior(output).matches_schema(Order)

Tool call assertions

text, tool_calls = my_llm.chat_with_tools(messages, tools=my_tools)

assert_behavior(text, tool_calls) \
    .calls_tool("lookup_order") \
    .mentions("order")

Multi-turn conversation testing

from llm_behave import ConversationTest, MockProvider

conv = ConversationTest(agent=my_agent)

conv.say("Hi, my name is Alex")
conv.say("I placed order #5678 last week")
response = conv.say("When will it arrive?")

# Does it remember context from earlier turns?
assert response.recalls("order")
assert response.recalls("Alex")

# Did it reverse what it said in turn 2? (returns a bool -- wrap in assert)
assert not response.contradicts_turn(2)

# Is tone consistent across the whole conversation?
# Note: currently compares overall meaning, so topic changes also lower the score.
assert response.consistent_tone_across_turns(threshold=0.6)

Drift detection (for CI)

Catch silent regressions when you update your model or prompts.

from llm_behave import DriftTest

# First run: save baseline
@DriftTest.baseline(save_as="support_refund_flow")
def get_baseline_output():
    return my_llm("I need a refund")

# Every CI run: compare against baseline
result = DriftTest.compare("support_refund_flow", current_output)
assert result.passed, f"Behavior drift detected: {result.details}"

pytest fixtures (auto-registered)

# These fixtures are available in any test file automatically

def test_with_mock(mock_provider, assert_llm):
    provider = mock_provider(responses=["I'll help with your refund right away."])
    output = provider.chat([{"role": "user", "content": "refund please"}])
    assert_llm(output).mentions("refund").tone("helpful")

def test_conversation(conversation):
    conv = conversation(responses=["Hello!", "Sure, I can help with that."])
    conv.say("Hi")
    response = conv.say("I need help")
    assert "help" in response.text.lower()

Providers

Built-in adapters for all major LLM providers:

from llm_behave.providers.openai_adapter import OpenAIProvider
from llm_behave.providers.anthropic_adapter import AnthropicProvider
from llm_behave.providers.ollama_adapter import OllamaProvider
from llm_behave import MockProvider  # for tests, no API calls

# All providers have the same interface
provider = OpenAIProvider(model="gpt-4o-mini")
provider = AnthropicProvider(model="claude-haiku-4-5")
provider = OllamaProvider(model="llama3")

output = provider.chat([{"role": "user", "content": "Hello"}])
text, tool_calls = provider.chat_with_tools(messages, tools=my_tools)

Bring your own provider by subclassing LLMProvider:

from llm_behave.providers.base import LLMProvider

class MyProvider(LLMProvider):
    def chat(self, messages, **kwargs):
        ...
    def chat_with_tools(self, messages, tools, **kwargs):
        ...

How it works

llm-behave uses all-MiniLM-L6-v2 — an 80MB sentence-transformer model that runs fully offline on CPU.

  • mentions() / not_mentions() — splits text into sentences, computes max cosine similarity between any sentence and your concept
  • tone() — batch-encodes input text against example sentences for each tone, returns max similarity
  • intent() — semantic similarity between output and your intent description
  • contradicts() — NLI (Natural Language Inference) model detects if the output contradicts a reference statement
  • contradicts_turn() — same NLI model, applied across conversation turns
  • grounded_in() / no_hallucination() — the same NLI model, run the other direction: each output sentence must be entailed by some context sentence, else it's flagged as ungrounded
  • returns_valid_json() / matches_schema() / has_keys() — pure Python + Pydantic, no model needed

Models load lazily on first use and are cached for the rest of the test session. Import time stays fast.


Performance

Measured after model warmup (model loads once per test session):

Assertion Time
mentions() ~32ms
tone() ~40ms
intent() ~32ms
4-assertion chain ~350ms

pytest markers

import pytest

@pytest.mark.behavioral
def test_refund_flow():
    ...

@pytest.mark.drift
def test_no_regression():
    ...

Run only behavioral tests:

pytest -m behavioral
pytest -m drift

Full install options

pip install llm-behave                          # core only
pip install llm-behave[semantic]                # + sentence-transformers + torch
pip install llm-behave[openai]                  # + openai SDK
pip install llm-behave[anthropic]               # + anthropic SDK
pip install llm-behave[ollama]                  # + ollama SDK
pip install llm-behave[all]                     # everything

Requirements

  • Python 3.10+
  • pytest 7.0+
  • For semantic assertions: pip install llm-behave[semantic]

License

MIT — free to use in personal and commercial projects.


Author

Built by Swanand Potnis — Pune, India.

Metadata

Release files for llm-behave 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llm-behave 0.2.1
File Size Uploaded Provenance Actions
llm_behave-0.2.1.tar.gz 40.2 kB Without provenance Details for llm_behave-0.2.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for llm-behave 0.2.1
File Interpreter ABI Platform Actions
llm_behave-0.2.1-py3-none-any.whl Python 3 none any Details for llm_behave-0.2.1-py3-none-any.whl

Total release size: 68.0 kB

Release files / llm_behave-0.2.1.tar.gz

Download URL llm_behave-0.2.1.tar.gz
Size 40.2 kB
Tags Source
SHA-256 checksum
How to use checksums
7adcee90fd216e3a04ea18e176f4fef14c2316528ba65effae448f6d8e769522
BLAKE2b-256 checksum
How to use checksums
84ab9bbab2460c561615f9e2f5ce6db647f664f4d28716325cf17b579618306d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / llm_behave-0.2.1-py3-none-any.whl

Download URL llm_behave-0.2.1-py3-none-any.whl
Size 27.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4ccb1e74c2884f43e66e7922c646541c90970ef15e42781b8cb55c11113ad69b
BLAKE2b-256 checksum
How to use checksums
eeb37037a59af284092279bfd6ae1bb615b90e7f8fbb10057e215db51fd60697
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Security for llm-behave 0.2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page