pytest-semantix

Semantic LLM output testing for pytest. Validate that your LLM outputs mean the right thing — not just that they match a string.

pip install pytest-semantix

Usage

The assert_semantic fixture

def test_chatbot_is_polite(assert_semantic):
    response = my_chatbot("handle angry customer")
    assert_semantic(response, "polite and professional")

Runs locally on CPU in ~15ms. No API key. Works with any LLM.

On failure:

AssertionError: Semantic check failed (score=0.12)
  Intent:  polite and professional
  Output:  "You're an idiot for asking that."
  Reason:  Text contains aggressive language

Markers

Use @pytest.mark.semantic to attach an intent to a test:

import pytest

@pytest.mark.semantic("polite and professional")
def test_with_marker(assert_semantic):
    response = my_chatbot("handle angry customer")
    assert_semantic(response)  # intent comes from the marker

Intent classes

Reuse intents across tests:

from semantix import Intent

class Polite(Intent):
    """The text must be polite and professional."""

def test_polite(assert_semantic):
    assert_semantic(my_chatbot("hello"), Polite)

Negation

Test that outputs do NOT match an intent:

from semantix import Intent

class MedicalAdvice(Intent):
    """The text provides medical diagnoses or treatment recommendations."""

def test_no_medical_advice(assert_semantic):
    assert_semantic(my_chatbot("my head hurts"), ~MedicalAdvice)

CLI Options

--semantic-report          Print a summary of all semantic assertions
--semantic-report-json=PATH  Write results to a JSON file
--semantic-threshold=FLOAT   Global default threshold (0.0-1.0)

Report example

$ pytest --semantic-report

======================== semantic assertion report =========================
  Total: 5  |  Passed: 4  |  Failed: 1

  [PASS] tests/test_bot.py::test_polite  [12ms]
  [PASS] tests/test_bot.py::test_helpful  [14ms]
  [FAIL] tests/test_bot.py::test_no_pii  (score=0.67)  Contains email address  [11ms]
  [PASS] tests/test_bot.py::test_on_topic  [13ms]
  [PASS] tests/test_bot.py::test_concise  [15ms]

============================================================================

JSON report

$ pytest --semantic-report-json=semantic-results.json

{
  "summary": { "total": 5, "passed": 4, "failed": 1 },
  "results": [
    {
      "nodeid": "tests/test_bot.py::test_polite",
      "intent": "polite and professional",
      "passed": true,
      "score": null,
      "reason": "",
      "duration_ms": 12.3
    }
  ]
}

How it works

pytest-semantix wraps semantix-ai's assert_semantic() function as a pytest fixture. Under the hood, it uses a local NLI (Natural Language Inference) model to check whether your LLM output entails the given intent. No network calls, no API keys, no tokens burned.

License

MIT