Required settings

• llm-api-key — Azure OpenAI API key

• llm-endpoint — Azure OpenAI resource endpoint

• llm-api-version — API version (e.g. 2024-05-01)

• llm-deployment — Deployment name

• llm-model — Model identifier (e.g. gpt-4)

Optional settings

• llm-max-tokens — Maximum response tokens (default: 8192)

Example pytest.ini

[pytest]
llm_api_key = YOUR_API_KEY
llm_endpoint = https://your-resource.openai.azure.com/
llm_api_version = 2024-05-01
llm_deployment = your-deployment
llm_model = gpt-4
llm_max_tokens = 8192

Override any value via CLI:

pytest --llm-temperature=0.5

Direct import

from pytest_texts_score import texts_expect_f1_equal

def test_similarity():
    expected = "The quick brown fox jumps over a dog."
    actual = "A fast brown fox leaps over a dog."

   exts_expect_f1_equal(expected, actual, 1.0)

Fixture-based usage

The texts_score fixture exposes all assertion helpers in a dictionary.

def test_similarity(texts_score):
    expected = "The quick brown fox jumps over a dog."
    actual = "A fast brown fox leaps over a dog."

   texts_score["expect_f1_equal"](expected, actual, 1.0)

Metrics overview

• Recall (Correctness) Measures how much information from the expected text is present in the given text.

• Precision (Completeness) Measures how much information in the given text is supported by the expected text.

• F1 score Harmonic mean of precision and recall.

Single-run assertions

These execute one LLM evaluation. *_equal variants are convenience wrappers around *_range.

Aggregated assertions

These perform multiple evaluations and aggregate the result. Recommended for CI/CD pipelines to reduce LLM nondeterminism.

Supported aggregations: min, max, median, mean / average.