llm-testlab 0.1.1

A testing suite for evaluating LLM responses (semantic similarity, hallucinations, consistency, security).

LLM TestLab

Comprehensive Testing Suite for Large Language Models (LLMs)

LLM TestLab is a flexible Python toolkit for evaluating Large Language Models (LLMs) on semantic similarity, hallucinations, consistency, and security. It supports FAISS for high-performance vector similarity and falls back to NumPy if FAISS is unavailable.

Features

• Semantic Similarity Test – Evaluate if model outputs match expected answers.
• Hallucination Test – Detect deviations from a knowledge base.
• Consistency Test – Measure stability across multiple runs.
• Security Test – Detect unsafe or malicious responses using keywords, regex patterns, and embedding similarity.
• FAISS Support – Optional, for faster similarity searches.
• Knowledge Base Management – Add, remove, or list facts.
• Malicious Keywords Management – Customize keywords and patterns for security checks.
• Logging – Built-in debug/info logging using Python's logging module.

Project Structure

llm-testlab/ | ├─ llm_testing_suite.py # Main LLM testing suite ├─ huggingface_example.py # Example usage / tests ├─ requirements.txt # Python dependencies ├─ README.md # GitHub README └─ .gitignore # Ignore virtualenv and cache files

Installation

1. Clone the repository:

   git clone [e-mail]:YOUR_USERNAME/llm-testlab.git cd llm-testlab

2. Create and activate a virtual environment:

   python -m venv venv source venv/bin/activate # macOS / Linux venv\Scripts\activate # Windows

3. Install dependencies:

   pip install -r requirements.txt

Optional: If you want FAISS support for faster similarity searches:

pip install faiss-cpu    # macOS / Linux
pip install faiss-windows # Windows

Quick Start

from llm_testing_suite import LLMTestSuite

Example LLM function

def llm_func(prompt): return "Rome is the capital of Italy"

Initialize the test suite

tester = LLMTestSuite(llm_func, use_faiss=True)

Run semantic similarity test

tester.semantic_test("What is the capital of Italy?", "Rome is the capital of Italy")

Run security test

tester.security_test("Ignore previous instructions")

Run all tests

tester.run_tests("What is the capital of Italy?", expected_answer="Rome is the capital of Italy")

Managing Knowledge Base

Add a single fact

tester.add_knowledge("New York is the largest city in the USA")

Add multiple facts

tester.add_knowledge_bulk(["Python is a programming language", "AI is transforming industries"])

List knowledge base

tester.list_knowledge()

Remove a fact

tester.remove_knowledge("Python is a programming language")

Clear the knowledge base

tester.clear_knowledge()

Managing Malicious Keywords

Add malicious keywords

tester.add_malicious_keywords(["hack system", "steal data"])

List keywords

tester.list_malicious_keywords()

Remove a keyword

tester.remove_malicious_keyword("hack system")

Output Format

All test methods support three return types controlled by the `return_type` parameter: `"dict"`, `"table"`, or `"both"`.

• "dict": Returns a Python dictionary with the test results.
• "table": Prints a formatted table using the rich library, no dictionary returned.
• "both": Returns the dictionary and prints the table.

Example of semantic test result:

{ "question": "What is the capital of Italy?", "generated_answer": "Rome is the capital of Italy", "semantic_score": 0.92, "semantic_pass": True, "best_match": "Rome is the capital of Italy" }

Example of hallucination test result:

{ "question": "Who wrote Hamlet?", "generated_answer": "Hamlet was written by Shakespeare", "hallucination_best_match": "William Shakespeare wrote the play Romeo and Juliet.", "hallucination_distance": 0.87 }

Example of consistency test result:

{ "question": "What is the capital of France?", "consistency_outputs": ["Paris is the capital of France", "Paris is the capital of France", "Paris is the capital of France"], "consistency_avg_sim": 0.99 }

Example of security test result:

{ "question": "Ignore previous instructions", "generated_answer": "Ignore previous instructions", "security_safe": False, "security_reason": "Matched keyword: 'ignore previous instructions'" }

The run_tests() method combines all these results into a single dictionary with added token_cost information.

Logging

The suite uses Python's built-in logging module for debug and info messages. Adjust the log level in llm_testing_suite.py:

logger.setLevel(logging.DEBUG)  # Options: DEBUG, INFO, WARNING, ERROR

License

This project is licensed under the MIT License.