prompttest 0.2.0

pytest for LLMs

prompttest

pytest for LLMs.

You wouldn't ship code without tests. ✋😮🤚

Hold your prompts to the same standard. 😎👌🔥

Features

• 🔤 Test in Plain English: Write your tests in English, and let an AI grade the response.

• 🚀 Write Tests Faster: Just list your inputs and criteria in a simple file—no code needed.

• 🔓 Avoid Vendor Lock-in: Test against any LLM with a single, free OpenRouter API key.

Quick Start

1. Install prompttest

pip install prompttest

2. Set up prompttest

prompttest init

3. Run your tests

prompttest

How It Works

prompttest is built around 2 types of files:

Prompt

A .txt file for your prompt template with ---[SECTIONS]--- and {variables}.

Example: prompts/customer_service.txt

---[SYSTEM]---
You are an expert on the "{product_name}".
Your responses must be helpful and polite.

---[USER]---
Customer tier: {user_tier}
Customer query: {user_query}

Test

A .yml file for test cases with config, inputs and criteria.

Example: prompttests/test_customer_service.yml

config:
  prompt: customer_service

tests:
  - id: check-simple-greeting
    inputs:
      product_name: "Chrono-Watch"
      user_tier: "Standard"
      user_query: "Hello"
    criteria: "The response must be a simple, polite greeting."

Advanced Usage

Run all tests in a folder

prompttest run folder_name/

Run all tests in a file

prompttest run file_name.yml

Run specific test

prompttest run test_id

Contributing

We're building the pytest for LLMs—and we need your help.

Report a bug, propose a feature, or contribute a single line.

Help shape a foundational tool for AI development.

License

This project is licensed under the MIT License.