HEMP: Help Evaluate My Prompt

You have a prompt and some example data. HEMP works out how that prompt should be tested, runs the test, and tells you whether a new version of the prompt is better or worse, down to the examples that got fixed or broke.

hemp plan  prompt.txt data.jsonl   # decide how to test the prompt (you review the plan)
hemp run   prompt.txt data.jsonl   # run it, score it, compare with simple baselines
hemp diff                          # after editing the prompt: what got better, what broke

• It reads your data, not just your prompt. An answer column with labels gets an exact-match check, JSON answers get field-by-field checks, free text gets meaning-based checks.
• Yes/no questions for an LLM judge cover what code can't check ("Is the answer polite?"). No 1–10 scores.
• You can check the judge. Label a few outputs yourself and HEMP reports how often the judge agrees with you, and tunes it to agree more.
• Baselines give scores context. "Always answer the most common label" and "no instructions at all" run alongside your prompt.
• Local and private by default. It runs on your machine with Ollama, or with any model LiteLLM supports (OpenAI, Gemini, Anthropic, ...). There's no account and no server.
• Text and images. A column of image paths is sent to the model as real images.

Install

pip install hemp-eval
pip install "hemp-eval[images]"   # also scale down large images before sending them

The command is hemp and the Python package is hemp. Python 3.11+ is required.

You also need a model. Either:

• Local (default): install Ollama, then ollama pull qwen2.5:7b and ollama pull nomic-embed-text. Keep Ollama running.
• Hosted: set HEMP_MODEL to a LiteLLM model name and its API key, e.g. HEMP_MODEL=openai/gpt-4o-mini with OPENAI_API_KEY, or HEMP_MODEL=gemini/gemini-2.0-flash with GEMINI_API_KEY.

Then check your setup with hemp doctor. It confirms each model answers, and says exactly what to fix if one doesn't. Use --offline to check without making any calls.

Quick start

1. Write your prompt with {column} where each row's data goes, in prompt.txt:

Classify the sentiment of this product review as exactly one of: positive, negative, neutral.
Reply with the label only.

Review: {review}

2. Collect examples in data.jsonl (CSV and JSON work too), ideally with the right answer. 20–100 rows is a good start.

{"review": "Great product, arrived early.", "label": "positive"}
{"review": "Broke after two days.", "label": "negative"}

3. Plan. Run hemp plan prompt.txt data.jsonl. HEMP profiles the data, asks the planner model how to test the prompt, and writes prompt.plan.yaml:

columns:
  input: [review]
  answer: label

checks:  # automatic, no LLM needed
  - check: allowed_values
    values: [negative, positive, neutral]
  - check: exact_match
    compare_to: label

questions:  # an LLM judge answers each; "yes" = pass
  - id: q1
    ask: Does the label fit the review's overall tone?

baselines:  # simple strategies your prompt should beat
  - majority_class
  - zero_shot

Read it, and delete or add checks and questions as you like. hemp validate prompt.plan.yaml checks your edits, and hemp metrics lists every available check.

4. Run. hemp run prompt.txt data.jsonl runs your prompt on every row exactly as written and scores it:

| Check             | Your prompt | majority_class (baseline) | zero_shot (baseline) |
| allowed_values    | 100% (8/8)  | 100% (8/8)                |   0% (0/8)           |
| exact_match_label | 100% (8/8)  |  38% (3/8)                |   0% (0/8)           |

Then it lists the examples your prompt got wrong. Results are kept in .hemp/hemp.db; hemp show lists past runs.

5. Improve and compare. Edit the prompt, run again, then hemp diff:

Changed: prompt   Same: plan, dataset, target model, judge model
| Check             | Run #1 | Run #2 | Change  | Fixed | Broke |
| exact_match_label | 75%    | 100%   | +25 pts |   3   |   0   |
Fixed (3): "It's ok"  before: positive  ->  now: neutral

Baselines are reused between runs when nothing they depend on changed, so each round only pays for your prompt.

Images

A column of image paths (.png, .jpg, .jpeg, .gif, .webp, .bmp), image URLs, or data:image/... URIs is an image column. Use it in the prompt like any other column:

Classify this photo as cat, dog or bird. Reply with the label only.
{image}

image,label
photos/cat_01.jpg,cat
photos/dog_01.jpg,dog

• What the model gets: the image itself, at the spot where {image} is, not the path.
• Paths: relative paths are looked up next to the data file.
• Size: with hemp-eval[images], images bigger than 1536 px are scaled down first. Set the limit with HEMP_IMAGE_MAX_SIDE; 0 turns it off.
• Baselines and judge: the zero_shot baseline sends just the image, and the judge sees the images too.
• Model: use one that can see images, such as a Gemini or GPT-4o model, or a local ollama_chat/gemma3:4b.

Check and tune the judge

Judge scores are only useful if the judge answers the way you would. After a run with questions:

hemp calibrate prompt.plan.yaml

HEMP shows outputs one at a time, with the judge's answer hidden. You answer each question yourself: y, n, s to skip, q to stop. It then reports how often the judge agrees with you, as accuracy and Cohen's kappa (which corrects for lucky agreement), per question.

With 8 or more labels, HEMP also tunes the judge with DSPy: it picks worked examples from half your labels and tests on the other half. If the tuned judge agrees with you more, it's saved next to the plan and hemp run uses it from then on.

Commands

Command	What it does
hemp doctor	Check that your models are reachable, with a fix for each problem.
hemp profile data.jsonl	Column types and roles (input, answer, id) and value stats. No model calls.
hemp plan prompt.txt data.jsonl	Write an editable evaluation plan. --no-llm uses rules only.
hemp validate plan.yaml	Check an edited plan.
hemp metrics	List the checks a plan can use and their settings.
hemp run prompt.txt data.jsonl	Run and score. Useful options: --target-model, --judge-model, --no-judge, --no-baselines.
hemp show [run]	List runs, or show one with its failed examples.
hemp diff [A] [B]	Compare two runs (default: the latest two). --fail-on-regression for CI.
hemp calibrate plan.yaml	Label outputs, measure and tune the judge.

Models

Used for	Default	Change with
Planning, running your prompt, judging	ollama_chat/qwen2.5:7b	HEMP_MODEL for all three, or HEMP_PLANNER_MODEL, HEMP_TARGET_MODEL, HEMP_JUDGE_MODEL
Comparing free-text answers by meaning	ollama/nomic-embed-text	HEMP_EMBED_MODEL

Any LiteLLM model name works. A judge at least as strong as the model being tested gives better scores.

Your own checks

Write a check in a Python file:

# my_metrics.py
import hemp


@hemp.metric("polite")
def polite(ctx: hemp.MetricContext, params: hemp.MetricParams) -> hemp.MetricResult:
    """The output says please or thanks."""
    text = ctx.output.lower()
    return hemp.MetricResult.check("please" in text or "thank" in text)

Pass it with --plugin my_metrics.py to hemp plan or hemp metrics. The planner can choose it, and the plan remembers the file.

Examples

The examples folder has ready-to-run projects:

Example	What it shows
sentiment	Labels: exact match and baselines
extraction	JSON output checked field by field
qa	Free-text answers, a judge question, and a custom check
colors	Images: name the color of each swatch (needs a vision model)

For example: hemp plan examples/sentiment/prompt.txt examples/sentiment/data.jsonl.

Development

git clone https://github.com/NithilanVishvanath/hemp
cd hemp
uv sync
uv run pytest                   # unit tests, no models needed
uv run pytest -m integration    # tests that call a real model
uv run ruff check . && uv run ruff format .
uv run mypy

License

MIT