PromptSafeAI 0.2.0

A lightweight toolkit for storing, running, and evaluating LLM prompts.

PromptVault

PromptVault is a lightweight Python SDK for tracking LLM prompt evaluations locally. You define the metrics. PromptVault tracks how they evolve.

Install

pip install PromptSafeAI

The package installs as PromptSafeAI but imports as promptvault:

from promptvault import experiment, run, evaluate

Quickstart

from promptvault import experiment, run, evaluate

with experiment(name="summarizer-test", model="claude-sonnet-4-6", vars={"text": text}):
    result = run("summarize")
    evaluate(result, {"under_100_words": lambda r: len(r.split()) < 100})
    print(result["response"])

→ experiment 'summarizer-test' started (fc5dbdd6)
  run ba5089cd | claude-sonnet-4-6 | 60in 38out | 720ms | $0.000750
  score | under_100_words: True
✓ experiment 'summarizer-test' complete

How it works

An experiment locks in a model and a vars dict. Every run inside it uses the same inputs, so comparisons across prompt versions and models are actually meaningful.

A run loads a prompt, calls the API, and auto-captures everything it returns: response, token counts, latency, cost, and stop reason. You write nothing to log these.

evaluate() runs your metric functions against the response and stores each result in a local SQLite file (promptvault.db). The metrics are yours — booleans, numbers, strings, whatever you decide "better" means.

Prompt files

Prompts live as plain text files in prompts/:

prompts/
  summarize.txt          # user template (required)
  summarize.system.txt   # system template (optional)

Templates use {{var}} placeholders, filled from the experiment's vars:

Summarize the following text in 2-3 sentences: {{text}}

Git tracks your prompt history; PromptVault tracks which runs used which version (via prompt_hash).

Querying your results

promptvault.db is plain SQLite. Query it however you like.

How a metric evolved across prompt versions:

SELECT r.prompt_hash, s.value
FROM runs r JOIN scores s ON s.run_id = r.run_id
WHERE r.prompt_name = 'summarize' AND s.metric_name = 'under_100_words'
ORDER BY r.timestamp;

How the same prompt performs across models:

SELECT r.model, s.metric_name, s.value
FROM runs r JOIN scores s ON s.run_id = r.run_id
WHERE r.prompt_name = 'summarize'
ORDER BY r.model;

Configuration

Two optional environment variables:

• PROMPTVAULT_HOME — anchor prompts/ and promptvault.db to one directory instead of the current working directory. Set this when your app runs from varying locations.
• PROMPTVAULT_RATES — path to a JSON file of per-token rates that extends or overrides the built-in cost table ({"model-id": {"input": 0.000003, "output": 0.000015}}). Models with no rate data record cost_usd as NULL — never a fake 0.0.

API keys come from the environment: ANTHROPIC_API_KEY for Claude models, OPENAI_API_KEY for GPT models.

Why not X

• DeepEval / RAGAS — opinionated built-in metrics, not yours.
• LangSmith / Braintrust — cloud services with accounts; not local.
• Rolling your own logger — no structure, and a combinatorial mess once you try to compare versions × models × metrics.

Roadmap

• v1.1 — LLM-as-judge built-in evaluator.
• v1.2 — pytest plugin so evals run in CI.
• v1.3 — CLI command to print an evolution table in the terminal.