prompt-ninja 1.0.1

Version, validate, and regression-test LLM prompts as software artifacts

Prompt Ninja

Prompt Ninja turns prompts into versioned, validated, and regression-tested software artifacts. A *.prompt.toml file keeps the instructions, model, typed inputs, output contract, and semantic tests together in one file you can commit alongside your code.

Install

pip install prompt-ninja
export OPENROUTER_API_KEY=your-key

That installs the CLI and the Python API. The optional server extra adds the FastAPI backend for the Board of Prompts UI:

pip install 'prompt-ninja[server]'

Generation, execution, semantic tests, updates, and AI-assisted repair use OpenRouter. Plain loading, rendering, and validation do not require an API key.

From a description to a tested prompt

1. Create the artifact

Describe the behavior you want. Prompt Ninja generates a versioned *.prompt.toml file with model settings, typed inputs, an output contract, and semantic tests:

mkdir -p prompts
prompt-ninja generate \
  --goal "Turn release notes into a concise customer update" \
  --output prompts/customer-update.prompt.toml

Open the generated file and review it like source code. Instructions live under [prompt], inputs under [[variables]], and behavioral expectations under [[tests]].

Structured JSON generation also writes an importable companion Pydantic module beside the TOML and sets metadata.output to that model's dotted path. Run the command from the project root. Unknown consumers are represented by metadata.used_by = []; Prompt Ninja does not invent a consumer path. Its generated tests use model-valid object expectations for structured output and natural-language semantic expectations for text output.

2. Add or change expectations

Add semantic tests directly to the artifact without prescribing exact output wording:

[[tests]]
name = "avoids internal terminology"
variable.release_notes = "The API gateway migration begins July 30."
expected_output = "A customer-friendly update that preserves the date and does not use internal engineering terminology."

3. Validate or repair the artifact

prompt-ninja validate prompts/customer-update.prompt.toml
prompt-ninja validate prompts

Validation locally checks the TOML schema, variables, defaults, model settings, output contract, and importable Pydantic paths. If validation fails, ask Prompt Ninja's versioned repair prompt to produce a corrected artifact:

prompt-ninja validate prompts/customer-update.prompt.toml --fix

The replacement must validate before it is written, and the original is preserved as .bak.

4. Run semantic regression tests

Run one prompt, one named expectation, or the complete prompt directory:

prompt-ninja test --prompt prompts/customer-update.prompt.toml --verbose
prompt-ninja test \
  --prompt prompts/customer-update.prompt.toml \
  --test-name "avoids internal terminology"
prompt-ninja test-prompts --prompts-dir prompts --plain

Tests use an LLM judge to evaluate meaning rather than exact wording. Commands exit non-zero on failure, and --plain produces CI-friendly output. Failed cases show the complete rationale plus actionable prompt or test-case changes.

5. Update the prompt from feedback

Ask Prompt Ninja to revise the prompt implementation. Tests and the pass threshold are protected, and the candidate runs the complete contract before promotion:

prompt-ninja update \
  prompts/customer-update.prompt.toml \
  "Preserve dates and avoid internal engineering terminology"

The command prints complete diagnostics. A passing candidate is written and the previous version is preserved as .bak; a failing candidate leaves the original artifact untouched.

6. Load it in your application

from prompt_ninja import PromptCollection, PromptNinja

prompt = PromptNinja.from_file("prompts/customer-update.prompt.toml")
result = await prompt.run_openrouter(
    {"release_notes": "Search launches July 30 with CSV export."}
)

prompts = PromptCollection(dir="prompts")
customer_update = prompts.customer_update

Prefer a UI?

The repository includes the Board of Prompts, a guided UI where multiple LLMs enhance your brief, draft alternatives, judge them, and produce the final tested *.prompt.toml artifact. You can attach up to five reference files and choose the models used at each stage.

See the UI setup instructions, run prompt-ninja COMMAND --help for CLI options, or browse the complete documentation.