promptmap

A data-engineering-inspired prompt orchestration tool for LLMs (think dbt for prompts).

Why promptmap? Treating complex LLM workflows as giant one-shot prompts is like navigating a city by staring at your feet, one wrong turn and you're lost. promptmap turns your prompts into a visual map of connected hypotheses (models) and reality-check landmarks (tests), with built-in tools to inspect runs (promptmap docs).

Mechanically, this means you: write modular prompts in Jinja2, chain them into a dependency graph with ref(), and validate outputs against reality with promptmap test. Iterate on complex LLM workflows without losing context or rewriting from scratch. (Read more on the philosophy in motivation.md).

Quick start

uv tool install promptmap --with anthropic  # or: openai | gemini
promptmap init --provider anthropic   # or: openai | gemini

Edit your .prompt files in map/, then run the pipeline, test expectations, and inspect the run:

export ANTHROPIC_API_KEY=your_key_here

promptmap run     # Execute the prompt DAG
promptmap test    # Validate outputs against map/*.testprompt
promptmap docs    # Generate HTML report of the run and run history
promptmap serve   # Launch the interactive local UI / HTTP server

Core concepts

1. Models (map/*.prompt)

Each .prompt file is a node in your DAG. Use ref('model_name') to inject the output of an upstream model:

{# map/outline.prompt #}
Based on this topic, create a detailed outline:

{{ ref('topic') }}

2. Model types (config(model_type=...))

Not every node in your pipeline needs to be a standard LLM call. Override model_type at the top of a .prompt file for deterministic steps, formatting, or coding agents:

(Optional extra model kinds like python_modal, agent_modal, and agent37 are available via pip install "promptmap[extras]").

3. Tests (map/*.testprompt)

Run promptmap test to validate the latest run's outputs against your expectations:

• LLM-as-a-judge: Prompt tests in map/*.testprompt (alongside your models) pass when the LLM returns {"results": "pass"}.
• Deterministic checks: Compare output fields directly in YAML cases (map/*.yml) using expect (no LLM judge needed).

4. Visualizing & serving (promptmap docs & promptmap serve)

Inspect prompt graph visually:

• promptmap docs generates a self-contained HTML report of every run—showing rendered prompts, model outputs, token usage, and test results across the DAG.
• promptmap serve starts a local server to interactively explore the DAG, trigger runs or inspect previous results.

Documentation