pbt — prompt-build-tool

A data-enineering-inspired prompt orchestration tool for LLMs.

Write modular prompts in Jinja2, reference the output of other prompts with ref(), and let pbt resolve dependencies.

Quick start

1. Install

pip install prompt-build-tool

# Also install the SDK for your LLM provider:
# pip install google-genai      # Gemini
# pip install openai            # OpenAI
# pip install anthropic         # Anthropic

# Optional: coding agents for model_type="agent"
# pip install "prompt-build-tool[agent]"   # mini-swe-agent
# npm i -g opencode-ai                     # opencode, with MCP servers

2. Generate example

pbt init --provider anthropic
# pbt init --provider openai
# pbt init --provider gemini

3. Set your API key

export ANTHROPIC_API_KEY=your_key_here
# export OPENAI_API_KEY=your_key_here
# export GEMINI_API_KEY=your_key_here

4. Run

pbt run

5. Extend prompt models

In the models/ directory write .prompt files:

models/
  topic.prompt
  outline.prompt
  article.prompt

Use ref('model_name') to inject the output of another model:

{# models/outline.prompt #}
Based on this topic, create a detailed outline:

{{ ref('topic') }}

Group models in folders as the project grows. A leading <number>_<letter>_ only sorts files in reading order and is not part of the model name, so 2_build/2_a_parts.prompt is still ref('parts'):

models/
  1_design/
    1_a_brief.prompt
    1_b_requirements.prompt
  2_build/
    2_a_parts.prompt
  3_qa/
    3_a_bom_line.prompt

Model names must be unique across all folders.

All standard Jinja2 syntax works too:

{# models/comparison.prompt #}
{% set languages = ['Python', 'Go', 'Rust'] %}
Compare these languages for building CLI tools:
{% for lang in languages %}
- {{ lang }}
{% endfor %}

Context from previous analysis:
{{ ref('initial_analysis') }}

Concepts (if you are familiar with data build tool)

Commands

pbt run

Execute all prompt models in dependency order.

pbt run

pbt ls

List discovered models and their dependency graph.

pbt ls

pbt test

Run tests/*.prompt files against the latest run's outputs. Each test passes when the LLM returns {"results": "pass"}. YAML cases can instead use expect to compare selected output fields directly, without a judge or test prompt files; see YAML expectations.

pbt test

Attaching model files (promptfiles) — a test attaches files the models produced with the same config() syntax models use, so the judge sees the actual bytes rather than a [file: …] handle. Name a model for all its files, or model.key for one:

{# tests/logo_is_a_fox.prompt #}
{{ config(promptfiles=["logo"]) }}
Is the attached image a fox? Respond {"results": "pass"} or {"results": "fail"}.

Names that aren't models are run-level --promptfiles. A test that declares no promptfiles gets every run-level promptfile, as before. The attached bytes are part of the test's cache key.

Classifier judge — instead of an LLM, a test can be judged by a classifier such as Jev or a local Ollaya model, which answers a yes/no question with a probability. Put the question above a --- line and the text to judge below it; the test passes when P(yes) ≥ 0.5 (or {{ config(threshold=0.8) }}):

{# tests/haiku_has_three_lines.prompt #}
Does this haiku have exactly three lines?
---
{{ ref('haiku') }}

Add a classify_call(state, question) -> float to client.py. pbt.systemone_classifier() builds one for any /v1/systemone API with no extra dependencies:

import pbt
classify_call = pbt.systemone_classifier()  # hosted Jev; reads TYPESAFE_API_KEY
# classify_call = pbt.systemone_classifier(model="laya:en", base_url="http://localhost:11435")  # Ollaya
test_judge = "classifier"                    # optional: make it the default judge

Runnable example with a local Ollaya (install, start, pbt test): examples/classifier_test.

Use case — classifier tests on a chained pipeline: tests/README.md.

The judge is chosen per test with {{ config(judge="llm") }} / {{ config(judge="classifier") }}, otherwise by pbt test --judge llm|classifier, otherwise by test_judge in client.py, otherwise llm. There is no fallback: a classifier test without a --- line, or one that attaches promptfiles, is an error. LLM-judged tests are unchanged — the whole file, --- included, goes to the LLM.

Gate tests — every test also guards pbt run by default. pbt run runs the models gate tests read (and everything upstream of them) first, then the gate tests, then the rest of the pipeline — only if every gate passed. A failing or erroring gate stops the run with exit code 1, and the downstream models are never called. With --select, a gate applies when all the models it reads are in the selected run, so pbt run -s article still checks outline before article. pbt test runs gate tests like any other test.

Opt a test out when it should only run under pbt test — for example a subjective quality check that should not block a run, or one whose judge needs a service pbt run should not depend on:

{# tests/memo_reads_well.prompt #}
{# gate=False: style judgement; a borderline verdict should not block the run #}
{{ config(gate=False) }}
Is this memo clear and concise? {{ ref('memo') }}
Respond {"results": "pass"} or {"results": "fail"}.

pbt docs

Write an HTML report of every run: each model's prompt, output, timing and files, plus the DAG.

pbt docs           # writes .pbt/docs/index.html
pbt docs --open    # and opens it

Passing variables to templates (promptdata())

Inject runtime variables into templates using the promptdata("name") function — similar to how dbt's source() and ref() work.

pbt run --promptdata tone=formal --promptdata audience=engineers

pbt.run("models", promptdata={"tone": "formal", "audience": "engineers"})

Access them in any .prompt file:

Write an article in a {{ promptdata("tone") }} tone for {{ promptdata("audience") }}.

{% if promptdata("topic") %}
Topic: {{ promptdata("topic") }}
{% else %}
Choose a fascinating topic of your choice.
{% endif %}

promptdata("name") returns None if the variable was not provided, so {% if promptdata("x") %} is always safe.

Customising the LLM backend (client.py)

pbt is unopinionated about which LLM you use. Create client.py at the project root (alongside your models/ directory) and define an llm_call function — usually 5 lines:

# client.py (Anthropic example)
import anthropic

def llm_call(prompt: str) -> str:
    client = anthropic.Anthropic()
    message = client.messages.create(
        model="claude-opus-4-6",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    return message.content[0].text

pbt will automatically discover and use this file. Run pbt init --provider <gemini|openai|anthropic|deepseek|qwen|kimi|xiaomi> to scaffold a starter client.py for your chosen provider; add --classifier to also scaffold a jev-like classify_call for classifier-judged tests (P(yes) from one-token logprobs where the provider returns them — OpenAI, DeepSeek, Qwen — otherwise a hard yes/no). If the file exists but does not define llm_call, pbt raises an error at startup.

RAG inside prompts (rag.py)

pbt has very little to say about RAG and leaves that up to you - you do this through the return_list_RAG_results(*args) function pbt give you access to in the .prompt template. pbt will pass this call to the do_RAG function you define in rag.py (at the project root, alongside your models/ directory):

# rag.py
def do_RAG(*args) -> list[str] | str:
    query = args[0]
    # your vector search, keyword lookup, etc.
    return ["Relevant document 1", "Relevant document 2"]

do_RAG receives whatever arguments you pass to return_list_RAG_results in the template. It can return a list[str] or a bare str (wrapped automatically). Return False or None to signal no results.

Use it in any .prompt file:

{% set hits = return_list_RAG_results(ref('topic')) %}
{% if hits[0] %}
A related article in our library: "{{ hits[0] }}"

Write a paragraph explaining how the topic below connects to it:
{{ ref('topic') }}
{% else %}
Write a paragraph introducing this topic as a fresh subject:
{{ ref('topic') }}
{% endif %}

If rag.py is absent and a template calls return_list_RAG_results, pbt raises a clear error at render time.

Passing files to models (promptfiles)

Models can receive files (PDFs, images, etc.) alongside the text prompt. Declare the files a model needs via config(), then provide the actual paths at runtime.

1. Declare in config:

{{ config(promptfiles=["my_document"]) }}
Summarise the attached document in 3 bullet points.

Multiple files use a JSON array:

{{ config(promptfiles=["report", "chart_image"]) }}

2. Provide file paths at runtime:

pbt run --promptfile my_document=report.pdf
pbt run --promptfile report=annual.pdf --promptfile chart_image=q4.png

pbt.run("models", promptfiles={"my_document": "report.pdf"})
pbt.run("models", promptfiles={"report": "annual.pdf", "chart_image": "q4.png"})

3. Custom llm_call with file and config support:

Accept optional files and/or config parameters in your client.py — pbt passes them if the signature declares them:

# client.py
def llm_call(prompt: str, files: list[str] | None = None, config: dict | None = None) -> str:
    # files  — resolved file paths declared via config(promptfiles=...)
    # config — the full config dict for this model, e.g. {"output_format": "json"}
    ...

Both parameters are optional and independent — declare either, both, or neither.

A model with {{ config(each="imgs") }} can iterate over a list of files, or over the files of a Dir; {{ config(promptfiles=[...]) }} then attaches each item's own file.

Template models (model_type="template")

A template model renders its Jinja and uses the result as its output, with no LLM call — for nodes that only reshape what upstream models already produced.

{# models/report.prompt #}
{{ config(model_type="template") }}

# {{ ref('title') }}

{{ ref('summary') }}

Python models (model_type="execute_python")

An execute_python model runs its template as Python instead of sending it to the LLM. Use it for the deterministic steps in a pipeline — counting, parsing, reshaping, arithmetic on an upstream result.

{# models/length.prompt #}
{{ config(model_type="execute_python") }}
output = len(ref('article').split())

The template is rendered first, then the result is executed. Inside the code, ref('name') returns an upstream model's output and model_outputs holds them all. Those ref() calls are also what build the dependency edges, exactly as in a normal prompt.

The output is whatever the code prints; if it prints nothing, a variable named output is used instead (dict and list are JSON-encoded, anything else via str()). Printing wins over output.

Results are cached on the rendered code, so unchanged code does not re-run. The code executes in-process with full builtins and no sandbox, so treat a .prompt file as trusted code.

Model kinds from installed packages

A package can ship its own model_type values. Declare them under the pbt.model_kinds entry point group and pip install is all a user needs — no import in client.py:

# the plugin's pyproject.toml
[project.entry-points."pbt.model_kinds"]
my_kind = "my_pkg.my_kind"        # a module that calls @pbt.model_kind on import
other   = "my_pkg:register"       # or a function pbt calls once

pbt loads them on the first model_type lookup. A plugin that fails to import is logged and skipped. Set PBT_DISABLE_PLUGINS=1 to turn discovery off.

Validation (validation/)

Create a validation/ directory with Python files matching model names. Each file must define validate(prompt, result) -> bool. If it returns False, the model is marked as an error and stops it use in downstream models.

# validation/article.py
import json
from pydantic import BaseModel, ValidationError


class Article(BaseModel):
    content: str
    author: str
    audience: str


def validate(prompt: str, result: str) -> bool:
    """Article output must be valid JSON matching the Article model."""
    try:
        data = json.loads(result)
        article = Article(**data)
    except (json.JSONDecodeError, ValidationError):
        return False
    return len(article.content) >= 200

Run with pbt run — validation fires automatically after each model's LLM call.

How to dynamically skip a model

Use {{ skip_and_set_to_value("value") }} to skip the LLM call during Jinja rendering and provide the output directly:

{% if "no action needed" in ref('previous_model') %}
{{ skip_and_set_to_value("No action needed.") }}
{% else %}
Summarise the following: {{ ref('previous_model') }}
{% endif %}

The model is recorded as a successful run, downstream templates can detect it with was_skipped('model_name'), and downstream ref() calls receive the value you provided.

Advanced usage

docs/ADVANCED_USAGE.md covers:

• Python API, including live progress callbacks
• Token usage
• Bulk testing with YAML cases
• pbt serve and the HTTP server
• Output format and config() keys
• Global instructions
• Map and reduce with each=
• Models that produce files
• Agent models
• Writing your own model kind

Internals and design decisions: docs/ARCHITECTURE.md.