Skip to main content

⚙️ PromptKit

Lint and test your LLM prompts before they reach production.

A typo in a prompt is a bug. PromptKit catches it like one.

PyPI Python CI mypy strict License

Documentation · Getting started · Issues

Prompts as f-stringsPrompts as data
prompt = f"""You are a support agent
for {product}. Be concise.

Customer wrote: {message}
"""
if order_id:
    prompt += f"Order: {order_id}"

msg = {"role": "user", "content": prompt}
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[msg],
)
text = resp.choices[0].message.content

Logic tangled with copy. A typo in {prodcut} ships silently. No system role, no review, no tests, no idea what it costs until the invoice arrives.

name: support_reply
description: Support reply, house style
version: 1.0.0
messages:
  - role: system
    template: |
      You are a support agent for
      {{ product }}.
      {% include '_partials/style.j2' %}
  - role: user
    template: |
      Customer wrote: {{ message }}
input_schema:
  product: str
  message: str

Reviewable in a pull request. A typo is a lint error. Real roles, a shared house style, and promptkit cost before you spend anything.

Install

pip install 'promptkit-core[openai]'      # or [anthropic] · [ollama] · [all]

The base install ships no HTTP client and no provider SDK. Loading, rendering, validation, composition, linting and cost estimation all work with nothing network-shaped in your dependency tree.

Catch the typo before you pay for it

Fat-finger it — {{ prodcut }} in the template while the schema still says product. An f-string would ship that to production. Here it does not get past the gate:

$ promptkit lint support_reply.yaml
support_reply
  PK001 'prodcut' is used but not declared (prodcut)
  PK002 'product' is declared but never used (product)

2 finding(s)

Exit code 1, so CI stops. Fix it, and the rest of the loop is free and offline:

$ promptkit lint support_reply.yaml
✓ support_reply

$ promptkit render support_reply.yaml --set product=Acme --messages
system
You are a support agent for Acme.
Answer in plain language. Prefer short sentences.

user
Customer wrote: placeholder

$ promptkit cost support_reply.yaml --model gpt-4o-mini
support_reply with gpt-4o-mini
Input tokens: 21 (exact)
Output tokens: 500 (assumed)
Estimated cost: $0.000303

Anything you leave out is filled with a placeholder, so you can render and price a prompt before you have real inputs. Not one of those commands needed an API key.

From Python

from promptkit import load_prompt, run_prompt
from promptkit.engines.openai import OpenAIEngine

prompt = load_prompt("support_reply.yaml")

with OpenAIEngine() as engine:
    completion = run_prompt(prompt, {"product": "Acme", "message": msg}, engine)

completion.text  # the reply
completion.usage.prompt_tokens  # 241, straight from the provider
completion.usage.estimated  # False — measured, never quietly guessed
engine.cost_of(completion)  # 8.895e-05

Need JSON back? Hand it a Pydantic model and get a validated object, with automatic re-prompting when the model gets it wrong:

result = run_structured(prompt, inputs, engine, output_model=Invoice)
result.value.total  # a float, and your type checker knows it

What you get

Catch it before you pay Nine lint rules with codes — undeclared variables, unused schema fields, unresolved includes. --strict and --format json for CI
Evals that gate a build contains, regex, json_schema, max_cost, max_latency, and an LLM judge. JUnit output, concurrent, non-zero on failure
Honest costing Token counts come from the provider. Usage.estimated says when a number was guessed, so a cost report can never quietly lie
One house style Share text across prompts with {% include %}. Edit the partial and every dependent prompt changes fingerprint, so caches invalidate correctly
A prompt is not code Templates render in a Jinja sandbox; includes resolve through a confined loader. Traversal, symlink escapes and __class__ tricks are refused, with tests to prove it
Identity that means something Prompts are identified by a fingerprint over messages, schemas and resolved includes — not a version string someone forgot to bump
Providers, plugged in OpenAI, Anthropic, Ollama, and any OpenAI-compatible endpoint. One error hierarchy, so except RateLimitError means the same thing everywhere
Ship your own engine Register through the promptkit.engines entry point from your own package. No PR to this repo

The CLI

run Render and send. --stream, --structured
render Render locally. Free, offline, no key
lint Nine rules. --strict, --format json
test Run eval suites. --format junit
diff Compare two prompts by fingerprint
info · list Inspect one prompt, or a whole tree
cost Token counts and cost before you spend
init · engines Scaffold a prompt · see what's installed

Variables come from --set key=value, --vars '{"json": true}', or --vars-file. --set coerces to the type your schema declares.

Not an agent framework

Tool calling, agent loops, conversation memory and RAG are permanently out of scope — a boundary, not a gap. Nothing traps you: engine.client is the real SDK client and completion.raw the real provider response, so dropping down is one attribute away.

Coming from 0.1.x

Your prompt files load unchanged — asserted per-file by the test suite. Most code needs two edits, and a codemod handles the mechanical ones:

python -m promptkit.codemod your_package/          # dry run, prints a diff
python -m promptkit.codemod your_package/ --write  # apply it

Details in the upgrade guide.


promptkit-core.ochotzas.com

Prompt files · Schemas · Composition · Structured output · Evaluation · Writing an engine

MIT licensed · Contributing · Architecture

If PromptKit saves you a debugging session, a ⭐ helps others find it.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

promptkit_core-1.0.0.tar.gz (99.6 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

promptkit_core-1.0.0-py3-none-any.whl (80.8 kB view details)

Uploaded Python 3

File details

Details for the file promptkit_core-1.0.0.tar.gz.

File metadata

  • Download URL: promptkit_core-1.0.0.tar.gz
  • Upload date:
  • Size: 99.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for promptkit_core-1.0.0.tar.gz
Algorithm Hash digest
SHA256 0b23b9d09d2892ff49c234bbf32ce3a1215d5209101af2c08c3f385988f47ab4
MD5 c05be3ce24e4d2c0f9c3f00bb50486e4
BLAKE2b-256 c5db9833dcf1aeafc2b7e5d35215b3adf66833b777dfe0bc507125164ba03ddd

See more details on using hashes here.

Provenance

The following attestation bundles were made for promptkit_core-1.0.0.tar.gz:

Publisher: cd.yml on ochotzas/promptkit-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file promptkit_core-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: promptkit_core-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 80.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for promptkit_core-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 93c7a4e5dd578612e6deb6dbc607c4f6d940ddb8a163a05fcb3243faf2083bc5
MD5 6c5fc3a83fa09adec7e4c683a8525143
BLAKE2b-256 e4b2a9055258693c779fa48e262b013060c6994946d5c6095523a3ef94f436f0

See more details on using hashes here.

Provenance

The following attestation bundles were made for promptkit_core-1.0.0-py3-none-any.whl:

Publisher: cd.yml on ochotzas/promptkit-core

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

1.0.0 This release

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page