PromptVC

Promptvc is a lightweight and local tool for Git-like version control and A/B testing of your LLM prompts. It acts as a dev tool and version control, designed to help you find the best prompts before you add it into your workflow. Unlike an excel sheet, this is cleaner. Trust us.

Table of Contents

• Why?
• Install
• Quick Start
• Usage
• CLI Commands
• Python API
• Full Example
• Config
• FAQ

Why?

Instead of guessing which prompt is better, test them:

1. Version A: "Summarize this article: {input}"
2. Version B: "Write a 2-sentence summary focusing on key insights: {input}"

See which one actually performs better with your data

Install

pip install promptvc

Quick Start

# step 1. initialize promptvc
promptvc init

# step 2. create two prompt versions
promptvc add summarizer "Summarize this article: {input}"
promptvc commit summarizer "version A - simple"

promptvc add summarizer "Write a 2-sentence summary focusing on key insights: {input}"  
promptvc commit summarizer "version B - structured"

# step 3. create and edit your test data in the _samples.jsonn

promptvc init-samples summarizer
# inside your summarizer.json, edit your data. oh and dont forget to input your api keys

# step 4. test
promptvc test summarizer 1 2 --llm openai

Usage

CLI Commands

init: Initialize new repo
add <name> <text>: Add prompt version  
commit <name> <msg>: Commit staged version
history <name>: View version history
checkout <name> <version>: Get version text
diff <name> <v1> <v2>: Compare versions
list: List all prompts
init-samples <name>: Create samples file
test <name> <v1> <v2>: Compare versions with LLM

Python API

from promptvc.repo import PromptRepo
import openai

def my_llm(prompt):
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

# A/B test your prompts
repo = PromptRepo()
samples = [{"input": "Your test data here"}]
results = repo.eval_versions("prompt-name", [1, 2], samples, my_llm)

# compare your outputs
for version_id, result in results.items():
    print(f"Version {version_id}:")
    for output in result['outputs']:
        print(f"  {output['output']}")

Full example

# test email writing prompts
promptvc add email-writer "Write a professional email about: {input}"
promptvc commit email-writer "formal version"

promptvc add email-writer "Write a friendly, concise email about: {input}"
promptvc commit email-writer "casual version"

# create your test cases

promptvc init-samples email-writer


# inside your json email-writer_samples.json:
# [
#   {"input": "quarterly sales meeting"},
#   {"input": "server maintenance window"},
#   {"input": "new product launch"}
# ]

# test
promptvc test email-writer 1 2 --llm openai

Config

llm_providers:
  openai:
    api_key: "your-key-here"
    default_model: "gpt-4o-mini"
  anthropic:
    api_key: "your-key-here"  
    default_model: "claude-3-sonnet-20240229"

MISC

Tests

For tests conducted, refer to TEST.md

Tutorial

Refer to TUTORIAL.md

Contribution

Refer to CONTRIBUTING.md

FAQ

1. Can the library do live interactive testing?

A: The library is not built for live, interactive testing. The Pipeline class can only test a pre-scripted sequence of turns, not a dynamic one. The purpose of this tool is to bring version control and A/B testing to the foundational, single-turn prompts.

2. Does it work with any LLM? A: Works with OpenAI, Anthropic for now