Strop
Strop is a command-line tool that improves LLM prompts by testing them against your own data. It runs your prompt, scores the outputs with a metric you define, asks a model to revise the prompt based on the failures, and keeps a revision only when the measured results get better.
Status
Alpha. Version 0.1.0. Commands and the strop.yaml format may change between releases.
What it does
You give Strop a prompt, a dataset, and a definition of success. Strop then repeats this loop, for as many turns as you allow:
- Score the current prompt on a validation set.
- Run it on a separate training set and collect the failures, grouped by error type.
- Ask an optimizer model for revised prompts that address those failures. Each revision must cite the examples that motivated it.
- Score every revision on the same validation rows. Keep one only if the primary metric improves and none of your constraints is broken.
- Start the next turn from the best prompt.
When the loop ends, you run the final prompt once on a held-out test set. Strop prints a colored diff between your original prompt and the final one.
Everything is local. Prompt versions, results, and cached model responses are stored in a .strop/ directory inside your project. Strop has no server and no account, and it sends nothing anywhere except the model providers you configure.
Install
Strop needs Python 3.11 or newer.
pip install strop-cli
strop --version
The package is called strop-cli on PyPI because the name strop is reserved there. The command it installs is strop. To keep Strop out of your other Python environments, install it with pipx install strop-cli instead. To work on Strop itself, see Development.
Strop calls models through LiteLLM, so you can use any provider and model it supports. Set the API key your provider expects as an environment variable, or put it in a .env file in your project directory. Strop has no default model; you name every model yourself.
Quick start
The repository includes a ready-to-run example in examples/ticket_routing: 150 synthetic support tickets in five categories, with a deliberately vague starting prompt. Clone the repository to get it.
cd examples/ticket_routing
strop init \
--name ticket-routing \
--task "Route a customer message to a support category" \
--prompt prompt.yaml \
--data data.csv --id-column ticket_id \
--input customer_message --expected category --metadata channel \
--output-type label --labels billing,refund,exchange,technical,other \
--metric macro_f1 --constraint "valid_output_rate>=0.98" \
--target-model <provider/model> --optimizer-model <provider/model>
strop init writes strop.yaml and a .strop/ directory, and tells you which providers will receive your dataset rows. Nothing is sent yet. Next, check and split the data:
$ strop data check
Dataset: data.csv
Records: 150, valid: 150, invalid: 0
Columns: ticket_id, customer_message, category, channel
$ strop data split
Created split 1 (seed 42) over 150 rows:
train 90
validation 30
test 30
The split is stored, so it never changes unless you ask for a new one. Now measure the starting prompt and start the loop:
strop baseline
strop optimize --max-turns 5
strop test --confirm
strop export
These commands call your models, so their output depends on which models you choose and is not shown here. strop optimize prints each turn as it goes and, at the end, the metric before and after, a table of every candidate and why it was kept or rejected, and the prompt diff. strop export writes the final prompt and a report to exports/.
How success is measured
Strop cannot know what a correct output is. You define it, and Strop measures your definition.
| Way to define success | Use it when |
|---|---|
Labeled data (--expected) |
You have the right answer for each row: classification, extraction. |
Rules (--rule) |
You can check the output directly: length, required words, format. |
A Python function (--metric-file) |
You can compute a score in code. |
A judge rubric (--rubric-file) |
Quality needs judgment and there are no reference answers. |
Judge scores come from another language model, so they are estimates. Strop labels them as subjective and never mixes them with deterministic metrics. See docs/metrics.md for every metric and examples/support_replies for rules, a custom metric, and judge rubrics together.
How a revision is accepted
A candidate becomes the best prompt only if the primary metric improves by a minimum amount, every constraint you set still holds, the share of valid outputs does not drop, no class loses more than a set amount of F1, and the candidate wins on more validation examples than it loses. Every failed rule is recorded with the candidate.
The test partition is locked in code. Nothing the optimizer sees comes from it, and strop test refuses to run without --confirm, or a second time without --allow-rerun, because tuning a prompt after looking at the test result makes the result optimistic. docs/how-it-works.md explains the rules in detail.
Commands
| Command | Purpose |
|---|---|
strop init |
Create strop.yaml, a starter prompt, and the .strop/ directory. |
strop data check |
Validate the dataset and list invalid rows with their row numbers. |
strop data split |
Create the train, validation, and test split. |
strop baseline |
Measure the initial prompt on the validation set. |
strop eval |
Score any prompt version on the train or validation set. Changes nothing. |
strop analyze |
Count and list how a prompt fails on the training data. |
strop optimize |
Run the optimization loop. Resumable. |
strop approve, strop reject |
Decide a candidate when running with --approval manual. |
strop test |
Run the final evaluation on the held-out test set. |
strop compare |
Run one prompt on several models and compare the results. |
strop diff, strop history |
Compare prompt versions and see every decision. |
strop prompt add, strop prompt show |
Save your own edit as a version, or print a version. |
strop rollback |
Make an earlier version the best prompt. Nothing is deleted. |
strop status |
Show the split, the best prompt, and recent evaluations. |
strop export |
Write the prompt, a JSON and Markdown report, and per-example results. |
strop clean |
Delete stored cache, outputs, or the whole .strop/ directory. |
Run strop <command> --help for the options of each command. The commands never ask questions: every choice is a flag or a value in strop.yaml, so they work in scripts. Commands that print results accept --json for machine-readable output, and every command accepts --dir to name the project directory.
Configuration
strop init writes strop.yaml. This is a minimal one:
project:
name: ticket-routing
task_description: Route a customer message to a support category
dataset:
path: data.csv
columns: {customer_message: input, category: expected}
output:
type: label
labels: [billing, refund, exchange, technical, other]
models:
target: {model: <provider/model>}
optimizer: {model: <provider/model>}
metrics:
primary: macro_f1
The prompt itself is a separate file with chat messages and a list of the messages the optimizer may change:
messages:
- role: system
content: |-
You route customer messages to a support team.
Reply with the category only.
- role: user
content: |-
Message: {{customer_message}}
optimize: [system]
The optimizer never touches messages outside optimize, never changes {{variables}}, and never changes text wrapped in <strop:fixed>...</strop:fixed>. Your file is not overwritten; revisions are stored as versions. The full reference is in docs/configuration.md.
Models and providers
The roles are target (the model your prompt is for), optimizer (proposes revisions), analyzer (proposes failure patterns, defaults to the optimizer), and judge (scores rubrics). Each can be a different provider and model. Model names are LiteLLM names, such as <provider>/<model>. Pass --optimizer-model or --analyzer-model to strop optimize to change a role for one run, and use strop compare --models a,b to compare target models on the same rows.
Strop retries temporary provider errors with exponential backoff. A call that still fails is recorded as an error for that one example and does not stop the run.
Data and privacy
- Dataset rows go to the providers of the models you configure. The target model receives every row it is run on. The optimizer and analyzer receive a bounded sample of failed train examples. The judge receives each example it scores.
strop initlists the providers. evidence_examplesandevidence_charsinstrop.yamllimit how much of the failed examples is shown to the optimizer and analyzer.- Dataset content is always passed to models as quoted data, and the meta-prompts tell the models not to follow instructions found inside it.
- API keys are read from the environment or
.envand are never stored by Strop. Key-like strings are removed from logs and provider error messages. .strop/holds model outputs and cached responses. Keep it and.envout of version control.strop cleandeletes what you choose.- A
metric.pyfile is your own code and runs as written, in a separate process. Strop never executes anything found in a dataset.
Exit codes
| Code | Meaning |
|---|---|
| 0 | Success. |
| 1 | The command failed. |
| 2 | Usage or configuration error. |
| 10 | strop optimize paused and waits for strop approve or strop reject. |
| 11 | strop optimize stopped because a budget was reached. |
| 12 | The command was interrupted. Progress is saved. |
Limitations
- A small validation set gives noisy comparisons. Strop warns below 30 validation rows and refuses to optimize below 10.
- The test set is only a fair estimate if you look at it once.
- Judge metrics are estimates, and a judge model can prefer its own style.
- The optimizer is a language model. It proposes revisions that fit the failures it is shown, and the validation rules decide whether they help. A revision that passes can still be tuned to the validation set, which is what the test set is for.
- Results depend on the models you use. A prompt tuned for one model may not transfer to another.
- Token and cost data come from the provider. When a provider reports none, Strop shows it as unavailable instead of estimating it.
- Strop optimizes single prompts, not agents or tool chains.
Development
Strop uses uv.
git clone https://github.com/baranylcn/strop.git
cd strop
uv sync
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run mypy
The tests use scripted models and make no network calls. See CONTRIBUTING.md for the project layout and conventions.
License
MIT. See LICENSE.
Metadata
Release files for strop-cli 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| strop_cli-0.1.0.tar.gz | 139.1 kB | With provenance | Details for strop_cli-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| strop_cli-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for strop_cli-0.1.0-py3-none-any.whl |
Total release size: 285.2 kB
Release files / strop_cli-0.1.0.tar.gz
| Download URL | strop_cli-0.1.0.tar.gz |
|---|---|
| Size | 139.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4bcf34928aa48e967c4daf1fa923a1b0375a8cdd2e1318ba39c66042891aedae
|
|
BLAKE2b-256 checksum How to use checksums |
b3be63a107f10c6637362927c73584e086a440bf5c3e5239c942bcfa9a7dbfc1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.
Transparency logRelease files / strop_cli-0.1.0-py3-none-any.whl
| Download URL | strop_cli-0.1.0-py3-none-any.whl |
|---|---|
| Size | 146.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
44ca0ee75efbb123d9bf8198233f2a71edc30ffdf17bee9e2ad048119ef94d8f
|
|
BLAKE2b-256 checksum How to use checksums |
8d064eac49b8b5c6267ffad9f84737c22b4b48ecdc7a7cd21054613d3cbe5bff
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.
Transparency logSecurity for strop-cli 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for strop_cli-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.