cfprompt 0.1.1

Counterfactual prompting with adjusted paraphrase baselines

cfprompt

Counterfactual prompting with adjusted paraphrase baselines.

Companion package to "Compared to What? Baselines and Metrics for Counterfactual Prompting."

Install

pip install cfprompt   # PyPI release pending; for now:
pip install -e .       # from source clone

Quickstart

import pandas as pd
import cfprompt

df = pd.DataFrame({"question": [...]})

study = cfprompt.Study(
    data=df,
    perturb_column="question",
    target_perturbation=my_swap_function,
    prompt_template="Q: {question}\nA:",
    target_model=cfprompt.HFModel("hf-internal-testing/tiny-random-LlamaForCausalLM"),
    paraphrase_model=cfprompt.OpenAIModel("gpt-4.1"),
    classes=["A", "B", "C", "D"],
    seed=42,
)
report = study.run_all(metrics=["jsd", "flip_rate"])
report.to_excel("results.xlsx")

See examples/ for runnable scripts.

CLI

cfprompt init my_study             # scaffold a starter directory
cfprompt validate config.yaml      # schema check
cfprompt run config.yaml           # execute the study

Security

cfprompt run executes Python code referenced by the config (the target_perturbation and extract_label module:function specs are imported and called). Do not run untrusted YAML files.

Pickle / .pkl files (Study.save / DiskCache): Study.load(path) and a cache_dir populated by another user are equivalent to executing arbitrary Python on load. Never load a .pkl from an untrusted source, and treat cache_dir as user-private (do not point it at a directory another user can write to).

License

MIT.