psca 0.1.0

Prompt Specification Curve Analysis: multiverse analysis for LLM prompt design.

P-SCA: Prompt Specification Curve Analysis

A specification curve analysis framework for evaluating the robustness of LLM-simulated public opinion across prompt design choices. P-SCA systematically varies six prompt dimensions — model, persona format, question framing, system prompt, temperature, and few-shot examples — to measure how sensitive LLM partisan-gap estimates are to arbitrary researcher decisions. Benchmarked against ANES 2024 ground-truth survey data.

Install

pip install psca

Or from source:

git clone https://github.com/YCRG-Labs/psca && cd psca
pip install -e .

Set API keys in .env:

OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...
GOOGLE_API_KEY=...
OPENROUTER_API_KEY=...

Quickstart

# Latin Hypercube sampling — the main run
psca lhs --n_specs 600 --output full_lhs.json

# Reproduce Gemini-excluded headline numbers
psca analyze --output full_lhs.json --exclude_models gemini-2.5-flash

# Derive empirical coverage thresholds (10k permutations)
psca threshold --output full_lhs.json --exclude_models gemini-2.5-flash --n_permutations 10000

# Permutation inference for the partisan signal
psca permutation --output full_lhs.json

# Variance decomposition + Fisher r-to-z dominance
psca analyze --output full_lhs.json
psca fisher --output full_lhs.json

# Bootstrap CIs on eta-squared (5,000 resamples)
psca bootstrap --output full_lhs.json

# ANES benchmark comparison (amplification factor)
psca anes --output full_lhs.json

# Flipped specification analysis
psca flipped --output full_lhs.json

# Saltelli sampling for Sobol sensitivity indices
psca saltelli --items gun_control --output saltelli_gun.json
psca sobol --output saltelli_gun.json

Or use the Python API:

import psca

specs = psca.generate_specifications(n_specs=600, seed=42)
df = psca.load_results("full_lhs.json", exclude_models=["gemini-2.5-flash"])
psca.variance_decomposition(df)
psca.derive_coverage_threshold(df, n_permutations=10000)

Models supported

GPT-5.4, GPT-5.4-nano, Claude Sonnet 4.6, Llama 3.3 70B, Mistral Small. Gemini 2.5 Flash is queried in the same multiverse design but excluded from primary analyses on parse-rate grounds (see paper §4.1).

Project structure

Path	Purpose
src/psca/config.py	Six prompt dimensions, 20 battleground-state profiles, ANES items, cost tables
src/psca/sampler.py	Latin Hypercube and Saltelli specification generators
src/psca/prompts.py	Prompt construction from spec + profile + item
src/psca/runner.py	Async multi-provider API runner with retries
src/psca/analysis.py	Partisan gaps, η², bootstrap CIs, Sobol, permutation tests, ANES benchmarks, threshold derivation
src/psca/cli.py	CLI entry point (psca ...)
ordering_test.py	Position bias test for forced-choice framing
patch_run.py	Reruns failed specifications from a previous run
download_anes.py	ANES 2024 data download and processing

Data and logs

Results files in results/*.json are the API call logs. Each record includes the model's raw text reply in the raw_response field alongside the parsed score and full specification metadata (model, persona, framing, system prompt, temperature, few-shot count, profile, item, repeat).

Citation

If you use P-SCA in academic work, please cite both the methodology paper and the software:

@software{crainic_psca_2026,
  author  = {Crainic, Jacob and Yee, Brandon and Koh, Pairie},
  title   = {{P-SCA}: Prompt Specification Curve Analysis},
  year    = {2026},
  version = {0.1.0},
  doi     = {10.5281/zenodo.PENDING},
  url     = {https://github.com/YCRG-Labs/psca}
}

See CITATION.cff for machine-readable metadata.

License

MIT — see LICENSE.