Fun Hypothesis

Test whether different prompt framings affect LLM output quality.

Hypothesis

LLM output quality varies based on how prompts are framed. This tool tests that hypothesis with rigorous double-blind methodology.

Quick Start

pip install fun-hypothesis

# Run with default "fun" framing
fun-hypothesis --prompt "Explain quantum computing"

# Try different framings
fun-hypothesis --prompt "Explain quantum computing" --framing pirate
fun-hypothesis --prompt "Explain quantum computing" --framing expert
fun-hypothesis --prompt "Explain quantum computing" --framing eli5

Built-in Framings

Methodology

1. Session A: Send raw prompt to LLM, collect response
2. Session B: Have LLM transform prompt with framing
3. Session C: Send framed prompt to LLM, collect response
4. Session D: Judge panel evaluates Response A (blind)
5. Session E: Judge panel evaluates Response C (blind)
6. Compare scores, iterate, aggregate, analyze

All sessions are independent (no context leakage). Judging is double-blind.

Requirements

• Python 3.10+
• Anthropic API key (ANTHROPIC_API_KEY environment variable)

Documentation

• How It Works
• Statistical Methodology

License

MIT