oneq1w 0.1.0

Generate evaluation criteria for any question using LLMs

1Q1W (One Question One World)

Generate a set of unique evaluation criteria from one question or a list of questions using LLMs. The pipeline expands scenarios, perspectives, and criteria through multiple iterations, then deduplicates and assigns scores.

Installation

One command (with conda/venv activated):

pip install "oneq1w[all] @ git+https://github.com/mims-harvard/1Q1W.git"

[all] adds Claude, Gemini, and sentence-transformers. Minimal: pip install git+https://github.com/mims-harvard/1Q1W.git

From local clone (for development):

git clone https://github.com/mims-harvard/1Q1W.git
cd 1Q1W
pip install -e ".[all]"

API Keys (.env)

Create a .env file in the project root with the keys for your chosen provider:

# OpenAI (default)
OPENAI_API_KEY=sk-...

# Azure OpenAI
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com

# Claude
ANTHROPIC_API_KEY=sk-ant-...

# Gemini
GOOGLE_API_KEY=...   # or GEMINI_API_KEY

# Grok
XAI_API_KEY=...

# DeepSeek
DEEPSEEK_API_KEY=...

# vLLM
VLLM_SERVER_URL=http://localhost:8000/v1

Load API keys before running or use python-dotenv to load .env in your script.

Quick Start

from oneq1w import CriteriaGenerator

gen = CriteriaGenerator(model="gpt-4o")

# Single question (string)
result = gen.generate("What is machine learning?")
print(result["final_criteria"])

# Single question (dict)
result = gen.generate({"id": "q1", "question": "What is AI?"})

# Batch
results = gen.generate([
    {"id": "q1", "question": "What is AI?"},
    {"id": "q2", "question": "How does deep learning work?"},
])

Input Arguments

1. CriteriaGenerator Init Parameters

Parameter	Type	Default	Description
model	str	"gpt-4.1"	Model name (auto-detects provider)
base_url	str	None	API base URL (required for vLLM)
api_key	str	None	API key (uses env vars if not provided)
temperature	float	0.4	Generation temperature
embedding_model	str	"text-embedding-3-small"	Model for embeddings (deduplication)
n_scenario_expands	int	3	Scenario expansion iterations
n_perspective_expands	int	4	Perspective expansion iterations
n_criteria_expands	int	3	Criteria expansion iterations
dedup_threshold	float	0.6	Cosine similarity threshold for deduplication (0-1)
max_workers	int	8	Parallel workers for batch processing
max_retries	int	5	Max retries on rate limit errors
debug	bool	False	Print raw LLM outputs for debugging

2. generate() Input Arguments

Input Type	Format	Required	Optional
str	"question text"	—	—
dict	{"id": "...", "question": "..."}	id, question	image, web_content
list	[{...}, {...}]	each item has id, question	image, web_content per item

• id: Identifier for the question (auto-assigned if omitted).
• question: The question text.
• image: Base64 string (with or without data:image/...;base64, prefix) for vision-capable models.
• web_content: Retrieved web context; appended to the question as [Retrieved Web Context] ... [End of Web Context].

Output Format

{
    "id": "q1",
    "question": "What is AI?",
    "scenarios": [...],
    "raw_perspectives": [...],
    "reviewed_perspectives": [...],
    "raw_criteria": [...],
    "reviewed_criteria": [...],
    "final_criteria": [
        {"criterion": "Explains core concepts", "points": 3},
        {"criterion": "Provides examples", "points": 2},
    ],
}

Supported Models

Provider	Models	Env Variable
OpenAI	gpt-4o, gpt-4, o1-mini, o3-mini, o4-mini	OPENAI_API_KEY
Azure OpenAI	gpt-4o, gpt-4	AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT
Claude	claude-3-opus, claude-3-sonnet, claude-3-haiku	ANTHROPIC_API_KEY
Gemini	gemini-pro, gemini-1.5-pro, gemini-2.0-flash	GOOGLE_API_KEY
Grok	grok-2, grok-3	XAI_API_KEY
DeepSeek	deepseek-chat, deepseek-reasoner	DEEPSEEK_API_KEY
vLLM	any model	VLLM_SERVER_URL or base_url param

vLLM: Uses a separate server for hosting. You need to install the vLLM package suitable for your server environment or use the official vLLM Docker image. Once the server is running, provide the model name and the server URL (e.g. VLLM_SERVER_URL=http://localhost:8000/v1 or base_url when initializing CriteriaGenerator).

Data

Raw data and generated criteria (gpt-4.1 with scenario expand x3, perspective expand x4, criteria expand x3) are available at https://huggingface.co/datasets/suyc21/1Q1W.

Test Scripts

This folder contains several test scripts:

Script	Description
test_providers.py	Tests different providers (OpenAI, Claude, Gemini, Grok, DeepSeek, vLLM) on HealthBench sample data or HLE sample data. Usage: python test_providers.py --provider openai
test_pipeline.py	Tests generating HealthBench data using gpt-4.1.
evaluate_criteria.py	Evaluates pre-generated criteria against expert rubrics. Computes coverage and uniqueness metrics. Use --analyze-only to compute scores only without re-running LLM evaluation. Usage: python evaluate_criteria.py -i results.json --analyze-only