1Q1W (One Question One World)

Generate a set of unique evaluation criteria from one question or a list of questions using LLMs. The pipeline expands scenarios, perspectives, and criteria through multiple iterations, then deduplicates and assigns scores.

Installation

From PyPI:

pip install oneq1w

From GitHub:

pip install "oneq1w @ git+https://github.com/mims-harvard/1Q1W.git"

From local clone (for development):

git clone https://github.com/mims-harvard/1Q1W.git
cd 1Q1W
pip install -e .

API Keys (.env)

Create a .env file in the project root with the keys for your chosen provider:

# OpenAI (default)
OPENAI_API_KEY=sk-...

# Azure OpenAI
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com

# Claude
ANTHROPIC_API_KEY=sk-ant-...

# Gemini
GOOGLE_API_KEY=...   # or GEMINI_API_KEY

# Grok
XAI_API_KEY=...

# DeepSeek
DEEPSEEK_API_KEY=...

# vLLM
VLLM_SERVER_URL=http://localhost:8000/v1

Load API keys before running or use python-dotenv to load .env in your script.

Quick Start

from oneq1w import CriteriaGenerator

gen = CriteriaGenerator(model="gpt-4o")

# Single question (string)
result = gen.generate("What is machine learning?")
print(result["final_criteria"])

# Single question (dict)
result = gen.generate({"id": "q1", "question": "What is AI?"})

# Batch
results = gen.generate([
    {"id": "q1", "question": "What is AI?"},
    {"id": "q2", "question": "How does deep learning work?"},
])

Input Arguments

1. CriteriaGenerator Init Parameters

2. generate() Input Arguments

• id: Identifier for the question (auto-assigned if omitted).
• question: The question text.
• image: Base64 string (with or without data:image/...;base64, prefix) for vision-capable models.
• web_content: Retrieved web context; appended to the question as [Retrieved Web Context] ... [End of Web Context].

Output Format

{
    "id": "q1",
    "question": "What is AI?",
    "scenarios": [...],
    "raw_perspectives": [...],
    "reviewed_perspectives": [...],
    "raw_criteria": [...],
    "reviewed_criteria": [...],
    "final_criteria": [
        {"criterion": "Explains core concepts", "points": 3},
        {"criterion": "Provides examples", "points": 2},
    ],
}

Supported Models

vLLM: Uses a separate server for hosting. You need to install the vLLM package suitable for your server environment or use the official vLLM Docker image. Once the server is running, provide the model name and the server URL (e.g. VLLM_SERVER_URL=http://localhost:8000/v1 or base_url when initializing CriteriaGenerator).

Data

Raw data and generated criteria (gpt-4.1 with scenario expand x3, perspective expand x4, criteria expand x3) are available at https://huggingface.co/datasets/suyc21/1Q1W.

Test Scripts

This folder contains several test scripts: