About Qworld

Evaluating large language models (LLMs) on open-ended questions is difficult because response quality depends on the question's context. Binary scores and static rubrics fail to capture these context-dependent requirements. Existing methods define criteria at the dataset level or generate them in a single pass, which limits their ability to explore the evaluation space implied by each question.

We introduce One-Question-One-World (Qworld), a method that generates question-specific evaluation criteria using a recursive expansion tree. Given a question, Qworld decomposes it into scenarios, perspectives, and fine-grained binary criteria through structured hierarchical and horizontal expansion. The resulting criteria specify what a high-quality answer must address for that question.

On HealthBench, Qworld covers 89% of expert-authored criteria and generates 79% novel criteria validated by human experts. Experts rate Qworld criteria higher in insight and granularity than those produced by prior methods. When applied to 11 frontier LLMs on HealthBench and Humanity's Last Exam, Qworld reveals capability differences in dimensions such as long-term impact, equity, error handling, and interdisciplinary reasoning that coarse rubrics do not distinguish.

By formulating criteria generation as structured coverage of question-implied evaluation axes, Qworld enables evaluation that adapts to each question rather than relying on fixed task-level criteria.

Installation

# From PyPI
pip install qworld

# With local embedding fallback (optional)
pip install "qworld[local-embeddings]"

# From GitHub
pip install "qworld @ git+https://github.com/mims-harvard/Qworld.git"

git clone https://github.com/mims-harvard/Qworld.git
cd Qworld
pip install -e .

Embedding Dependency Behavior

API Keys

Create a .env file in the project root with the keys for your chosen provider:

# OpenAI (default)
OPENAI_API_KEY=sk-...

# Azure OpenAI
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com

# Claude
ANTHROPIC_API_KEY=sk-ant-...

# Gemini
GOOGLE_API_KEY=...   # or GEMINI_API_KEY

# Grok
XAI_API_KEY=...

# DeepSeek
DEEPSEEK_API_KEY=...

# vLLM
VLLM_SERVER_URL=http://localhost:8000/v1

Quick Start

Qworld supports three usage scenarios:

1. 🌐 Try it online — Want to experience Qworld quickly? Try our interactive demo at qworld.openscientist.ai.
2. 🤖 Agentic systems — Want to integrate Qworld into your agentic workflow? Check out our skills for plug-and-play usage.
3. 📊 Batch generation — Need to generate criteria for an entire dataset? Use the code below — pass a list of questions and set max_workers to parallelize.

from qworld import CriteriaGenerator

gen = CriteriaGenerator(model="gpt-4.1")

# Single question (string)
result = gen.generate("What is machine learning?")
print(result["final_criteria"])

# Single question (dict)
result = gen.generate({"id": "q1", "question": "What is AI?"})

# Batch
results = gen.generate([
    {"id": "q1", "question": "What is AI?"},
    {"id": "q2", "question": "How does deep learning work?"},
])

CriteriaGenerator Parameters

generate() Input

• image — Base64 string (with or without data:image/...;base64, prefix) for vision-capable models.
• web_content — Retrieved web context; appended as [Retrieved Web Context] ... [End of Web Context].

Output Format

{
    "id": "q1",
    "question": "What is AI?",
    "scenarios": [...],
    "raw_perspectives": [...],
    "reviewed_perspectives": [...],
    "raw_criteria": [...],
    "reviewed_criteria": [...],
    "final_criteria": [
        {"criterion": "Explains core concepts", "points": 3},
        {"criterion": "Provides examples", "points": 2},
    ],
}

Supported Models

vLLM requires a separate server. Install the vLLM package for your environment or use the official Docker image. Once the server is running, provide the model name and server URL (e.g. VLLM_SERVER_URL=http://localhost:8000/v1 or base_url in CriteriaGenerator).

Data

Raw data and generated criteria (gpt-4.1 with scenario expand ×3, perspective expand ×4, criteria expand ×3) are available on 🤗 HuggingFace.