Czech-SimpleQA

Problems and answers from OpenAI's SimpleQA eval translated into Czech. This work is based on the data from the paper:

Measuring short-form factuality in large language models Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, William Fedus arXiv preprint arXiv:2411.04368, 2024. https://arxiv.org/abs/2411.04368

There is a post on my blog with more detailed results!

What the Data Looks Like:

I Just Want the Eval Data

The file with the data lives at src/czech_simpleqa/czech_simpleqa.csv.gz, this is the full URL. Getting it with pandas looks like this:

import pandas as pd

eval_data = pd.read_csv(
    "https://raw.githubusercontent.com/jancervenka/"
    "czech-simpleqa/refs/heads/main/src/czech_simpleqa/czech_simpleqa.csv.gz"
)

I Want to Use the Python Package

The package contains everything required to run the eval end-to-end and collect the results. You can install it with pip or any other Python package manager:

pip install czech-simpleqa
python -m czech_simpleqa.eval \
    --answering_model claude-3-5-haiku-20241022 \
    --grading_model gpt-4o \
    --output_file_path output/claude-3-5-haiku-20241022.csv \
    --max_concurrent_tasks 30

CLI Arguments

• --answering_model: Model that will generate predicted answers to the problems in the eval.
• --grading_model: Model that will grade the predicted answers from the answering model.
• --output_file_path: Where to store the .csv file with the eval results.
• --max_concurrent_tasks: Maximum number of concurrent model calls (default 20).

Output File Schema

Supported Models

Models from OpenAI and Anthropic are currently supported. Environment variables OPENAI_API_KEY or ANTHROPIC_API_KEY need to be configured.