SightRead

Can your vision-language model read the board at first sight?

SightRead is a benchmark for exact game-state extraction from screenshots. Not whether a model can play chess or mahjong — whether it can see the board: every seat, every tile, every piece, every number, exactly.

Existing game benchmarks (GameWorld, LMGame-Bench, MET-Bench) score agents playing games, confounding perception with policy. SightRead removes policy entirely and audits perception: field-by-field, run-by-run, dollar-by-dollar.

Why this matters

In the original mahjong benchmark run that seeded this project (6 models × 6 screenshots × 3 runs, 22 fields, hand-verified ground truth):

The headline findings are more interesting than the leaderboard:

• Exact hands are the hardest field by 2× (25.7% exact) — but per-tile F1 shows models are close; all-or-nothing scoring hides it.
• Seat winds flip between runs while seat points score 100% — a clean dissociation between reading numbers (OCR) and spatial relational reasoning (which badge belongs to which seat). This is SightRead's central question.
• Populated discard piles: ~19% correct for every model. Unsolved.
• A decoy example in the prompt catches models that echo the prompt instead of looking at the image — now a first-class metric: grounding fidelity.
• A free local model beats every paid API on the hardest component metric.

What SightRead measures

Architecture

A core (game-agnostic) + packs (per-game contributions):

sightread-bench/
├── src/sightread/        # core: comparators, scorer, runner, cost, reports
├── packs/
│   ├── mahjong/          # complete: parser ported from the original benchmark
│   │   ├── pack.yaml     #    field spec + grounding traps
│   │   ├── extract.py    #    format-agnostic response parser
│   │   ├── prompt.md     #    the extraction prompt
│   │   ├── gt/           #    hand-verified ground truth (JSON)
│   │   └── images/       #    screenshots stay LOCAL (gitignored - copyrighted)
│   └── chess/            # scaffold: FEN-driven GT generation, contamination notes
├── models.example.yaml   # OpenAI-compatible endpoints; llama-server works as-is
└── results/              # calls.jsonl -> results.csv + leaderboard.md

Adding a game = adding a directory. See docs/adding-a-game.md. Scoring runs identically across games; the leaderboard is generated, not hand-edited.

Quickstart

pip install -e .

# validate a pack (schema, GT completeness, enum values)
sightread check --pack packs/mahjong

# run models (resume-safe; safe to kill and re-run)
cp models.example.yaml models.yaml   # add API keys via env vars
sightread run --pack packs/mahjong --models models.yaml --runs 3 --out results

# score + leaderboard
sightread score --pack packs/mahjong --calls results/calls.jsonl --out results

Local models: llama-server --port 5006 and point base_url at http://127.0.0.1:5006/v1 — the same OpenAI-compatible path, $0/call.

Packs

Roadmap

• port remaining parser unit tests from the original harness (68 total)
• chess pack: contamination-resistant renderer (random piece sets/skins)
• held-out private packs per game (submission-time evaluation)
• HuggingFace dataset + leaderboard Space
• prompt-scaffolding ablation: does describing the seat layout fix wind flips?

Citation

@software{sightread2026,
  title  = {SightRead: Exact Game-State Extraction Benchmark for Vision-Language Models},
  author = {ansteinhuynh},
  year   = {2026},
  url    = {https://github.com/ansteinhuynh/sightread-bench}
}

License

MIT. Screenshots from game platforms are not included in this repository — packs ship ground truth + prompts; capture images locally per the pack README.