gomokubench 0.1.1

A reproducible search-vs-LLM Gomoku benchmark for AI labs, model builders, and researchers.

GomokuBench

GomokuBench is a lightweight benchmark for testing frontier LLMs against a search-powered Gomoku engine in a setting that is simple, adversarial, reproducible, and easy to inspect move by move.

It is built for AI companies, model builders, and researchers who want a fast way to answer a practical question:

Can a general-purpose language model consistently beat a classical search algorithm in a fully specified board game?

Until 2026.4.26, no LLM in this benchmark has beaten the built-in AlphaBeta search engine. If you find one, please share it with us.

GomokuBench is designed to be easy to plug into model APIs, easy to run from the command line, and easy to audit after the game ends. Every move can be replayed, every prompt is explicit, and every result is saved as structured JSON.

Author: Homer Quan
GitHub: homerquan/GomokuBench

Why This Benchmark

• Simple game, hard reasoning: Gomoku has clear rules and no hidden information, so failures are easier to interpret.
• Search vs LLM: benchmark a deterministic AlphaBeta engine against modern chat models under the same rules.
• Fast iteration: add a new model with a small JSON config and start benchmarking right away.
• Useful outputs: save per-game logs, final boards, and aggregate win/loss results for later analysis.
• Good for demos and research: use it for model evals, prompting experiments, tool-use studies, and public scoreboards.

Current Results

As of 2026.4.26, every model listed below is still down 10:0 against the built-in AlphaBeta search engine.

Model	AlphaBeta (Search)	LLM
nemotron-3-super	10	0
gemma4:latest	10	0
deepseek-v4-pro-together	10	0
kimi-k2.6-novita	10	0
gpt-5.5-pro-openrouter	10	0
gpt-5-mini	10	0
gemini-3-flash-preview	10	0

Quick Start

Install from PyPI:

pip install gomokubench

Play against the engine:

gomoku play

Benchmark an LLM:

gomoku benchmark --model nemotron-3-super -r 10

Give the model a better chance with a weaker engine:

gomoku benchmark --model nemotron-3-super -r 10 --ai-level easy

Requirements

• Python 3
• numpy

Install

Or install from source:

pip install .

Play

gomoku play

Optional flags:

• --player black|white
• --ai-first
• --ai-level easy|standard|hard

The CLI always uses a 19x19 board.

AI levels:

• easy: shallower search with a small amount of move randomness
• standard: the default benchmark setting
• hard: deeper search

Moves use x,y with 1-based coordinates, for example 10,10.

Benchmark

Run an LLM against the built-in alpha-beta AI:

gomoku benchmark --model nemotron-3-super -r 10

To give the LLM a better chance, benchmark against a weaker engine:

gomoku benchmark --model nemotron-3-super -r 10 --ai-level easy

To watch the rounds play out in the console while benchmarking:

gomoku benchmark --model nemotron-3-super -r 10 -v

What this does:

• Loads the model config from models/nemotron-3-super.json
• Runs 10 rounds total
• Uses balanced starts: 5 rounds with the AI moving first and 5 rounds with the LLM moving first
• Always uses a 19x19 board
• Uses the selected AI level, defaulting to standard
• -v prints each round, move, and board state in the console
• Saves the benchmark report to benchmarks/nemotron-3-super.json

The benchmark report is saved as JSON and includes the summary plus per-game move logs and final boards.

What Gets Saved

Each benchmark run saves a JSON report in benchmarks/ with:

• model name and provider
• board size and AI level
• total wins, losses, and draws
• which side moved first in each round
• full move logs
• final board states

That makes GomokuBench useful both as a quick CLI demo and as a small research harness for repeatable comparisons across model versions and providers.

Adding Models

This repo now includes a few example model configs in the models/ folder.

You can add another model by creating a new JSON config that uses an OpenAI-compatible chat completions API format.

In general:

• add a new config file under models/
• point it at an OpenAI-compatible baseURL
• set the remote model name
• add any required API key env var to .env

Examples in this repo include Ollama-compatible, Hugging Face Router, and OpenRouter model configs.