Flatbench

Search engine benchmark suite — compare Flatseek against Elasticsearch, tantivy, Typesense, Whoosh, ZincSearch, SQLite, and DuckDB.

Benchmarks: build speed, search latency, wildcard, range queries, and aggregations. Results saved as JSON + Markdown to ./output/.

Install

pip install flatbench

Requires Python ≥ 3.10, Docker (for full engine comparison).

Quick Start

1. Start all search engines (Docker)

flatbench make up

Starts: Flatseek API (port 8000), Elasticsearch (9200), Typesense (8108), ZincSearch (4080).

2. Generate a dataset

flatbench generate -s article -r 500000 -o ./data/article.csv

3. Run benchmark comparison

flatbench compare -e flatseek_cli,elasticsearch,tantivy,typesense,whoosh,zincsearch -s 500000

Results → output/benchmark_YYYYMMDD_HHMMSS.json + .md.

CLI Reference

Commands

Generate

flatbench generate --schema <schema> --rows <N> --output <path> [--format csv|jsonl]

Schemas: standard, ecommerce, logs, nested, sparse, article, adsb, campaign, devops, sosmed, blockchain

Compare

flatbench compare --engines <engines> --sizes <sizes> [options]

Options:

Engines: flatseek, flatseek_cli, elasticsearch, tantivy, typesense, whoosh, zincsearch, sqlite, duckdb

Sizes: multiple sizes supported, e.g. --sizes 1000 10000 500000

Run

flatbench run --engine <engine> --data <path> --index-dir <path> [-o output] [--iterations N]

Serve

flatbench serve [--dir ./output] [--port 8080]

Opens the report viewer in your browser automatically.

Make (Infrastructure)

flatbench make <targets...>     # Run Makefile targets (default: help)
flatbench make up               # Start all services (docker-compose up -d)
flatbench make down             # Stop services (keep volumes)
flatbench make clean             # Stop and remove volumes
flatbench make status            # Show service status
flatbench make logs              # View logs (follow mode)
flatbench make benchmark NROWS=500000   # Run benchmark via Make

Service management:

Examples

# Generate article dataset (500K rows)
flatbench generate -s article -r 500000 -o ./data/article.csv

# Compare at single scale
flatbench compare -e flatseek_cli,elasticsearch -s 500000

# Compare at multiple scales
flatbench compare -e flatseek,tantivy -s 1000 10000 500000

# Use existing CSV (reuse generated data)
flatbench compare -e flatseek,elasticsearch -s 500000 -S ./data/article.csv

# RAM-backed index (tmpfs mode, faster builds)
flatbench compare -e flatseek,tantivy -s 500000 -m tmpfs

# Compare and auto-serve report
flatbench compare -e flatseek,tantivy -s 500000 --serve

# Run benchmark via Make
flatbench make benchmark NROWS=500000 ENGINES="flatseek_cli,elasticsearch,tantivy"

Service URLs:

Available Schemas

Benchmark Operations

Output

Results written to ./output/ with timestamps:

output/
├── benchmark_20260501_142947.json   # Full structured results
├── benchmark_20260501_142947.md     # Markdown summary
└── index.json                        # Report manifest (for web viewer)

Report Viewer

Live: bench.flatseek.io — hosted Flatbench report viewer.

Local: Run flatbench serve --port 8080 or open report_viewer.html directly in browser.

Build Static Site

Build output directory as a static site (for self-hosted or Vercel deploy):

make build
# or
bash build.sh

Output → public/ directory with index.html, output/*.json, output/*.md.

Deploy to Vercel

make deploy        # Deploy to production (flatbench.vercel.app)
make deploy-preview  # Deploy preview build

Project Structure

flatbench/
├── Dockerfile              # Flatseek API server container
├── docker-compose.yml       # All engine containers
├── Makefile                 # Infrastructure + build commands
├── build.sh                 # Static site build script
├── report_viewer.html       # Web UI for browsing results
├── pyproject.toml           # flatbench package definition
├── src/flatbench/
│   ├── cli.py               # CLI entry point
│   ├── benchmarks/           # Benchmark orchestration + report generation
│   ├── generators/           # Synthetic data generators (schema-aware)
│   ├── runners/              # Engine runners (HTTP API / CLI)
│   │   ├── flatseek_api.py   # Flatseek HTTP API runner
│   │   ├── flatseek_cli.py   # Flatseek CLI runner
│   │   ├── elasticsearch.py   # Elasticsearch runner
│   │   ├── tantivy.py        # tantivy (Rust) runner
│   │   ├── typesense.py      # Typesense runner
│   │   ├── whoosh.py         # Whoosh runner
│   │   ├── zincsearch.py     # ZincSearch runner
│   │   ├── sqlite.py         # SQLite FTS5 runner
│   │   └── duckdb.py         # DuckDB full-text runner
│   └── output/               # Benchmark results (JSON + Markdown)

Adding a New Engine

from flatbench.runners import BaseRunner, BenchmarkResult, register_engine

@register_engine("myengine")
class MyEngineRunner(BaseRunner):
    name = "myengine"
    supports_aggregate = False
    supports_range_query = True
    supports_wildcard = True

    def build_index(self, data_path: str, **kwargs) -> BenchmarkResult:
        # Bulk API indexing logic
        pass

    def search(self, query: str, iterations: int = 10, **kwargs) -> BenchmarkResult:
        # Search via HTTP API
        pass

Then add to --engines list: --engines flatseek,myengine,...

Benchmark Results (Latest: 500K rows, article schema)

Latest Full results: bench.flatseek.io

Overall Score (60% speed · 40% correctness)

Key Takeaways

• Correctness matters: Flatseek is the only engine with zero correctness errors. Tantivy misses 99.4% of range query hits.
• Search: Tantivy fastest (0.7ms p50), but wrong. Flatseek second-fastest correct (7.9ms).
• Build: Tantivy wins (21s for 500K), but Flatseek build is reasonable (217s).
• Aggregation: Competitors (ES, tantivy) are 20–300× faster — Flatseek aggregation is a known weakness.