flatperf 0.1.1

Profiling toolkit for Flatseek build pipeline

Flatperf

Performance diagnostics toolkit for Flatseek — identify bottlenecks in indexing and query pipelines.

---

Installation

PyPI

pip install flatperf

Requires Python 3.10+. Dependencies (flatseek, flatbench) are installed automatically.

Development

git clone https://github.com/flatseek/flatperf.git
cd flatperf
pip install -e .

---

CLI Commands

flatperf generate

Generate test CSV data for profiling.

flatperf generate --schema article --rows 1000 --output /tmp/test.csv
flatperf generate --schema transactions --rows 100000 --output /tmp/trans.csv

Schema	Fields
article	id, title, content, author, category, published_at, views, status
users	id, name, email, country, created_at, age, is_active
transactions	id, user_id, amount, currency, merchant, location, created_at, status

Flag	Default	Description
--schema	article	Data schema to generate
--rows	1000	Number of rows to generate
--output	(required)	Output CSV file path

---

flatperf profile

Profile a single Flatseek build.

flatperf profile /tmp/data.csv --rows 100000

# Save profile for snakeviz
flatperf profile /tmp/data.csv --top 30 --out /tmp/build.prof
snakeviz /tmp/build.prof

Sample Output:

input:    /tmp/article_100k.csv
rows:     100,000
workers:  1
output:   /var/folders/c_/.../flatprofile_idx_abc123

══════════════════════════════════════════════════════════════════════════════
 wall summary
══════════════════════════════════════════════════════════════════════════════
 wall:        93.14s
 docs:        100,000
 throughput:  1,074 docs/s

══════════════════════════════════════════════════════════════════════════════
 top 25 hotspots — sorted by tottime
══════════════════════════════════════════════════════════════════════════════
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   387094   28.534    0.000   28.534    0.000 {method 'acquire' of '_thread.lock' objects}
    900000   25.236    0.000   43.227    0.000 builder.py:875(_index_value)
  ...

Flag	Default	Description
csv	(required)	CSV / JSON file or directory to index
-w, --workers N	1	Number of parallel build workers
-r, --rows N	(unbounded)	Profile only the first N rows
-n, --top N	25	Hotspots to show in each ranking
--out PATH	(none)	Write a binary .prof file
--keep-output	off	Keep the temporary index after profiling
--flatseek-src DIR	auto	Path to flatseek source

---

flatperf compare

A/B benchmark a Flatseek build, repeated N times.

flatperf compare ./data/big.csv --rows 100000 --runs 3 --tag "baseline"
flatperf compare ./data/big.csv --rows 100000 --runs 3 --tag "after-cache"

Sample Output:

input:   /tmp/article_100k.csv
rows:    100,000
workers: 1
runs:    3

  run 1/3:    93.14s  (1,074 docs/s)
  run 2/3:    91.82s  (1,089 docs/s)
  run 3/3:    94.01s  (1,064 docs/s)

────────────────────────────────────────────────────────────────────────────
tag                      runs   min        median     max        docs/s (med)
────────────────────────────────────────────────────────────────────────────
baseline                 3      91.82s     93.14s     94.01s          1074

Flag	Default	Description
csv	(required)	CSV / JSON file or directory
-n, --runs N	3	Number of repeated builds
-w, --workers N	1	Parallel workers per run
-r, --rows N	(unbounded)	Trim input to first N rows
--tag NAME	build	Label for the result row
--flatseek-src DIR	auto	Path to flatseek source

---

flatperf search

Profile a single search query.

flatperf search ./data "program:raydium AND signer:*7xMg*"
flatperf search ./data "ERROR" --top 30 --out /tmp/search.prof

Sample Output:

══════════════════════════════════════════════════════════════════════════════
 search result
══════════════════════════════════════════════════════════════════════════════
 query:      program:raydium
 total:      12,456 matches
 returned:   20 docs
 wall:       0.0034s
 qps:        294.1 queries/s

══════════════════════════════════════════════════════════════════════════════
 top 25 hotspots — sorted by tottime
══════════════════════════════════════════════════════════════════════════════
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      542    0.001    0.000    0.001    0.000 {method 'acquire' of '_thread.lock' objects}
       42    0.000    0.000    0.003    0.000 query_engine.py:234(_search_trigrams)
  ...

Flag	Default	Description
data_dir	(required)	Index directory
query	(required)	Search query string
-n, --top N	25	Hotspots to show
-p, --page-size N	20	Number of results per page
--flatseek-src DIR	auto	Path to flatseek source

---

flatperf bench-search

Benchmark search queries, repeated N times to get latency percentiles.

flatperf bench-search ./data "program:raydium" --runs 100
flatperf bench-search ./data "status:ERROR" --runs 50 --tag "error-queries"

Sample Output:

query:    program:raydium
data:     ./data
runs:     100

  run 1/100: 3.45ms  (12,456 matches)
  run 2/100: 3.21ms  (12,456 matches)
  ...

────────────────────────────────────────────────────────────────────────────
query                         runs   min(ms)    median     p95       p99       qps
────────────────────────────────────────────────────────────────────────────
program:raydium               100    2.98       3.34       4.12      5.87      299.4

Flag	Default	Description
data_dir	(required)	Index directory
query	(required)	Search query string
-n, --runs N	10	Number of repeated queries
-p, --page-size N	20	Number of results per page
--tag NAME	query	Label for the result row
--flatseek-src DIR	auto	Path to flatseek source

---

flatperf join

Profile a single join query.

flatperf join ./data "dataset:logs" "service:api" --on trace_id

Sample Output:

══════════════════════════════════════════════════════════════════════════════
 join result
══════════════════════════════════════════════════════════════════════════════
 query_a:    dataset:logs
 query_b:    service:api
 join_on:    trace_id
 total:      12,450 pairs
 returned:   20 pairs
 wall:       0.0089s
 qps:        112.4 joins/s

Flag	Default	Description
data_dir	(required)	Index directory
query_a	(required)	First query
query_b	(required)	Second query
--on	(required)	Shared field to join on
-n, --top N	25	Hotspots to show
-p, --page-size N	20	Number of results per page
--flatseek-src DIR	auto	Path to flatseek source

---

flatperf bench-join

Benchmark join queries, repeated N times.

flatperf bench-join ./data "dataset:logs" "service:api" --on trace_id --runs 50

Sample Output:

query_a:   dataset:logs
query_b:   service:api
join_on:   trace_id
data:      ./data
runs:      50

────────────────────────────────────────────────────────────────────────────
query                             runs   min(ms)    median     p95       p99       qps
────────────────────────────────────────────────────────────────────────────
dataset:logs+service:api           50    7.12      8.94      14.23     22.10     111.9

Flag	Default	Description
data_dir	(required)	Index directory
query_a	(required)	First query
query_b	(required)	Second query
--on	(required)	Shared field to join on
-n, --runs N	10	Number of repeated runs
-p, --page-size N	20	Number of results per page
--tag NAME	auto	Label for the result row
--flatseek-src DIR	auto	Path to flatseek source

---

flatperf aggregate

Profile a single aggregation query (terms, stats, cardinality, histogram).

# Terms aggregation
flatperf aggregate ./data --aggs '{"terms":{"field":"category","size":10}}'

# Stats on numeric field
flatperf aggregate ./data --aggs '{"stats":{"field":"amount"}}'

# With query filter
flatperf aggregate ./data -q "status:ACTIVE" --aggs '{"terms":{"field":"author","size":20}}'

# Cardinality - unique users
flatperf aggregate ./data --aggs '{"cardinality":{"field":"user_id"}}'

Sample Output:

══════════════════════════════════════════════════════════════════════════════
 aggregate result
══════════════════════════════════════════════════════════════════════════════
 query:      status:ACTIVE
 aggs:       {"terms":{"field":"category","size":10}}
 wall:       0.0089s
 hits:       45,230

 terms aggregation (category):
   tech      12,450 docs
   news       9,820 docs
   sports     7,230 docs

Supported Types	Description
terms	Bucket aggregation — count docs per field value
stats	Min, max, sum, avg, count on numeric field
cardinality	Count unique values (approximate)
histogram	Bucket by numeric interval

Flag	Default	Description
data_dir	(required)	Index directory
-q, --query	(none)	Lucene query to filter docs
--aggs	(none)	JSON aggregation config
-s, --size N	10	Max buckets for terms aggregation
-n, --top N	25	Hotspots to show
--flatseek-src DIR	auto	Path to flatseek source

---

flatperf bench-aggregate

Benchmark aggregation queries, repeated N times.

flatperf bench-aggregate ./data --aggs '{"terms":{"field":"category","size":10}}' --runs 50
flatperf bench-aggregate ./data -q "type:article" --aggs '{"stats":{"field":"views"}}' --runs 100 --tag "article-stats"

Sample Output:

query:    {"terms":{"field":"category","size":10}}
data:     ./data
filter:   status:ACTIVE
runs:     50

────────────────────────────────────────────────────────────────────────────
aggregation                   runs   min(ms)   median    p95       p99       qps
────────────────────────────────────────────────────────────────────────────
terms(category)                 50    6.12      8.34     12.45     18.23     119.8

Flag	Default	Description
data_dir	(required)	Index directory
-q, --query	(none)	Lucene query to filter docs
--aggs	(none)	JSON aggregation config
-s, --size N	10	Max buckets
-n, --runs N	10	Number of repeated runs
--tag NAME	aggregate	Label for the result row
--flatseek-src DIR	auto	Path to flatseek source

---

flatperf encrypt

Profile index encryption.

flatperf encrypt ./data --passphrase "mysecretpass" --top 30
flatperf encrypt ./data --passphrase "mysecretpass" --workers 8

Sample Output:

══════════════════════════════════════════════════════════════════════════════
 encrypt summary
══════════════════════════════════════════════════════════════════════════════
 wall:       45.23s

══════════════════════════════════════════════════════════════════════════════
 top 25 hotspots — sorted by tottime
══════════════════════════════════════════════════════════════════════════════
  524288    32.45    0.000   32.45    0.000 query_engine.py:156(encrypt_bytes)
   65600     8.12    0.000   12.34    0.000 builder.py:452(_encrypt_file)
  ...

Flag	Default	Description
data_dir	(required)	Index directory
--passphrase	(required)	Encryption passphrase
-w, --workers N	auto	Parallel workers
-n, --top N	25	Hotspots to show
--flatseek-src DIR	auto	Path to flatseek source

---

flatperf decrypt

Profile index decryption.

flatperf decrypt ./data --passphrase "mysecretpass"

Flag	Default	Description
data_dir	(required)	Index directory
--passphrase	(required)	Decryption passphrase
-n, --top N	25	Hotspots to show
--flatseek-src DIR	auto	Path to flatseek source

---

flatperf compress

Profile index compression.

flatperf compress ./data
flatperf compress ./data --level 9
flatperf compress ./data --workers 8

Sample Output:

══════════════════════════════════════════════════════════════════════════════
 compress summary
══════════════════════════════════════════════════════════════════════════════
 wall:        28.45s
 before:     256.3 MB
 after:      89.7 MB
 saved:      166.6 MB  (2.86x)

══════════════════════════════════════════════════════════════════════════════
 top 25 hotspots — sorted by tottime
══════════════════════════════════════════════════════════════════════════════
  65536     18.23    0.000   18.23    0.000 {built-in method zlib.compress}
      1      5.12    0.000   23.45    0.000 builder.py:892(cmd_compress)
  ...

Flag	Default	Description
data_dir	(required)	Index directory
-l, --level N	6	Compression level 1-9
-w, --workers N	auto	Parallel workers
-n, --top N	25	Hotspots to show
--flatseek-src DIR	auto	Path to flatseek source

---

flatperf delete

Benchmark index deletion.

flatperf delete ./data --runs 3
flatperf delete ./data --workers 16

Sample Output:

data:     /tmp/flatperf_deleterun_0_abc123
runs:     3

  run 1/3:    12.34s
  run 2/3:    11.89s
  run 3/3:    13.21s

────────────────────────────────────────────────────────────────────────────
operation              runs   min        median     max
────────────────────────────────────────────────────────────────────────────
delete                 3      11.89s     12.34s     13.21s

Flag	Default	Description
data_dir	(required)	Index directory to delete
-n, --runs N	3	Number of repeated runs
-w, --workers N	auto	Parallel workers
--flatseek-src DIR	auto	Path to flatseek source

---

Reading the Output

cProfile output shows two rankings:

• tottime — seconds spent inside the function itself (excluding callees). Optimize the top of this list to win wall time.

• cumtime — seconds spent inside the function plus everything it called. Use it to find which code subtree dominates a long-running call.

A function with high cumtime but low tottime is just a wrapper — its expensive callee is the actual cost. Look further down.

Recipe — Find the Next Bottleneck

# 1. Generate test data and establish a baseline
flatperf generate --schema article --rows 100000 --output /tmp/data.csv
flatperf compare /tmp/data.csv --rows 100000 --runs 3 --tag "baseline"

# 2. Profile to see where time goes
flatperf profile /tmp/data.csv --rows 100000 --top 30 \
    --out /tmp/build.prof
snakeviz /tmp/build.prof

# 3. Pick the highest-tottime function and inspect it. Edit. Test.

# 4. Confirm the gain
flatperf compare /tmp/data.csv --rows 100000 --runs 3 --tag "after-X"

# 5. For search: profile individual queries
flatperf search /tmp/data "common_query" --top 30

# 6. Benchmark search latency
flatperf bench-search /tmp/data "common_query" --runs 100

Repeat until the top-of-list is "intrinsic work" (per-cell tokenize, per-term encode) — not setup, parsing, or IO that can be cached.

Development

pip install -e .
pip install pytest
pytest

License

Apache 2.0