table2text

Repo maps for tables. table2text reads one messy CSV and prints a compact, token-budgeted brief — the structure, the schema, the traps, and the exact pandas code to load it — so an LLM agent can write a correct extraction script without ever opening the file.

$ table2text Freighting_goods.csv
FILE: Freighting_goods.csv — 221 rows x 184 cols, utf-8, delimiter ','
VERDICT: report sheet, not clean — 9 stacked tables inside
MAP: 0-2 banner | 4-5 metadata | 7-20 prose | T1 23-36 | T2 39-64 | ... | 207-219 footnotes
TABLES:
T1 23-36 hdr 23-24 d2/7g 12r x 31c: [0-2] 3 str; [3-30] 28 float
...
QUIRKS:
  9 stacked tables — read each by its own row offsets
  T1-T4, T8: multi-row header (row span on each TABLES line): flatten the column index after load
  T1-T3, T6: forward-fill after load (labels written once per group): [0] Activity, [1] Type
  T1-T9: cols beyond each table's own column count are empty padding (file is 184 wide): trim after load
RECIPE:
```python
import pandas as pd
p = 'Freighting_goods.csv'
specs = [(23,12,2,31), (39,24,2,19), ...]
dfs = [pd.read_csv(p, skiprows=s, nrows=n,
       header=list(range(h)) if h else None).iloc[:, :c]
       for s, n, h, c in specs]

That file is a real UK-government spreadsheet: banner rows, a metadata block, 14 rows of prose guidance, nine stacked sub-tables with two-row spanning headers, forward-filled label columns, and 150 columns of empty padding. Every conventional schema tool (DuckDB sniff_csv, csvkit, qsv, frictionless) assumes header-on-row-1 and one clean rectangle, and reads it wrong. Agents figure it out by scrolling and re-reading — an expensive detective loop, repeated per file, per session. table2text runs that loop once, deterministically, and hands over the conclusions.

Install

pip install .            # from a checkout
pip install .[exact-tokens]   # + tiktoken for exact token counting

Python ≥ 3.11, stdlib only. tiktoken is optional: without it the budget is enforced against a conservative chars/4 estimate.

Usage

table2text data.csv                 # the brief, ≤2000 tokens, to stdout
table2text data.csv --budget 500    # tighter: coarser view of the same facts
table2text data.csv --json          # full table-AST as JSON (for programs)

One command, two flags, zero config. Clean CSVs produce naturally tiny briefs (~150 tokens) at any budget — the budget is a ceiling, never a target.

Why the default budget is 2000 tokens — do not "helpfully" raise it

The brief must be a single-gulp artifact: small enough that every agent harness ingests it in one untruncated tool-call read, so no agent ever greps within it. The binding constraint is the stingiest mainstream harness — Codex CLI truncates tool output at 256 lines or ~10 KiB (~2.5k tokens), head+tail, which silently deletes the middle of anything larger (and the middle of this brief is the schema). 2000 tokens with a 240-line guard is the largest brief that survives every harness intact. Context-rot research points the same way: model accuracy degrades with input length from the tens of thousands of tokens, so a brief should be a negligible fraction of an agent's remaining healthy context.

If the budget is smaller than a file's structural floor (the irreducible cost of naming its regions, tables and read parameters), the floor brief ships anyway, with a warning on stderr — a complete description over budget beats a within-budget one that omits the map or the recipe.

How it works — Extract → Rank → Cut (no AI anywhere)

The architecture mirrors Aider's repo map: deterministic structure extraction, a salience formula, and a greedy fill into a hard token budget.

1. Extract (dialect.py, rows.py, regions.py, header.py, columns.py, ast.py): sniff the dialect; classify every row (banner / metadata / prose / header / data / blank) with Pytheas-style type-coherence rules; group rows into regions, splitting stacked tables on repeated headers; resolve multi-row headers into a column tree with spanning groups and units; compute per-column stats in one streaming pass. Output: a lossless table-AST — coordinates, names, types, aggregates. Bulk cell data is never stored (≤3 sample rows, cells truncated).
2. Rank (rank.py): structure, boundaries, quirks and recipes are never cut; key/enum columns render rich; homogeneous bulk columns collapse into pattern lines; individual data rows are represented only by stats.
3. Cut (render.py, budget.py, brief.py): a fixed degradation ladder (drop samples → drop per-column stats → merge columns into runs) renders the richest version that fits --budget. Smaller budgets yield a coarser view of the same facts, never different facts.

Everything is deterministic heuristics — cheap, fast, reproducible, no LLM calls in the pipeline.

Development

uv run --with pytest,tiktoken python3 -m pytest -q

Test fixtures include 24 real UK-government GHG Conversion Factors 2025 CSVs (gov.uk, Open Government Licence v3.0 — see tests/fixtures/ghg-2025-condensed/README.md), exercising nearly every messy-spreadsheet idiom. The acceptance bar is executable: extractors derived only from the brief text must reproduce hand-verified ground-truth rows (tests/test_acceptance.py).

Design docs: CLAUDE.md (project context), docs/M3-design-spec.md (renderer/budget design and its rationale).