TDF — Token-Dense Format

Convert documents into a compact, reversible representation for LLM context.

TDF removes the redundancy that document formats carry for humans — repeated table headers, per-page running headers, boilerplate clauses, markup scaffolding — and emits what's left in a structured, parseable form. On table- and boilerplate-heavy documents this cuts tokens roughly in half. On prose it does very little, and this README says exactly where the line falls.

pip install tdf-converter
tdf stats report.pdf
tdf convert report.pdf --to tdf -o report.tdf

---

Where TDF helps, and where it doesn't

Measured against a Markdown rendering of the same parsed document, o200k_base, legend included. Source documents are third-party — none authored for this project.

Document	Source	Tokens saved
owid-co2-data.csv (3k rows)	owid/co2-data	53.5%
absolute_beginners.rst	numpy/numpy	8.7%
PEP 8	python/peps	5.6%
deployment.md	kubernetes/website	2.1%

The pattern is consistent and worth stating plainly: TDF's savings come from structural redundancy. Wide tables with repeating values, repeated legal or policy boilerplate, running page headers, and slide chrome all compress well. Prose and code documentation do not, because Markdown is already close to token-optimal for them and there is little redundancy left to remove.

If your documents are mostly prose, TDF will not save you money. Use Markdown.

Against a raw CSV — which is what most people actually paste into a model — the tabular win is smaller than the Markdown comparison suggests:

	vs Markdown table	vs raw CSV
owid-co2-data.csv	53.5%	15.0%
orders.csv	45.8%	39.2%

Both numbers are real. The Markdown comparison is the fair one if your pipeline renders tables to Markdown (most document converters do). The raw-CSV comparison is the fair one if you paste CSVs directly.

Synthetic benchmark corpus (higher numbers, but generated by this project)

bench/make_samples.py generates seven documents that deliberately span the four redundancy patterns TDF targets. These numbers are reproducible (python bench/make_samples.py && python bench/benchmark.py) but the corpus was constructed to contain the redundancy being removed, so treat them as an illustration of the mechanism rather than as evidence of real-world savings.

File	Markdown	TDF	Saved
operating_review.pdf	3,588	1,232	65.7%
services_agreement.docx	1,945	1,031	47.0%
orders.csv	16,982	9,196	45.8%
sales_report.xlsx	20,505	11,296	44.9%
handbook.html	4,732	2,660	43.8%
quarterly_deck.pptx	1,559	977	37.3%
runbook.md	1,355	978	27.8%

The Markdown baseline is not a strawman: MarkItDown independently produces token counts within ~1% of this project's own Markdown renderer on six of the seven files.

Tokenizer-independent. Savings differ by less than 0.5pp between o200k_base and cl100k_base on every file tested. The gains come from removing text, not from exploiting one tokenizer's quirks.

---

Fidelity

TDF is reversible. tdf verify parses the emitted output back into the document model and compares it against the original, reporting two numbers:

• distinct_recall — fraction of distinct content words recovered.
• occurrence_ratio — fraction of word occurrences recovered.

Both are reported because they measure different things and the second is lower:

Document	distinct_recall	occurrence_ratio
orders.csv	100.0%	100.0%
services_agreement.docx	100.0%	100.0%
owid-co2-data.csv	100.0%	99.7%
operating_review.pdf	100.0%	97.2%
handbook.html	100.0%	93.7%
quarterly_deck.pptx	100.0%	93.1%
PEP 8	99.5%	95.6%

The occurrence gap is mostly intentional. A running page header repeated across 40 pages is emitted once under !R, and the parser cannot know how many times to repeat it. The information ("this string ran on every page") is preserved; the repetition is not. No lossy elision is active by default in any of the rows above.

The PEP 8 row is not intentional. Eight distinct tokens are dropped, including identifiers appearing inside code samples. This is a known defect, not a design decision — see Known limitations.

What these metrics do not measure

Both are bag-of-words comparisons. They are blind to ordering and to cell-to-row assignment. A table where two rows' values have been swapped scores 100% on both. Do not treat a passing tdf verify as a guarantee of semantic correctness, particularly for data where row alignment carries meaning.

---

Skeleton mode and progressive loading

Beyond compression, TDF can emit a navigable map of a document instead of its contents.

tdf convert large_report.pdf --to skeleton     # section index, ~1-3% of full tokens
tdf expand 3 7 11                              # emit only the sections you want

With --tier, low-density regions are replaced by !E markers carrying an id, a kind, a token count and a one-line gist. An agent reads the map, decides what it needs, and requests regions by id:

tdf convert large_report.pdf --tier --to tdf   # elides index-like regions
tdf expand-elided x1                           # resolve one region

This is the pointer-not-payload pattern applied at document-region granularity. It is the most differentiated capability here — no other document converter offers it — and it is also the least validated. See Known limitations.

---

Usage

tdf convert <file> [--to tdf|md|skeleton] [-o OUT] [--no-legend] [--tier] [--raw]
tdf diff    <old> <new>            # structural document diffing (50-99% token savings over raw comparison)
tdf stats   <file> [--json]        # token counts across all output modes
tdf verify  <file> [--json]        # round-trip fidelity report
tdf expand  <file> <section-ids>   # emit selected sections
tdf expand-elided <file> <id>      # resolve an !E region
tdf validate <file.tdf>            # check structural invariants

Inputs: .pdf .docx .pptx .xlsx .xlsm .csv .tsv .html .md .txt .log

--no-legend omits the ~130-token syntax legend from each document. Use it when you can place the legend in your system prompt once instead of paying for it per document — the difference matters in RAG, where retrieving 20 chunks otherwise means paying for the legend 20 times.

from tdf.readers import read
from tdf.emit import render_tdf
from tdf.columnar import encode_columns

doc = read("report.pdf")
books = encode_columns(doc)          # must run on the same doc object
print(render_tdf(doc, codebooks=books))

---

Format

!T <n> <caption>      table declaration          !D  phrase dictionary
!C <col> <col> ...    column header row          §n  dictionary reference
!V <col>              column value codebook      ^   repeat cell above
!R                    running header/footer       !P  page boundary
!F key=value          constant field             !E  elided region

A table with coded columns:

!V product
a Bearing Standard
b Coupler XL
!C order_id product region amount qty
10382 a g 74974 21
10315 a g 65256 38

Full grammar in docs/SPEC.md. tdf validate checks any .tdf file against the structural invariants.

---

How this compares

	TDF	LLMLingua / LongLLMLingua	Gist / soft prompts	Docling / Marker / MinerU
Mechanism	structural redundancy removal	perplexity-based token pruning	learned embeddings	layout extraction
Reversible	yes	no	no	n/a
Needs a model	no	small LM	training + embedding access	ML layout models
Compression	~1.5–2x on tables	2–5x	up to 20–26x	n/a
Runtime	milliseconds, CPU	GPU-assisted	GPU	GPU/CPU

TDF is not a competitor to Docling, Marker or MinerU. They extract structure from PDFs using trained layout models; TDF encodes an already-parsed document. Its built-in PDF reader is PyMuPDF-based heuristics and is not competitive with theirs on hard pages. Feeding TDF from one of those tools is the better pipeline: they extract, TDF encodes.

Against the prompt-compression family, TDF occupies a different point: deterministic, reversible, and usable through any API with no model access. It compresses less. It also composes — structural compression first, semantic compression on top, if you need both.

---

Known limitations

Accuracy impact is not yet measured. This is the most important caveat in this README. TDF is validated for token reduction and round-trip reversibility. Whether an LLM answers questions as accurately from TDF as from Markdown has not been established.

There is reason for genuine concern. Published work on table serialization consistently finds that redundancy helps comprehension — formats that repeat column names per row let models read each row independently instead of cross-referencing distant headers, and they outperform denser formats despite costing more tokens. TDF's !V codebooks and §n dictionaries increase that cross-reference distance further than any format in those studies. An evaluation is in progress; until it lands, treat compression as the only validated benefit and test on your own workload before deploying.

PEP 8 recall gap. ~0.5% of distinct tokens are dropped on that document, concentrated in code samples. Under investigation.

Small documents can grow. The legend is a fixed ~130-token cost. On a short prose document — bench/samples_tables/prose_only.pdf gains 115 tokens — TDF is a net loss. Use --no-legend, or use Markdown.

Fidelity metrics are order-blind. See above.

No OCR. Scanned PDFs yield nothing. Use MinerU or Marker upstream.

Elision honesty is unmeasured. Whether models correctly request an elided region rather than fabricating an answer has not been quantified. Until it is, --tier should be considered experimental.

---

Roadmap

Not yet shipped. Listed so the gap between claims and code stays visible.

• Hybrid emission — choose per block between Markdown and dense TDF, so output is never larger than Markdown and prose stays in its native form. This turns the range above into a floor guarantee.
• Accuracy harness — token-vs-accuracy Pareto frontier across formats and models, with ablations isolating !V, §n and ^. Mechanisms that cost accuracy will be disabled by default regardless of their compression contribution.
• DoclingDocument reader — accept Docling output as input, inheriting its table detection and OCR.

---

Development

git clone https://github.com/meashumishra/TDF.git
cd TDF
pip install -e ".[bench]"
python bench/make_samples.py        # required — without it 22 tests silently skip
python bench/make_table_pdfs.py
pytest tests/ -v

Expected: 88 passed, 3 skipped.

tiktoken downloads its BPE tables on first use; in a sandboxed environment set TIKTOKEN_CACHE_DIR to a pre-populated cache.

Issues and PRs: https://github.com/meashumishra/TDF

License

MIT