packlm 1.0.0

Token-efficient structured data format for LLMs

PackLM ·

Token-efficient structured data format for LLMs.
Define fields once, list values — 60–80% fewer tokens than JSON.

---

The problem with JSON

Every time you send a list of objects to an LLM, JSON repeats every key name on every row:

[
  {"student_name": "Alice Smith", "age": 20, "grade": "A",  "city": "Delhi"},
  {"student_name": "Bob Kumar",   "age": 19, "grade": "B",  "city": "Mumbai"},
  {"student_name": "Charlie Roy", "age": 21, "grade": "A+", "city": "Delhi"}
]

~60 tokens — most of it structural noise ("student_name": repeated 3× = 12 tokens wasted on one key alone).

The PackLM way

@R student_name age grade city

R "Alice Smith"  20  A   Delhi
R "Bob Kumar"    19  B   Mumbai
R "Charlie Roy"  21  A+  Delhi

~22 tokens — schema declared once, values only per row.

	JSON	PackLM	Saving
3 rows, 4 fields	~60 tokens	~22 tokens	~63%
100 rows, 4 fields	~1,800 tokens	~505 tokens	~72%
500 rows, 4 fields	~9,000 tokens	~2,505 tokens	~72%

Saving grows linearly with row count — the schema cost is paid once.

---

Installation

pip install packlm

Or just copy packlm.py into your project — zero dependencies, standard library only.

---

Quick start

from packlm import PackLM

students = [
    {"name": "Alice",   "age": 20, "grade": "A",  "city": "Delhi"},
    {"name": "Bob",     "age": 19, "grade": "B",  "city": "Mumbai"},
    {"name": "Charlie", "age": 21, "grade": "A+", "city": "Dehradun"},
]

# Encode
packed = PackLM.encode(students)
print(packed)
# @R name age grade city
# R Alice 20 A Delhi
# R Bob 19 B Mumbai
# R Charlie 21 A+ Dehradun

# Decode back to Python objects
restored = PackLM.decode(packed)
# [{"name": "Alice", "age": "20", "grade": "A", "city": "Delhi"}, ...]

---

Using with LLMs

Add one line to your system prompt and send PackLM instead of JSON:

import anthropic
from packlm import PackLM, SYSTEM_PROMPT

client = anthropic.Anthropic()

packed = PackLM.encode(my_data)

response = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=1024,
    system=SYSTEM_PROMPT,   # tells the LLM how to read PackLM
    messages=[{"role": "user", "content": f"Analyse this data:\n\n{packed}"}]
)

System prompt to add to any LLM call:

You receive data in PackLM format.
Lines starting with @ define the schema: @ALIAS field1 field2 ...
Other lines are data rows: ALIAS val1 val2 ... — values match schema order.
~ means null. Multi-word values are double-quoted.

---

Format at a glance

Syntax	Description	Example
@ALIAS f1 f2 f3	Schema definition — declare once	@ST name age grade
ALIAS v1 v2 v3	Data row — values in schema order	ST Alice 20 A
~	Null / missing value	ST Bob ~ B
""	Explicitly empty string	ST Charlie "" A
"value with spaces"	Multi-word string	ST "Alice Smith" 20 A
# comment	Line comment, ignored by parser	# student records
(blank line)	Ignored, use for readability

---

CLI usage

# Encode a JSON file to PackLM
python packlm.py encode data.json

# Decode a PackLM file back to JSON
python packlm.py decode data.packlm

---

Token savings estimate

import json
from packlm import PackLM

original = json.dumps(my_data)
packed   = PackLM.encode(my_data)

stats = PackLM.token_savings_estimate(original, packed)
print(stats)
# {"json_tokens": 860, "packlm_tokens": 210, "tokens_saved": 650, "percent_saved": 75.6}

---

Why this matters at scale

At $15 / 1M input tokens (e.g. Claude Opus):

Monthly data volume	JSON cost	PackLM cost	Saving
10M tokens	$150	~$37	$113 / month
100M tokens	$1,500	~$375	$1,125 / month

---

Specification

See SPEC.md for the full grammar, edge cases, and rules for implementing PackLM in other languages.

---

Contributing

PRs welcome! See issues for ideas.

Things that would be great:

• JavaScript / TypeScript implementation
• Rust implementation
• Benchmarks against real tokenizers (tiktoken, sentencepiece)
• Streaming decoder

---

License

MIT — see LICENSE.