tonst — Token Optimization & Security Tool

tonst is a Python library that sits between your application and a paid LLM API. Before each request leaves your machine it removes personal data and cuts the tokens you pay for; after the response comes back it puts the personal data back.

It runs in your own process. There is no proxy, server or account, and nothing is sent anywhere except the request you were already making.

sequenceDiagram
    participant App as Your app
    participant T as tonst (runs in your process)
    participant API as Your LLM API
    App->>T: prompt or messages (with PII)
    Note over T: 1. Redact PII into placeholders<br/>2. Trim, compact history, filter tools and chunks<br/>3. Order the request for prompt caching
    T->>API: redacted, smaller request
    API-->>T: response (placeholders only)
    Note over T: 4. Put the real values back
    T-->>App: response + savings report

What it does

Feature	What it saves or protects	Where it helps
PII and secret redaction	Emails, phones, addresses, cards, IPs, API keys and (with GLiNER) names, companies and codenames never reach the provider; answers come back with the real values, and the quality cost is measured	Every request
Mechanical trim	Duplicate lines and wasted whitespace	Every request
Tool / MCP definition filtering	Sends only the tool definitions a request needs (−51% to −55% cost in live tests)	Agents and tool-calling apps
Rolling history compaction	Summarizes old turns instead of re-sending or silently dropping them	Long chats and agent loops
RAG context optimization	Drops duplicate and (optionally) irrelevant retrieved chunks	Retrieval pipelines
Prompt-caching structuring	Orders and marks requests so the provider's own cache discounts the repeated part	Stable system prompts and reference docs
Savings log + tonst stats	A local record of what was saved, with no prompt content	Monitoring

It works with Anthropic, OpenAI and Gemini, and with any other provider through a function you supply. Measured results are summarized below and detailed in docs/results.md.

---

Contents

• Install
• Quickstart
• Using tonst in your application
  • 1. Connect your model
  • 2. Pick the features for your workload
  • 3. Recipes
  • 4. Production checklist
• Configuration reference
• Results
• Privacy: what's hidden and how answers hold up
• Limitations
• Documentation
• Whitepaper, license and contributing

---

Install

Requires Python 3.10+.

pip install tonst

For production, pin a version so an update never surprises you (pip install "tonst==0.2.0"). The latest unreleased code is on GitHub: pip install "git+https://github.com/v-nightwolf/tonst.git".

The only required dependency is requests. Two optional pieces add capability:

Optional piece	What it adds	Install
GLiNER	Free-text PII detection (names, companies, codenames) on CPU, ~0.2–1 s per request	pip install "tonst[gliner]" (pulls in torch, transformers, sentencepiece, protobuf)
Ollama	A local model for history summaries or compression, e.g. ollama pull gemma2:2b	Separate app, not a pip package

Neither is needed for the quickstart. If an optional piece is missing, the feature that uses it is skipped rather than breaking the request. If GLiNER is selected but can't load, tonst logs a loud warning that names are not being hidden.

Quickstart

Wrap the function you already use to call your model. tonst takes a function from prompt text to response text:

import anthropic
from tonst import TonstClient

api = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY

def call_model(prompt: str) -> str:
    resp = api.messages.create(
        model="claude-sonnet-4-6", max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    return "".join(b.text for b in resp.content if b.type == "text")

client = TonstClient(call_fn=call_model)

answer, report = client.query(
    "Customer jo@example.com says order 4471 arrived broken.\n"
    "Customer jo@example.com says order 4471 arrived broken.\n"
    "Draft a short, polite reply."
)

print(answer)                     # real email address restored in the reply
print(report.redacted_fields)     # 1 -- the model saw [[EMAIL_…]], not the address
print(report.tokens_saved, report.percent_saved)   # the duplicate line was trimmed
print(report.local_overhead_ms)   # tonst's own time before the API call

Any provider works the same way: call_model is your code, so it can call OpenAI, Gemini, Bedrock, a self-hosted model or an internal gateway.

---

Using tonst in your application

1. Connect your model

tonst never calls a provider on its own. You give it one of two functions:

	call_fn(prompt: str) -> str	messages_fn(messages: list[dict]) -> str
Receives	One flattened, redacted prompt string	The redacted {"role", "content"} list, system messages included
Best for	Single-shot requests: classification, extraction, drafting	Chat apps and anything that needs roles or prompt caching
Can report real usage	No	Yes: return (text, usage)

With messages_fn, use the adapters in tonst.adapters to convert the message list for your provider and to read the provider's usage block back. Returning usage lets tonst report real billed and cached tokens, and lets cache-aware compaction see how well the provider is actually caching.

Anthropic, with prompt-caching breakpoints set for you:

import anthropic
from tonst import TonstClient
from tonst.adapters import to_anthropic, usage_from_anthropic

api = anthropic.Anthropic()

def call_claude(messages):
    resp = api.messages.create(model="claude-sonnet-4-6", max_tokens=1024,
                               **to_anthropic(messages))
    text = "".join(b.text for b in resp.content if b.type == "text")
    return text, usage_from_anthropic(resp)

client = TonstClient(messages_fn=call_claude)

OpenAI, which caches repeated prefixes automatically:

from openai import OpenAI
from tonst.adapters import to_openai, usage_from_openai

oa = OpenAI()

def call_openai(messages):
    resp = oa.chat.completions.create(model="gpt-4o", messages=to_openai(messages))
    return resp.choices[0].message.content, usage_from_openai(resp)

Gemini (REST), where implicit caching is automatic but best-effort:

import os, requests
from tonst.adapters import to_gemini, usage_from_gemini

URL = "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.8-flash:generateContent"

def call_gemini(messages):
    resp = requests.post(URL, headers={"x-goog-api-key": os.environ["GEMINI_API_KEY"]},
                         json=to_gemini(messages), timeout=120).json()
    parts = resp["candidates"][0]["content"]["parts"]
    return "".join(p.get("text", "") for p in parts if not p.get("thought")), usage_from_gemini(resp)

A client built with messages_fn also handles query(), query_structured() and query_rag(): they send it a single user message.

2. Pick the features for your workload

Everything except regex redaction and mechanical trim is off until you turn it on.

Your workload	Turn on	Why
Any app sending user text	redaction_backend="regex" (default); "gliner" if prompts contain names or free-text personal data	Keeps PII off the provider
Chat app or support bot	query_messages() with a RollingSummary per conversation, background_summary=True	Bounded history without losing facts, no added latency
Chat app on a provider with prompt caching	Also compaction_cache_aware=True and pass usage back from messages_fn	Summarizes only when it pays off against the cache
Agent or tool-calling app	select_tools() / ToolSession on your tool list	Tool definitions are often most of the prompt
Huge tool catalog on Anthropic	build_anthropic_deferred_tools()	Uses Anthropic's own tool search
RAG pipeline	query_rag()	Drops duplicate and irrelevant chunks
Large stable system prompt or reference docs	query_structured(), or messages_fn with to_anthropic()	Puts the repeated part first so it caches
You want to see the savings	savings_log=True, then tonst stats	Local log, no prompt content

3. Recipes

Chat app or support bot

Keep one RollingSummary per conversation and pass the full history each turn. tonst sends the recent turns verbatim, folds older turns into a structured summary (goal, decisions, key facts, open items), and carries exact references such as order numbers and case IDs forward word for word.

from tonst import TonstClient, RollingSummary, AnthropicSummarizer

client = TonstClient(
    messages_fn=call_claude,                       # from step 1
    use_history_compaction=True,
    compaction_summarizer=AnthropicSummarizer(),   # Claude Haiku 4.5; omit to use local Ollama
    compaction_cache_aware=True,                   # hold summaries back while the cache is cheaper
    compaction_token_threshold=3000,               # summarize in batches of ~3k tokens
    savings_log=True,
    app_name="support-bot",
)

state = RollingSummary()          # one per conversation
history = [{"role": "system", "content": "You are Acme's support assistant."}]

def handle_turn(user_text: str) -> str:
    history.append({"role": "user", "content": user_text})
    reply, report = client.query_messages(history, keep_last_n=6,
                                          rolling_state=state, background_summary=True)
    history.append({"role": "assistant", "content": reply})
    return reply

To keep a conversation across requests (a web app, a queue worker), store the state next to the conversation:

client.wait_for_background_work(timeout=30)    # let a running summary finish first
db.save(conversation_id, state.to_dict())      # plain JSON: summary text, counters, pinned references
# next request:
state = RollingSummary.from_dict(db.load(conversation_id))

Choosing the summarizer:

Summarizer	Cost	Quality in live tests	Needs
AnthropicSummarizer() (Claude Haiku 4.5)	~1 cent over a 24-turn chat	8/8 facts kept	ANTHROPIC_API_KEY
GeminiSummarizer() (Gemini 3.5 Flash-Lite)	~0.3 cents over a 20-turn chat	8/8 facts kept	GEMINI_API_KEY
Local Ollama model (default, gemma2:2b)	Free	5/8 facts kept; slower	Ollama running

Summarizers only ever receive already-redacted text. If one fails, the turns stay verbatim and tonst retries before falling back to dropping them; report.history_tokens_lost shows anything that was dropped.

Agent or tool-calling app

tonst doesn't run your agent loop, so use the tool filter directly where you build each request. ToolSession only ever adds tools during a conversation, never removes them, so the tool list stays byte-identical between turns and keeps hitting the provider's cache.

from tonst import ToolSession, select_tools

session = ToolSession(ALL_TOOLS, top_k=8)          # one per conversation or agent run

def next_request(messages, user_text):
    sel = session.select(user_text)
    # sel.tools: the definitions to send (original objects, original order)
    # sel.fell_back: True when tonst wasn't confident and kept every tool
    return api.messages.create(model="claude-sonnet-4-6", max_tokens=1024,
                               tools=sel.tools, messages=messages)

# Stateless, single request:
sel = select_tools(ALL_TOOLS, "Create a Jira ticket for the refund bug", top_k=5)

select_tools() accepts Anthropic, OpenAI (Chat Completions and Responses) and Gemini function-declaration shapes as they are. It ranks by keyword relevance, and when a request shares fewer than two words with every tool it sends all of them instead of guessing. For catalogs of hundreds of tools on Anthropic, build_anthropic_deferred_tools(ALL_TOOLS) hands the choice to Anthropic's tool search instead. See docs/tools-and-rag.md.

RAG pipeline

answer, report = client.query_rag(
    question="How long do refunds take?",
    chunks=retrieved_chunks,                 # strings, or dicts with "text"/"content"
    system="Answer only from the provided context.",
    top_k=4,                                 # optional: also drop low-relevance chunks
)
print(report.chunks_in, "→", report.chunks_sent)

Duplicates and near-duplicates are always removed. Relevance filtering happens only if you ask for it (top_k, min_relative_score or max_tokens), and is skipped when no chunk clearly matches the question.

Large stable prompts and prompt caching

Provider caches only discount a repeated prefix. Put the part that never changes first and the per-request part last:

from tonst import PromptParts

parts = PromptParts(
    system="You are a contracts analyst.",
    stable_blocks=[playbook_text, clause_library],   # identical on every call
    variable=user_question,                          # changes every call
)
answer, report = client.query_structured(parts)

Redaction placeholders are deterministic hashes, so a stable block that contains PII redacts to the same bytes every time and still caches. For Anthropic's explicit cache_control markers, either use messages_fn with to_anthropic(), or build the request yourself with redact_and_trim_parts() + build_anthropic_cache_request(). Provider details, minimum cacheable lengths and the generic config for other providers are in docs/caching-and-providers.md.

4. Production checklist

Privacy

• The model provider receives only redacted text. The placeholder → value mapping stays in memory for the duration of the call and is never logged.
• Optional remote helpers (AnthropicSummarizer, GeminiSummarizer, AnthropicTokenCounter, GeminiTokenCounter) also receive only redacted text.
• The savings log stores counts and redaction labels (e.g. EMAIL: 2), never prompt text, values or placeholder hashes.
• regex catches structured PII only. If your prompts contain names or free-text personal details, use redaction_backend="gliner", and test recall on your own data first (docs/redaction.md).

Latency

Step	Typical added time
Regex redaction, trim, tool selection, compaction bookkeeping	a few milliseconds
GLiNER redaction (CPU)	~150 ms – 1.3 s, depending on hardware
Background summary	0 on the request path
Blocking summary with a local 2B model	4–12 s on the turn it runs
Exact token counter	one extra network round trip per count (~300 ms measured on Gemini)

Every report carries per-step timings (redaction_ms, compaction_ms, call_ms, local_overhead_ms, …) so you can check this on your own hardware.

State and concurrency

• RollingSummary and ToolSession are per-conversation state; persist them with to_dict() (ToolSession.load_state() / RollingSummary.from_dict() to restore). They hold only redacted text.
• Each RollingSummary has its own lock. Background summaries run on one worker thread per client; call client.wait_for_background_work() before shutting down or saving state.
• Concurrency has been tested for the GLiNER redaction path (4 workers: same results, higher latency). Sharing one TonstClient across many threads hasn't been load-tested; one client per worker process is the conservative choice.

Failure behaviour

• Every optional step fails soft: if Ollama, GLiNER or a remote summarizer is unavailable, that step is skipped and the request still goes out.
• The one lossy fallback is history compaction: if summaries keep failing, old turns are eventually dropped. report.history_tokens_lost and tonst stats show when that happens.
• The API call itself is your code. tonst doesn't retry it or change its timeouts.

Monitoring

tonst stats                    # all apps, from ~/.tonst/savings.jsonl
tonst stats --app support-bot --since 2026-09-01
tonst stats --json             # for dashboards

Set TONST_SAVINGS_LOG to put the log somewhere else, and input_price_per_million= on the client to see estimated dollars. Token counts are chars/4 estimates unless you pass token_counter= (see docs/measurement.md); messages_fn usage adds the provider's real prompt and cached-token counts to each entry.

---

Configuration reference

The TonstClient options you're most likely to set. The full list, with reasoning for each default, is in the TonstClient.__init__ docstring (tonst/client.py).

Option	Default	Meaning
call_fn / messages_fn	—	How tonst calls your model; pass at least one (step 1)
redaction_backend	"regex"	"none", "regex", "gliner" or "ollama"
use_history_compaction	False	Summarize old turns in query_messages() instead of only dropping them
compaction_summarizer	local Ollama	AnthropicSummarizer(), GeminiSummarizer() or any fn(prompt, model, timeout) -> str
compaction_token_threshold	3000	How many tokens of old turns to batch into one summary
compaction_cache_aware	False	Postpone summaries that wouldn't pay off against the provider's cache
compaction_cache_pricing	"anthropic"	"anthropic", "gemini" or a (write, read) price-multiplier tuple
use_local_compression	False	Rewrite prompts shorter with a local Ollama model (single-shot requests only)
local_model	"gemma2:2b"	Ollama model for local steps
savings_log	None	True, a file path, or a SavingsLog
app_name, input_price_per_million	None	Labels and pricing for the savings log
token_counter	None	AnthropicTokenCounter(...) / GeminiTokenCounter(...) for exact counts
placeholder_style	"hash"	"hash" (stable, keeps prompt caching working) or "readable" ([[NAME_1]])
placeholder_hint	True	Tell the model what the placeholders are (strongly recommended for Claude)
email_style	"split"	[[EMAIL_1]]@[[DOMAIN_1]]; "whole" for one token per address
extra_redaction	()	Add "ACCOUNT_ID" and/or "MONEY" (money breaks arithmetic tasks)
restore_secrets	False	Put withheld keys back into answers (off: shown as [REDACTED])
secret_notice	False	Append a one-line "secrets were withheld" note to the answer

Method	Use it for
query(prompt)	One flat prompt
query_messages(messages, keep_last_n=6, rolling_state=None, background_summary=False)	Chat history
query_structured(PromptParts(...))	Stable prefix + variable question
query_rag(question, chunks, ...)	Retrieved context

Each returns (response_text, OptimizationReport). Useful report fields: original_tokens, sent_tokens, tokens_saved, percent_saved, redacted_fields, redacted_types, secrets_withheld, history_*, chunks_in / chunks_sent, provider_prompt_tokens / provider_cached_tokens, and the *_ms timings.

---

Results

Headline numbers. Each one says how it was measured; the full tables, methods and every intermediate run are in docs/results.md.

What	Result	How it was measured
Tool filtering, 36 tools, 30 tasks	Cost −51% (Claude Sonnet 4.6) and −55% (Gemini 3.8 Flash), with the same success rate as sending every tool	Live API
Rolling compaction, 20-turn chat on Gemini 3.8 Flash	Cost −15.1%, prompt tokens −43%, 8/8 facts kept, p95 latency 4.0 s → 2.7 s	Live API
Rolling compaction, 24-turn chat on Claude Sonnet 4.6	Cost −2.7% with Haiku summaries (8/8 facts); caching already makes old turns cheap there	Live API
Cache-aware compaction, short chats	Avoids a +14.9% loss that summarizing too early caused in a 12-turn Claude chat	Live API
Prompt caching, Gemini 3.1 Flash-Lite, 6 domains	Net cost −53% across 24 calls	Live API
Redaction + trim + compression, 360 prompts in 6 domains	Tokens −21.1%; 100% structured-PII recall with 0 leaks; 87.8% free-text PII recall with GLiNER	Local pipeline (API mocked)
Mechanical trim on long, redundant inputs (privacy benchmark's heavy cases)	Input tokens −33% to −51% on re-quoted email threads and −34% to −51% on padded meeting transcripts, with redaction on; roughly break-even on log dumps and small RAG sets	Live API
Answer quality with redaction on, 100 work prompts	−0.08 (Claude Sonnet 4.6) and −0.38 (Gemini 3.8 Flash) on a 1–10 judge vs. unredacted; 100% of must-have values kept; 0 names, emails, phones, addresses or secrets reached the provider	Live API, blind LLM judge
Compaction on long chats with caching	−18% at 40 turns, −55% at 100 turns	Simulation, calibrated to the live runs

Savings depend on your traffic. A short, clean prompt gets 0% from trimming, and tonst reports 0% rather than inventing a saving.

---

Privacy: what's hidden and how answers hold up

Redaction is only useful if the answers stay good. tonst measures that with a published benchmark, and hides more than the obvious fields.

Does hiding the data make answers worse?

A little, and we measured how much. The benchmark in experiments/privacy_quality/ sends 120 realistic work prompts — support replies, contracts, HR notes, invoices, config files with keys, meeting notes, small tables, translations, long email threads, and 10 hold-out prompts written after the detectors were tuned — to Claude Sonnet 4.6 and Gemini 3.8 Flash, once as-is and once through tonst. A separate model grades each pair blind, in random order (judge: Gemini 3.8 Flash). All people, companies and keys in the prompts are invented. Results from the release run of tonst 0.2.0 (2026-09-28), default settings plus GLiNER:

	Claude Sonnet 4.6	Gemini 3.8 Flash
Answer quality, masked vs. as-is (1–10 judge, 100 main prompts)	−0.08 (9.2 → 9.1)	−0.38 (9.7 → 9.3)
Same, with placeholder_style="readable"	−0.47	−0.31
Hold-out prompts (10, written after tuning)	+0.3	−0.4
Answers containing every must-have value (names, totals, IDs)	100%	100% (94% on hold-outs)
Placeholders left in answers	0	0

The full tables, per-category results and how to rerun it are in experiments/privacy_quality/.

Most of the remaining gap has a known cause: the model can't write a name it never saw in another script (e.g. Hindi), can't tell someone's gender from a placeholder, and occasionally mixes up whose contact details are whose. Hiding money amounts breaks arithmetic, which is why that category is off by default.

What gets hidden

Category	Examples	How	Default
Emails	priya.nair@veltrix.io	pattern	on
Phone numbers	+91 98450 21733, (415) 555-0144, 090000 12345	pattern	on
Postal addresses	1180 Folsom Street, San Francisco, CA 94103, Flat 9B, …, Gurugram 122003	pattern	on
Secrets	Anthropic/OpenAI/AWS/GitHub/Slack/Google keys, JWTs, private keys, password=…	pattern	on
Cards, SSN-like IDs, IP addresses	4111 1111 1111 1111, 10.24.8.117	pattern	on
Names, companies, project codenames	Omar Haddad, Veltrix Logistics, Project Bluefin	GLiNER, a small local model, plus name rules	with redaction_backend="gliner"
Account / invoice / order numbers	INV-22243, customer #928381	pattern	opt-in: extra_redaction=["ACCOUNT_ID"]
Money amounts	$14,821.32, ₹12 lakh	pattern	opt-in: extra_redaction=["MONEY"]

In the benchmark (default settings plus GLiNER), no names, emails, addresses, secrets, codenames or IP addresses reached either provider; the only miss was one company mentioned as "the Brightwell Health clinic", in 2 of 480 masked prompts. Detection is never perfect: test on your own data before relying on it. Public services such as api.anthropic.com are deliberately left visible — hiding them from the provider protects nothing.

How the placeholders work

• Keyed, stable placeholders (default). [[EMAIL_8ddc0a70]] is an HMAC of the value with a key that never leaves your machine (~/.tonst/placeholder.key, or TONST_PLACEHOLDER_KEY). The same value gets the same placeholder on every call, so provider prompt caching keeps working, and a provider can't confirm a guessed value by hashing it. placeholder_style="readable" gives [[EMAIL_1]]-style numbering instead.
• One placeholder per person. "Omar Haddad" → [[NAME_1]], a later "Omar" → [[NAME_1.first]], "Dr. Haddad" → Dr. [[NAME_1.last]], so the model can write "Hi [[NAME_1.first]]" and the reply says "Hi Omar".
• Emails keep their shape. [[EMAIL_1]]@[[DOMAIN_1]]: addresses at the same company share a domain placeholder, so "group these by company" still works.
• A short note to the model (on by default, only when something was hidden) says the tokens stand for real values. Extra lines are added only when they apply: first/last-name parts when a person was hidden, whose contact details are whose, and that [[SECRET_…]] tokens are exposed credentials. About 45 tokens, up to ~120 with every line; in chats the fixed part sits in the system message, where prompt caching makes repeat reads ~90% cheaper. Without the note, Claude treated placeholders as template blanks in ~40% of answers.
• Secrets never come back. Keys are withheld from the provider and shown as [REDACTED] in answers; report.secrets_withheld counts them.
• Tolerant restore. Placeholders the model reformats (NAME_1, [NAME_1]) are still restored; ones it invents become neutral blanks ([email]) instead of raw tokens. StreamRestorer does the same for streamed answers.

---

Limitations

• Token counts are estimates by default (characters ÷ 4). That was within 2% of Gemini's real counts but 1.77× too low for Claude requests with tools. Pass token_counter= when numbers matter.
• Detection isn't perfect. Pattern rules miss unusual formats, and GLiNER misses some names and companies (e.g. a company only mentioned as "the X clinic"). Anything missed is sent as-is. Test on your own data.
• Some answers get a little worse with redaction on. The model can't transliterate a hidden name, infer gender, or calculate with hidden money amounts (which is why hiding money is opt-in); see the benchmark.
• Privacy costs a few tokens. When something was hidden, the note to the model (~45–120 tokens) and the longer stable placeholders add input tokens. On short prompts that can outweigh trimming; placeholder_hint=False turns the note off (not recommended for Claude).
• The mapping lives in memory. Keep the request's result until the answer arrives; an answer handled after a restart can't be restored.
• Compaction is lossy by design. Summaries keep facts well with an API summarizer, less well with a local 2B model, and pinned references keep identifiers exact. Anything dropped is reported, not hidden.
• Tool selection is lexical. Paraphrased requests often fall back to sending every tool: safe, but no saving.
• Provider coverage: Anthropic and Gemini have been tested live; the OpenAI module is built from OpenAI's documentation but hasn't been run against a real key yet.
• Not included yet: async clients (streamed answers can be restored with StreamRestorer). The API call is always your own synchronous function.

Documentation

Document	Contents
docs/results.md	Every benchmark and live run, with methods
experiments/privacy_quality/	The answer-quality benchmark for redaction: cases, runner, how to reproduce
docs/redaction.md	Detectors, backends, placeholders, names, emails, secrets, streaming
docs/compaction.md	Stateless and rolling compaction, summaries, cache-aware mode, background summaries
docs/tools-and-rag.md	Tool/MCP definition filtering, deferred loading, RAG chunk optimization
docs/caching-and-providers.md	Prompt-caching structuring, per-provider details, other providers, live cache tests
docs/measurement.md	Exact token counting and the savings log
docs/testing.md	Unit tests, demos, benchmarks, live tests, repository layout
ROADMAP.md	Decisions, open questions and the full history of results

Some documents cite research notes under research/; those notes aren't published in this repository.

Repository layout

tonst/              the library
test_*.py           unit tests (CI runs all of them)
experiments/        answer-quality benchmark for redaction (privacy_quality/)
examples/           runnable demos: demo.py (no API key needed), real_api_demo.py, cache_savings_demo_*.py per provider
benchmarks/         offline benchmark and live API tests (benchmark_savings_features.py, live_test_*.py, benchmark_tonst.py)
scripts/research/   one-off diagnostic and tuning scripts behind the findings in docs/ (need GLiNER or Ollama)
docs/               detailed documentation

Run scripts from the repository root, e.g. python3 examples/demo.py. API keys go in a .env file in the repository root (gitignored).

Whitepaper, license and contributing

• Whitepaper: Beyond the Prompt · DOI 10.5281/zenodo.22745266
• License: MIT (see LICENSE).
• Tests: pip install -r requirements-dev.txt && pytest (319 tests, run on every push for Python 3.10–3.13). Live API tests and benchmarks are described in docs/testing.md.
• Issues and pull requests are welcome.