savetoken 0.1.1

Convert Repomix/repominify output into compact AI-friendly codebase summaries using free LLMs.

savetoken 🪙

Transform your Repomix codebase dump into a high-fidelity AI context brief — ~83% fewer tokens than sending the raw zip, ~88% fidelity to the original.

Instead of pasting hundreds of thousands of tokens of source code into an AI, savetoken produces a structured codebrief.md that the AI can reason about just as effectively, at a fraction of the cost.

---

How it fits in your workflow

Your project
    ↓
npx repomix                          # pack codebase → repomix-output.xml
    ↓
savetoken summarize repomix-output.xml   # analyse → codebrief.md
    ↓
Paste codebrief.md into your AI prompt   # 17% of tokens, ~88% fidelity

---

Install

pip install savetoken

Requires Python 3.10+. Zero runtime dependencies — stdlib only (ast, urllib, json, hashlib, re).

---

Quick start

# 1. Pack your codebase with Repomix
npx repomix

# 2. Generate context brief (Gemini Flash is free and recommended)
export GEMINI_API_KEY=your_key_here
savetoken summarize repomix-output.xml

# Output: codebrief.md — paste into any AI prompt

Python API

from savetoken import SaveToken

st = SaveToken(provider="gemini", api_key="...")
summary = st.summarize("repomix-output.xml")
st.save(summary, "codebrief.md")

---

What's in the output

savetoken extracts seven categories of information, each via the most accurate method available:

Section	Source	Fidelity
Strategic files (schemas, configs, entry points, tests)	verbatim	100%
Hot functions (most-called by PageRank)	verbatim	100%
File docs (module/class/fn docstrings, signatures, constants)	AST	~95%
Execution flows (entry → service → repo → db)	call graph trace	~90%
Call graph (cross-file function calls, prod/test split)	AST	~90%
Environment contract (all env vars, defaults, required)	AST	100%
SQL schema (tables, columns, FKs, indexes)	AST	~95%
Critical blocks (HACK/FIXME with ±10 lines context)	verbatim	100%

No LLM is called for Python files. Docstrings and signatures are extracted directly from source — zero hallucination, instant, cacheable. LLM calls are only used for: project overview, flow descriptions, and non-Python file summaries.

---

Providers

Provider	Model	Free tier	Env var
gemini ⭐	Gemini 2.0 Flash	✅ Generous, 1M ctx	GEMINI_API_KEY
groq	Llama 3.3 70B	✅ Fast	GROQ_API_KEY
mistral	Mistral Small Latest	✅ Available	MISTRAL_API_KEY
openai	gpt-4o-mini	Paid	OPENAI_API_KEY
deepseek	deepseek-chat	Very cheap	DEEPSEEK_API_KEY

Gemini is recommended — 1M token context handles any codebase size.

---

CLI reference

savetoken summarize repomix-output.xml
savetoken summarize repomix-output.xml \
  --provider groq \
  --output codebrief.md \
  --format markdown \
  --lang pt \
  --target-ratio 0.50 \
  --flow-depth 5 \
  --force        # ignore cache, regenerate everything
savetoken cache-stats
savetoken clear-cache

Flag	Default	Description
--provider	gemini	LLM provider
--output	codebrief.md	Output path
--format	markdown	markdown or json
--lang	en	Language for generated descriptions
--target-ratio	0.50	Max output/input token ratio
--flow-depth	5	Max hops in execution flow traces
--force	off	Bypass cache
--cache-dir	.savetoken_cache	Cache directory
--quiet	off	Suppress per-file progress

---

Caching

Every file is hashed by content. On re-runs, only changed files are re-processed. The cache lives in .savetoken_cache/ — commit it to share across your team. Cache keys are schema-versioned: any model change automatically invalidates stale entries.

savetoken cache-stats   # entries, size, schema version
savetoken clear-cache

---

Real-world numbers (datalock — 59 files, 271k tokens)

Raw codebase (zip):  271,252 tokens   100%   fidelity 100%
savetoken output:     47,204 tokens    17%   fidelity  ~88%
Savings:                              83%

Token breakdown:

• Strategic verbatim (11 files): 26%
• File Docs / AST (48 files): 44%
• Call graph (1,139 edges): 25%
• Flows, env, overhead: 5%

---

VS Code extension

Located in vscode-extension/. Commands:

• SaveToken: Summarize Codebase — runs on repomix-output.xml
• SaveToken: Open Summary
• SaveToken: Clear Cache

Settings: savetoken.provider, savetoken.apiKey, savetoken.language, savetoken.outputFile, savetoken.autoSummarizeOnRepomix.

---

Project structure

savetoken/
├── savetoken/
│   ├── core.py                 Orchestrator — full pipeline
│   ├── models.py               All dataclasses (FileSummary, Flow, etc.)
│   ├── classifier.py           STRATEGIC / AST / REGULAR / SKIP per file
│   ├── budget.py               Token budget allocation (post-classification)
│   ├── analyzer.py             AST call graph + inheritance (prod/test split)
│   ├── ast_types.py            Function signature extraction via AST
│   ├── docstring_extractor.py  Module/class/fn docstrings + constants (NEW)
│   ├── ranker.py               PageRank centrality → hot functions
│   ├── flows.py                Execution flow tracer (BFS on call graph)
│   ├── env_extractor.py        Environment variable extractor
│   ├── schema_sql.py           SQL schema (Django/SQLAlchemy/Alembic/raw SQL)
│   ├── critical_context.py     HACK/FIXME blocks with ±10 line context
│   ├── deduplicator.py         Remove cross-section redundancy
│   ├── output.py               Markdown + JSON renderers
│   ├── cache.py                File-based cache with schema versioning
│   ├── cli.py                  CLI entry point
│   ├── exceptions.py           SaveTokenError hierarchy
│   └── providers/
│       ├── base.py             Abstract provider + prompts
│       ├── gemini.py           Gemini 2.0 Flash
│       ├── groq.py             Groq / Llama 3.3 70B
│       ├── mistral.py          Mistral Small
│       └── openai.py           OpenAI-compatible (OpenAI, DeepSeek, Ollama)
├── vscode-extension/           VS Code extension
├── tests/                      117 unit + integration tests
├── examples/                   Usage examples
└── pyproject.toml

---

License

MIT