tokensaver-ai 1.3.0

Compile repositories into minimal agent context with exact token compression metrics.

TokenSaver

Stop making coding agents reread your whole repo.

TokenSaver compiles your repository into a compact context bundle that agents can use before they touch raw source. Instead of burning tokens re-parsing files on every task, agents start with the structure that actually matters: modules, APIs, routes, config, commands, and exact compression metrics.

That means faster repo understanding, less wasted context, and more consistent answers across large codebases.

Used across 11 anonymized real-world repositories with up to 77.15x measured compression.

Works with Cursor, Claude Code, Codex, Windsurf, and MCP-enabled workflows.

---

Who It’s For

TokenSaver is for teams using coding agents on repositories that are too large, too modular, or too repetitive to keep re-reading from scratch.

It helps most when you have:

• large monorepos or multi-module apps
• mobile + web + backend repositories
• repeated agent sessions in the same codebase
• architecture, API, navigation, or config questions
• agents that keep spending context on repo discovery before doing useful work

When TokenSaver Helps Most

TokenSaver is most useful when the hard part is understanding the repo, not writing one isolated function.

Typical cases:

• onboarding to an unfamiliar codebase
• answering "where does this live?" or "how is this wired?"
• tracing APIs, routes, modules, and config
• keeping agent context stable across multiple tasks
• reducing repeated raw-source reads in long sessions

Why Not Just Let the Agent Read the Repo?

You can. But on larger repositories, that gets expensive and noisy fast.

Without TokenSaver:

• agents repeatedly crawl raw source to rediscover structure
• context gets wasted on files that are not immediately relevant
• answers can vary across sessions depending on what was read first
• large codebases take longer to orient around before real work begins

With TokenSaver:

• agents start with a compact map of the repo
• architecture, API, route, and config questions are easier to answer from artifacts first
• raw source is read only when implementation detail is actually needed
• token usage becomes measurable instead of guesswork

Why TokenSaver?

TokenSaver helps agents start with structure instead of raw source:

Without TokenSaver	With TokenSaver
Agent reads 800K tokens of raw source	Agent reads ~15K tokens of structured context
Every prompt re-parses the entire codebase	Artifacts are pre-built, cached, and incremental
No awareness of project structure	Module graph, API index, route map, config keys
Context window fills up fast	Exact per-artifact and repo-level compression metrics

Real-world proof: up to 77.15x measured compression across 11 anonymized real-world repositories and 15/15 public fixtures passing with 100% framework detection.

Published real-world benchmarks (anonymized, exact tiktoken counts):

Repo	Framework	Plugin	Files	Source Tokens	Bundle Tokens	Compression
Confidential Flutter App A	flutter	flutter	10,264	8,510,170	157,867	53.91x
Confidential Flutter App B	flutter	flutter	105	222,337	2,882	77.15x
Confidential React Native App A	react_native	react_native	896	804,476	28,466	28.26x
Confidential React Native App B	react_native	react_native	570	71,748	5,063	14.17x
Confidential Node Backend A	node	generic	32	19,998	5,498	3.64x
Confidential Next.js App A	nextjs	nextjs	59	18,747	1,999	9.38x
Confidential Android App A	android_native	android_native	2,255	4,427,242	2,180,896	2.03x
Confidential iOS App A	ios_swift	ios_swift	2,449	5,903,230	84,544	69.82x
Confidential React Web App A	react	react_web	27	16,390	2,332	7.03x
Confidential Angular App A	angular	angular	456	254,702	23,379	10.89x
Confidential Angular App B	angular	angular	140	101,390	12,850	7.89x

These are exact measured results from 11 anonymized real-world repositories across 8 frameworks. Compression ratios vary by project structure and framework. Public fixture benchmarks are used for contract validation and support coverage, not headline compression claims. See Published Results for per-artifact breakdowns.

---

Quick Start

Standard install

python3 -m pip install --user tokensaver-ai

If tokensaver is not found after install, add your user Python bin directory to PATH:

export PATH="$(python3 -m site --user-base)/bin:$PATH"

Install with MCP support

python3 -m pip install --user "tokensaver-ai[mcp]"

With the MCP extra installed, tokensaver init . also writes Cursor and Claude MCP configs automatically.

Add to any repo

cd /path/to/your-repo
tokensaver init .

This one command:

• builds the TokenSaver bundle
• writes Cursor, Claude Code, Codex, and Windsurf rule files
• writes Cursor and Claude MCP configs when MCP support is installed
• prints the repo’s compression summary

How Agents Use It

Instead of opening raw source first, agents can start from the generated bundle:

• PROJECT_SUMMARY.json for framework, entrypoints, and project shape
• COMMANDS.json for run, build, test, and lint commands
• MODULE_GRAPH.json for structure and dependencies
• API_INDEX.json for endpoints and services
• ROUTE_INDEX.json for navigation and pages
• CONFIG_INDEX.json for env/config keys
• METRICS.json for exact compression numbers

That means agents begin with structure first, then drill into source only when necessary.

Before and After

Without TokenSaver:

• agent opens many files just to answer "how does this repo work?"

With TokenSaver:

• agent reads the bundle first
• finds the relevant module, route, API, or config quickly
• opens only the few source files needed for implementation detail

Scan

tokensaver scan /path/to/repo

Outputs exact file counts, token counts, detected framework, and language breakdown.

Build

tokensaver build /path/to/repo

Generates 7 artifacts in docs/tokensaver/ and auto-installs agent integrations for Cursor, Claude Code, Codex, and Windsurf.

Incremental Rebuild

tokensaver build /path/to/repo
# Only changed artifacts are regenerated (SHA-256 diffing)

tokensaver build /path/to/repo --force
# Force full rebuild

Impact Analysis

tokensaver impact /path/to/repo
# Shows blast-radius: which modules, APIs, routes are affected by recent changes

tokensaver impact /path/to/repo --files src/auth/login.py,src/models/user.py

MCP Server

python3 -m pip install --user "tokensaver-ai[mcp]"
cd /path/to/repo
tokensaver init .
tokensaver serve /path/to/repo
# Starts the TokenSaver MCP server for interactive querying

---

Supported Frameworks (17 stacks, 12 plugins)

First-Class Plugins (deep extraction)

Framework	Plugin	Detection	What's Extracted
Flutter	flutter	pubspec.yaml	GetX routes, Dart API URLs, RemoteConfig keys, module graph
React Native	react_native	package.json → react-native	Stack.Screen navigation, Axios/fetch APIs, RN Config, module graph
Next.js	nextjs	package.json → next	App Router + Pages Router, API routes, server actions, next.config, NEXT_PUBLIC_* env
Workspace (multi-app)	workspace	multiple nested app roots such as frontend/ + backend/	Merged artifacts from child apps, including nested routes, APIs, config, module graph, and commands
Angular	angular	package.json → @angular/core	RouterModule routes, HttpClient APIs, environment.ts config, @Component/@Injectable
React (web)	react_web	package.json → react	React Router v5+v6, fetch/axios APIs, REACT_APP_* env, lazy imports
FastAPI	python_web	deps → fastapi	Decorator routes, Pydantic models, middleware, env config
Django	python_web	manage.py or deps → django	URL patterns, ORM models, middleware, settings.py keys
Flask	python_web	deps → flask	Route decorators, models, middleware, env config
Spring Boot	spring_boot	build.gradle/pom.xml → spring-boot	@GetMapping/@PostMapping, @Entity models, JPA repos, application.properties/.yml
Android Native	android_native	build.gradle (non-Spring)	Activities, Fragments, Jetpack Compose composable() routes, Retrofit APIs, BuildConfig, strings.xml
iOS (Swift)	ios_swift	.xcodeproj / Package.swift	SwiftUI Views, NavigationLink, UIKit ViewControllers, URLSession/Alamofire APIs, UserDefaults, @AppStorage, Info.plist
Go	go	go.mod	net/http, Gin, Chi, Echo, Fiber routes, structs, os.Getenv, Viper config, go.mod deps

Generic Fallback (all other projects)

Framework	Detection	What's Extracted
Node.js / Express	package.json	Express routes, mount paths, process.env, module graph
Python (generic)	*.py files	Decorator routes, env config, module graph
PHP	composer.json or *.php files	Generic PHP routes, env config, composer scripts, module graph
Rust	Cargo.toml	Module graph, framework detection

---

Benchmark Results (15 fixture suite)

All 15 fixtures pass with ok status, 100% framework detection accuracy:

Fixture	Framework	Status	Plugin
Flutter	flutter	ok	flutter
React Native	react_native	ok	react_native
Next.js	nextjs	ok	nextjs
Angular	angular	ok	angular
React Web	react	ok	react_web
FastAPI	fastapi	ok	python_web
Django	django	ok	python_web
Spring Boot	spring_boot	ok	spring_boot
Android Native	android_native	ok	android_native
iOS Swift	ios_swift	ok	ios_swift
Go	go	ok	go
Node.js	node	ok	generic
Python	python	ok	generic
PHP	php	ok	generic
Workspace	workspace	ok	workspace

---

Build Outputs

tokensaver build generates these artifacts:

Artifact	Contents
PROJECT_SUMMARY.json	Framework, languages, entrypoints, manifests
COMMANDS.json	Build/dev/test/lint commands from package.json, Makefile, CI
MODULE_GRAPH.json	Module names, file counts, token counts
API_INDEX.json	API endpoints with methods, paths, source locations
ROUTE_INDEX.json	UI routes / URL patterns with navigation graph
CONFIG_INDEX.json	Environment variables, settings keys, config references
METRICS.json	Exact compression metrics per artifact and overall

Every artifact carries schema_version in _meta. See Output Schema.

---

Agent Integrations

tokensaver build automatically installs integration files so agents pick up context without manual referencing:

Agent	File Generated	How It Works
Cursor	.cursor/rules/tokensaver.mdc	Auto-injected as context rule
Claude Code	CLAUDE.md	Read by Claude on project open
Codex	AGENTS.md	Read by Codex on project open
Windsurf	.windsurfrules	Read by Windsurf on project open
Cursor MCP	.cursor/mcp.json	Interactive MCP querying
Claude MCP	.mcp.json	Interactive MCP querying

---

Architecture

tokensaver/
  core/             # Scan, build orchestration, plugin protocol, shared models
    plugin_api.py    # TokenSaverPlugin protocol
    registry.py      # Plugin registry (ordered matching)
    helpers.py       # Shared regex patterns, utilities
    models.py        # ArtifactResult, BuildContext
  plugins/           # Framework-specific extractors
    flutter.py       # Flutter (GetX, Dart APIs, RemoteConfig)
    react_native.py  # React Native (Stack navigation, Axios)
    nextjs.py        # Next.js (App Router, API routes, server actions)
    angular.py       # Angular (RouterModule, HttpClient, environment.ts)
    react_web.py     # React web (React Router, fetch/axios, REACT_APP_*)
    python_web.py    # FastAPI / Django / Flask
    spring_boot.py   # Spring Boot (annotations, JPA, properties)
    android_native.py # Android (Compose, Retrofit, BuildConfig, strings.xml)
    ios_swift.py     # iOS (SwiftUI, UIKit, URLSession, UserDefaults)
    go_mod.py        # Go (net/http, Gin, Chi, Echo, Fiber)
    generic.py       # Fallback for all other stacks
  scanner.py         # Framework detection + token accounting
  build.py           # Build orchestration + incremental diffing
  snapshot.py        # SHA-256 snapshot for incremental builds
  impact.py          # Blast-radius change-impact analysis
  integrations.py    # IDE/agent config file generation
  mcp_server.py      # MCP server (FastMCP)
  benchmark.py       # Reproducible benchmarking + suite runner
tokensaver_cli.py    # CLI entry point

---

CLI Reference

tokensaver scan <path>                                    # Scan and report token counts
tokensaver init <path> [--output-dir <dir>] [--force]     # One-shot repo onboarding
tokensaver build <path> [--output-dir <dir>] [--force]    # Build artifacts
tokensaver impact <path> [--files f1,f2]                  # Blast-radius analysis
tokensaver serve <path>                                   # Start MCP server
tokensaver metrics <path>                                 # Print existing metrics
tokensaver benchmark <path>                               # Run reproducible benchmark
tokensaver benchmark-suite <manifest>                     # Run multi-repo benchmark suite
tokensaver diff-snapshots <old> <new>                     # Compare benchmark snapshots

---

Adding a New Plugin

TokenSaver's plugin system is designed for easy extension. Each plugin is ~200-300 lines:

1. Create tokensaver/plugins/your_framework.py
2. Implement the TokenSaverPlugin protocol: name, frameworks, build_artifacts(ctx)
3. Return 4 artifacts: module_graph, api_index, route_index, config_index
4. Register in tokensaver/core/registry.py
5. Add framework detection in tokensaver/scanner.py

@dataclass(frozen=True)
class YourPlugin:
    name: str = "your_framework"
    frameworks: set[str] = frozenset({"your_framework"})

    def build_artifacts(self, ctx: BuildContext) -> list[ArtifactResult]:
        return [
            build_module_graph(ctx),
            build_api_index(ctx),
            build_route_index(ctx),
            build_config_index(ctx),
        ]

---

Metric Semantics

TokenSaver reports only measurable compression metrics — no estimates, no dollar claims:

Metric	Definition
source_tokens	Exact tiktoken (o200k_base) count for source files
output_tokens	Exact count for generated artifact
compression_ratio	source_tokens / output_tokens
union_source_tokens	Deduplicated token count across all source files
bundle_tokens	Total tokens across all artifacts

---

Benchmarking

tokensaver benchmark .                    # Single repo
tokensaver benchmark-suite manifest.json  # Multi-repo suite
tokensaver benchmark-suite manifest.json --public-only  # Safe for publishing
tokensaver diff-snapshots old.json new.json  # Regression detection

See Benchmark Guide for manifest format.

---

Documentation

• Output Schema — canonical artifact shapes
• Compatibility Policy — versioning and backward compatibility
• Known Limitations — current boundaries
• Benchmark Guide — manifest format, privacy workflow
• Contributing
• Changelog

---

License

MIT — commercial use allowed.