Skip to main content

Token Optimizer Plugin

Version: 1.0.0
Author: Aveek Sengupta (aveekforyou@gmail.com)
License: public

A local-first, prompt-only, end-to-end token-optimization toolkit for VS Code Copilot. It attacks both halves of a turn's token cost:

  • Input side — compress large workspace context fed INTO the agent into a reversible, traceable context pack (temp/context_pack_vN/), so the agent reasons over compressed context with full source traceability.
  • Output side — rewrite the agent's own replies coming OUT into lean, high-signal language ("Shrink the mouth, not the brain").

One inviolable invariant spans both sides: code, commands, file paths / URLs / identifiers, error & log lines, and numbers / versions stay byte-for-byte exact. Everything runs as prompt instructions + deterministic stdlib Python helpers — no network calls, no telemetry, no external LLM APIs. The agent is the LLM.


Components

Three Skills + Three Agents

Component Kind Side Role
token-optimizer skill input+output Combined entrypoint; activates both sides; delegates all real work to the two sibling skills, adds no new compression logic. Supports one-shot and session mode.
input-token-optimizer skill input Builds the context pack (5 artifacts) under temp/context_pack_vN/.
output-token-optimizer skill output Lean output-style rewriter + companion actions (speak, lean-commit, lean-review, lean-compress-file).
token-optimizer-agent agent input+output Thin wrapper → token-optimizer skill → combined YAML verdict.
input-token-optimizer-agent agent input Thin wrapper → input-token-optimizer skill → input YAML verdict.
output-token-optimizer-agent agent output Thin wrapper → output-token-optimizer skill → output YAML verdict.

Input-Side Compression Modes & Savings Bands

Mode Behaviour Savings Band
quick Aggressive compression, precision over coverage 60–80%
balanced (default) Blend of coverage and compression 40–60%
lossless Broader detail retention, lower savings 20–40%

Required Input Artifacts (temp/context_pack_vN/)

  1. context_pack.md — task-focused compressed context (human-readable).
  2. context_manifest.json — reversible chunk → source mapping (100% manifest coverage).
  3. context_retrieval_index.json — tag/symbol/file/time lookup.
  4. context_stats.md — compression statistics + confidence metrics.
  5. context_pack_reasoning_log.md — audit trail of compression decisions.

Output-Side Compression Levels

Level Behaviour Example
lite Full sentences; no preamble/hedging/question-restatement "Wrap the object in useMemo; a new reference is created every render."
full (default) Sentence fragments, imperative voice, minimal connectives "New ref each render. Wrap object in useMemo."
ultra Telegraphic fragments, symbols over words "New ref/render — useMemo it."

Say normal mode, verbose, or stop terse at any time to revert.


Installation (PyPI)

pip install token-optimizer-plugin

Python API Usage

from token_optimizer import compute_stats, savings_report

# Output-side savings & invariant check
report = savings_report(
    "Great question! Please run `pytest -q` in src/app/main.ts on v1.2.0.",
    "Run `pytest -q` in src/app/main.ts (v1.2.0).",
)
print(report)

# Input-side context pack stats
stats = compute_stats(raw_chars=400, compressed_chars=100)
print(stats.savings_percent)  # 75.0

CLI Usage

# Output-side savings report
token-optimizer savings --raw "Please run `npm test` on v1.0.0" --lean "Run `npm test` (v1.0.0)"

# Input-side context pack stats
token-optimizer stats --raw-chars 400 --compressed-chars 100

# Rank files for a goal
token-optimizer rank --goal "error triage" README.md

# Type-aware file compression
token-optimizer compress --goal "optimize" skills/input-token-optimizer/scripts/context_pack_builder.py

Runtime Requirements & Testing

  • Python 3.9+ (stdlib only — no third-party runtime dependencies).
  • Run the full aggregate test suite (33 tests across both sibling suites) from the plugin root:
python skills/token-optimizer/assets/tests/run_all_tests.py

Metadata

Release files for token-optimizer-plugin 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for token-optimizer-plugin 1.0.0
File Size Uploaded Provenance Actions
token_optimizer_plugin-1.0.0.tar.gz 34.7 kB With provenance Details for token_optimizer_plugin-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for token-optimizer-plugin 1.0.0
File Interpreter ABI Platform Actions
token_optimizer_plugin-1.0.0-py3-none-any.whl Python 3 none any Details for token_optimizer_plugin-1.0.0-py3-none-any.whl

Total release size: 45.8 kB

Release files / token_optimizer_plugin-1.0.0.tar.gz

Download URL token_optimizer_plugin-1.0.0.tar.gz
Size 34.7 kB
Tags Source
SHA-256 checksum
How to use checksums
d5cb7e0e6644dfb14b73000176d0de8ee677ac024d6abcbe4b8a3473b1214479
BLAKE2b-256 checksum
How to use checksums
21fa765efc0d56bf6b8fb2230e194992c12463afe24c1f5c8013975d10f87ce2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Release files / token_optimizer_plugin-1.0.0-py3-none-any.whl

Download URL token_optimizer_plugin-1.0.0-py3-none-any.whl
Size 11.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
723eb298a284839c1a1da940415983c46a864c5db11af58275eb2257d667ee50
BLAKE2b-256 checksum
How to use checksums
d3273871a186fa93d55c67cf86956f22ce6dcd85672b57b1fb4cba15d6afc12e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Security for token-optimizer-plugin 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for token_optimizer_plugin-1.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page