tokens.md

Tokens.md is my tool for saving tokens when speaking to chatbots by converting files en-masse to Markdown. It turns PDFs, Office documents, e-books, structured data, HTML, web pages, and whole code repositories into clean, token-efficient Markdown you can paste straight into an LLM.

OpenAI's tiktoken is used to estimate how many tokens are saved (pretty accurately). In some preliminary results, this conversion is usually anywhere between 60% and 95% of tokens saved (smaller files benefit "more" since a larger proportion of their data is overhead).

Outline

• Features
• Install
• Running
  • 1. CLI Usage (tmd)
  • 2. Simple Script Execution (python src/main.py)
  • 3. Web Front End (tmd ui & Next.js)
• Supported Formats
• Development
• Documentation
• License

Features

The features of this tool are all encompassed by the tmd CLI commands. The front end displays this with an accessible GUI.

• tmd convert: convert files to Markdown.
• tmd clip: convert files to markdown and copy the result to your clipboard.
• tmd watch: watch a hot folder and auto-convert new files as they appear.
• tmd fetch: pull a web page and save clean article Markdown.
• tmd repo: collapse an entire code repository into a single Markdown manifest.
• tmd merge: combine many files into one master document with a Table of Contents.
• tmd delta: show how many tokens were saved.
• --budget: an extra argument to force output size in tokens.

See the usage guide for examples

Install

Pre-Game

First, make sure you have Python 3.13+, uv, and TypeScript support on your machine. Clone this repository:

git clone https://github.com/intelligent-username/tokens.md

And create the environment:

uv venv --python=3.13 .venv

# On Linux or Mac
source .venv/bin/activate

# On Windows
.venv\Scripts\activate

uv pip install -e .               # editable install (provides `tmd`)

To update dependencies, just run uv sync.

Or install the runtime dependencies directly:

uv pip install -r requirements.txt

Running

You can run tokens.md in three distinct ways depending on your workflow preference. Here they are, ordered from easiest to hardest to use.

1. Simple Script Execution (python src/main.py)

If you want to run the program directly without installing it as a package, run src/main.py using your Python interpreter:

python src/main.py

You can pass standard CLI arguments directly to the script:

python src/main.py convert input/ -o output/

Both bare tmd and python src/main.py automatically resolve default input/ and output/ folders relative to the project root.

2. Web Front End (tmd ui & Next.js)

The single-page web interface wraps all tmd capabilities into an intuitive side-by-side visual workbench featuring drag-and-drop file upload, URL fetching, clipboard copying, and token budgeting.

Method A: Single CLI Command (tmd ui)

uv pip install -e ".[web]"
tmd ui

This launches the backend server on http://127.0.0.1:8642 and opens your browser.

Note: tmd ui automatically looks for built static assets inside tmd_ui_static/ (installed package source) or frontend/out/ (local repository source). If found, it serves both the REST API and the frontend directly on http://127.0.0.1:8642 as a single unified process.

Method B: Standalone Development Servers If you are developing or modifying the React/Next.js interface and want hot-reloading:

1. Start the FastAPI backend server:

   python -m backend
   

2. In a separate terminal, start the Next.js development server:

   cd frontend
   npm install
   npm run dev
   

3. Open http://localhost:3000 in your web browser. (The Next.js dev server proxies API calls to http://127.0.0.1:8642).

3. CLI Usage (tmd)

After installing via pip install -e ., the tmd command is registered in your environment.

Navigate to the folder containing the files you want to convert and run the following commands. You may manipulate sub-folders and file names, or omit them to use the defaults.

# Convert all supported files in current repository and write to out/ folder
tmd convert . --loc="out"

# Convert current folder into current directory
tmd convert . --loc

# Bare `tmd` command uses default input/ and output/ directories
tmd

# Other CLI subcommands
tmd merge input/ -o output/merged.md
tmd fetch https://example.com/article -o output/article.md
tmd repo . -o output/repo.md

Development

Install dev dependencies and run the test suite and linters:

uv pip install -e ".[dev]"

Testing

Backend (pytest):

pytest                           # Run tests with coverage report
pytest --cov-report=html         # Generate HTML coverage report in htmlcov/

Frontend (vitest):

cd frontend
npm install                       # First-time setup
npm test                          # Run tests once with coverage
npm run test:watch                # Watch mode for development

Run both test suites in parallel with a unified coverage summary:

tmd test                          # Run both backend and frontend test suites
python scripts/test.py            # Direct script invocation
tmd test -v                       # Verbose: full test output

Linting

python scripts/lint.py            # Cross-platform: Ruff + Prettier + ESLint
.\scripts\lint.ps1                # Windows PowerShell script
python scripts/lint.py --fix      # Auto-fix linter and formatting errors

Testing PyPI Releases in Isolated Docker Sandbox

To test installed PyPI releases in an ephemeral state, take the following steps:

# Launch disposable Python container
docker run --rm -it python:3.12-slim bash

# Inside the container shell:
pip install pipx
pipx install tokens-md
pipx ensurepath
export PATH="$HOME/.local/bin:$PATH"

# Test the tmd executable
tmd --version
tmd --help

# More commands...
exit

Exiting the shell (exit) automatically destroys the container and leaves zero footprint on your system. These steps can be done in a permanent environment as well (a real docker container or native install), but pipx won't be performing a native install.

Note on pipx vs pip:

• pipx install tokens-md: Recommended for global CLI use—installs tmd as an isolated system-wide executable without needing an active virtualenv.
• pip install tokens-md (or uv pip install tokens-md): Best when working inside an active Python virtual environment where you want tmd registered directly in that environment's PATH.

Documentation

• docs/USAGE.md: full usage guide for every tmd subcommand.
• docs/ARCHITECTURE.md: how the converter registry works and how to add new formats.

License

This project is licensed under the GNU AFFERO GENERAL PUBLIC LICENSE. See more here