tokens.md

Tokens.md is my tool for saving tokens when speaking to chatbots by converting files en-masse to Markdown. It turns PDFs, Office documents, e-books, structured data, HTML, web pages, and whole code repositories into clean, token-efficient Markdown you can paste straight into an LLM.

OpenAI's tiktoken is used to estimate how many tokens are saved (pretty accurately). In some preliminary results, this conversion is usually anywhere between 60% and 95% of tokens saved (smaller files benefit "more" since a larger proportion of their data is overhead).

Outline

• Features
• Install
• Running
  • 1. CLI Usage (tmd)
  • 2. Simple Script Execution (python src/main.py)
  • 3. Web Front End (tmd ui & Next.js)
• Supported Formats
• Development
• Documentation
• License

Features

The features of this tool are all encompassed by the tmd CLI commands. The front end displays this with an accessible GUI.

• tmd convert: convert files to Markdown.
• tmd clip: convert files to markdown and copy the result to your clipboard.
• tmd watch: watch a hot folder and auto-convert new files as they appear.
• tmd fetch: pull a web page and save clean article Markdown.
• tmd repo: collapse an entire code repository into a single Markdown manifest.
• tmd merge: combine many files into one master document with a Table of Contents.
• tmd delta: show how many tokens were saved.
• --budget: an extra argument to force output size in tokens.

See the usage guide for examples

Install

Pre-Game

First, make sure you have Python 3.13+, uv, and TypeScript support on your machine. Clone this repository:

git clone https://github.com/intelligent-username/tokens.md

And create the environment:

uv venv --python=3.13 .venv

# On Linux or Mac
source .venv/bin/activate

# On Windows
.venv\Scripts\activate

uv pip install -e .               # editable install (provides `tmd`)

To update dependencies, just run uv sync.

Or install the runtime dependencies directly:

uv pip install -r requirements.txt

Running

You can run tokens.md in three distinct ways depending on your workflow preference. Here they are, ordered from easiest to hardest to use.

1. Simple Script Execution (python src/main.py)

If you want to run the program directly without installing it as a package, run src/main.py using your Python interpreter:

python src/main.py

You can pass standard CLI arguments directly to the script:

python src/main.py convert input/ -o output/

Both bare tmd and python src/main.py automatically resolve default input/ and output/ folders relative to the project root.

2. Web Front End (tmd ui & Next.js)

The single-page web interface wraps all tmd capabilities into an intuitive side-by-side visual workbench featuring drag-and-drop file upload, URL fetching, clipboard copying, and live token compression flow meters.

Method A: Single CLI Command

uv pip install -e ".[web]"
tmd ui

This launches the backend API on http://127.0.0.1:8642 and opens the browser interface.

Method B: Development Server (Frontend + Backend)

1. Start the FastAPI backend server:

   python -m backend
   

2. In a separate terminal, start the Next.js frontend dev server:

   cd frontend
   npm install
   npm run dev
   

3. Open http://localhost:3000 in your web browser.

3. CLI Usage (tmd)

After installing via pip install -e ., the tmd command is registered in your environment.

Navigate to the folder containing the files you want to convert and run the following commands. You may manipulate sub-folders and file names, or omit them to use the defaults.

# Convert a folder or file to Markdown
tmd convert input/ -o output/

# Bare `tmd` command uses default input/ and output/ directories
tmd

# Other CLI subcommands
tmd merge input/ -o output/merged.md
tmd fetch https://example.com/article -o output/article.md
tmd repo . -o output/repo.md

Development

Install dev dependencies and run the checks:

uv pip install -e ".[dev]"
pytest
ruff check .
mypy src

Testing PyPI Releases in Isolated Docker Sandbox

To test installed PyPI releases in an ephemeral, 100% isolated environment using pipx via Docker:

# Launch disposable Python container
docker run --rm -it python:3.12-slim bash

# Inside the container shell:
pip install pipx
pipx install tokens-md
pipx ensurepath
export PATH="$HOME/.local/bin:$PATH"

# Test the tmd executable
tmd --version
tmd --help

Exiting the shell (exit) automatically destroys the container and leaves zero footprint on your system.

Documentation

• docs/USAGE.md: full usage guide for every tmd subcommand.
• docs/ARCHITECTURE.md: how the converter registry works and how to add new formats.

License

This project is licensed under the GNU AFFERO GENERAL PUBLIC LICENSE. See more here