tokenflame

Token-level behavioral profiler for LLM generation. "Flame graph for LLM reasoning."

Works with prepared RAG context and structured prompt inputs. Not a document ingestion framework.

Install

From PyPI (coming soon): pip install tokenflame

From source (current): git clone https://github.com/bh3r1th/tokenflame cd tokenflame/packages/backend pip install -e .

Quick Start

tokenflame run --prompt "What is DNA?" --model-a openai/gpt-5.5 --model-b ollama/llama4-maverick

Output Files

Traces are auto-named with timestamp and prompt slug by default: tokenflame_what_is_dna_20260525_172300.json tokenflame_what_is_dna_20260525_172300.html Use --out to specify an explicit filename: tokenflame run --prompt "..." --out my_trace.json --html

With RAG Context

tokenflame run
--prompt "What does the policy say about returns?"
--context-file retrieved_chunks.txt
--context-mode system
--model-a openai/gpt-5.5
--model-b ollama/llama4-maverick
--out trace_returns.json
--html
--open

View a Saved Trace

tokenflame view trace_returns.json tokenflame view trace_returns.html

Supported Context Formats

PDF and DOCX are not supported. Extract text first, save as .txt.

Configure

Create ~/.tokenflame/config.toml:

[providers] openai_api_key = "sk-..." anthropic_api_key = "sk-ant-..." groq_api_key = "..." ollama_host = "http://localhost:11434"

[ui] default_model_a = "openai/gpt-5.5" default_model_b = "ollama/llama4-maverick"

Or use environment variables: OPENAI_API_KEY ANTHROPIC_API_KEY GROQ_API_KEY OLLAMA_HOST

Available Models

Models without logprob support stream tokens but show no entropy signal.

What tokenflame Shows

• Entropy heatmap — which tokens were uncertain vs confident
• Tokenizer diff — where GPT and Llama split text differently
• DTW alignment — where outputs structurally diverged
• Divergence markers — exact fork points scored by similarity
• Replay — scrub through generation token by token

What tokenflame Does NOT Do

• Retrieval or embedding
• Document parsing (PDF, DOCX)
• Eval scoring or CI/CD gates
• Agent tracing
• Multi-turn conversation

Trace Schema

All traces are saved as JSON following schema/trace.schema.json. Build your own viewer on top of the schema — it is stable.

License

MIT