tokenome

Code provenance for AI-assisted work. Your code says what changed; tokenome says why. It captures your conversations with AI coding tools (Claude Code, Claude Desktop, ChatGPT, Gemini), indexes them on your own machine, and links any line of code to the conversation that produced it.

Search is hybrid (BM25 keyword plus vector meaning) over a local Typesense. You reach it from the desktop app, the command line, or a web page; an AI agent reaches it over MCP. Nothing leaves the machine unless you join a team server and opt a project in.

Install

Desktop app (macOS Apple silicon, Linux). Download the installer from github.com/tokenome/releases. The app runs the daemon, bundles its own Python and Typesense, and installs a tokenome command at ~/.local/bin/tokenome that uses the app's runtime.

Command line only (Linux, servers, anywhere uv runs). The distribution on PyPI is tokenome-ai; the command and the import name are tokenome:

uv tool install tokenome-ai
uv tool upgrade tokenome-ai

Needs uv and Python 3.12 or later. Typesense is downloaded on first run; no Docker. If the desktop app is installed on the same machine, uv tool install replaces the app's ~/.local/bin/tokenome with the wheel's own: two runtimes, one data directory. Keep one or the other.

From a checkout (contributors): uv sync, then prefix every command below with uv run. The examples use that form; a wheel or app install drops the prefix.

Quick Start

# 1. Start Typesense (auto-downloads binary on first run)
uv run tokenome start

# 2. Initialize the collection
uv run tokenome setup

# 3. Ingest your Claude Code transcripts
uv run tokenome ingest ~/.claude/projects/ --format claude-code

# 4. Search your conversation history
uv run tokenome search "Python type hints"

# 5. Check what's in the index
uv run tokenome status

Ingesting Conversations

Claude Code (CLI transcripts)

Claude Code stores full JSONL transcripts in ~/.claude/projects/<project>/<uuid>.jsonl.

Alongside the conversation text, the tool actions Claude Code took are captured as compact, searchable lines - [Edit: src/config.py], [Bash: uv run pytest], [Read: …], [Grep: …] - so you can find what was done, not just what was said. Full file contents and diffs are omitted to keep the index lean.

# Single transcript
uv run tokenome ingest ~/.claude/projects/my-project/session.jsonl

# Entire directory of transcripts
uv run tokenome ingest ~/.claude/projects/my-project/ --format claude-code

# Preview without indexing
uv run tokenome ingest ~/.claude/projects/ --format claude-code --dry-run

Claude Desktop / Cowork (API)

Claude Desktop conversations are fetched via the claude.ai API using your session cookie.

# 1. Get your sessionKey from Claude Desktop:
#    - Open Claude Desktop
#    - Press Cmd+Option+I (DevTools)
#    - Go to Application > Cookies > claude.ai
#    - Copy the "sessionKey" cookie value

# 2. Set it as an environment variable
export TOKENOME_CLAUDE_SESSION_KEY=sk-ant-sid01-...

# 3. Fetch and index your conversations
uv run tokenome fetch-claude --max-conversations 50

# Preview first
uv run tokenome fetch-claude --max-conversations 10 --dry-run

Claude Cowork Output Files

Cowork writes deliverables (reports, code, spreadsheets) to your filesystem. Index them as AI-generated documents. For code files inside a git repository, git blame metadata (committer and timestamp per line) is automatically harvested.

uv run tokenome ingest ~/projects/cowork-output/ --format cowork-output

ChatGPT Exports

Export your ChatGPT data (Settings > Data Controls > Export Data), unzip, and ingest the conversations.json:

uv run tokenome ingest conversations.json --format chatgpt-export

Other Formats (Generic JSON)

For flat JSON arrays of {role, content} messages from third-party exporters:

uv run tokenome ingest chat-export.json --format generic-json

Automatic Ingestion (Recommended)

The simplest way to keep your memory current: just run the app and pick your AIs

• no paths to configure.

uv run tokenome app

app auto-discovers conversation sources on your machine, auto-enables the ones it detects, and polls them in the background while the app is open. Open the Sources panel in the web UI to see what was detected, toggle sources on or off, or click Ingest now:

Source	How it's found	Notes
Claude Code	~/.claude/projects/ (auto)	Detected and polled automatically
Claude Desktop / Cowork	claude.ai API	Available once TOKENOME_CLAUDE_SESSION_KEY is set
ChatGPT	drop folder ~/.tokenome/import/chatgpt/	Drop an exported conversations.json there
Gemini	drop folder ~/.tokenome/import/gemini/	Drop a flat-JSON export there

Directory and drop-folder sources are polled every minute (only new/changed files are re-indexed); the Claude Desktop API is polled every 15 minutes. The drop folders are created automatically - the Sources panel shows their exact paths. Within a changed file, only the turns that are new or changed are re-embedded (see Incremental ingestion), so a live, growing session costs roughly one turn's work per poll, not the whole thread.

For headless/CLI-driven ingestion, the watch mode and Claude Code hook below remain available.

Auto-Ingestion (Watch Mode)

Watch a directory for new transcripts and ingest them automatically:

uv run tokenome watch ~/.claude/projects/ --format claude-code --interval 5

Press Ctrl+C to stop.

Connect Claude Code (Recommended)

One command gives Claude Code memory - the MCP server, a skill that teaches Claude when and how to use it, the retrospective-prompt nudge, automatic session ingestion, and /tokenome:why / /tokenome:search:

uv run tokenome app            # once: starts the daemon and mints your API token
uv run tokenome claude install # registers the MCP server + installs the plugin

The daemon serves MCP at http://127.0.0.1:8741/mcp; the plugin's hooks and Claude's tool calls all go through it, so team routing (when enrolled) and local auth apply everywhere. Keep the daemon running (the tokenome desktop app, or tokenome app) whenever you want memory available. Details: plugins/claude/README.md and docs/design/DESIGN_8_CLAUDE_SKILL.md.

The sections below are the manual, piece-by-piece alternative.

Auto-Ingestion (Claude Code Hook)

Index every Claude Code session automatically when it ends - no manual step. Register tokenome ingest-hook as a SessionEnd hook. It reads the hook payload from stdin, finds the session's transcript_path, and ingests it.

Add this to your Claude Code settings.json (~/.claude/settings.json for all projects, or .claude/settings.json for one project). Use the absolute path to tokenome's installed entry point, since the hook fires in any directory:

{
  "hooks": {
    "SessionEnd": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "/absolute/path/to/tokenome/.venv/bin/tokenome ingest-hook",
            "timeout": 120,
            "async": true
          }
        ]
      }
    ]
  }
}

"async": true lets Claude Code background the hook so it never delays session exit. The hook is intentionally silent and always succeeds - if Typesense isn't running, it skips quietly. Diagnostics are appended to ~/.tokenome/hook.log.

Typesense must be running for the hook to index (tokenome start). The hook re-reads the full transcript each time it fires, but ingestion is incremental: only new or content-changed turns are re-embedded and upserted, so re-runs are cheap and idempotent (see Incremental ingestion).

Team sharing (experimental)

Share opted-in projects with your organization's self-hosted tokenome server (see docs/remote-search-spec.md and packages/tokenome-server/). Nothing is shared by default; secrets are scrubbed client-side before anything leaves the laptop, and pushes queue offline-safe with retry.

# One-time: enroll this laptop (token comes from your org admin, out-of-band)
uv run tokenome remote enroll --server https://tk.example.org --token <token>

# Opt a project in - from then on its new/changed segments mirror to the
# server automatically as the poller indexes them (secret-redacted, with
# locally computed vectors, so the server never re-embeds)
uv run tokenome remote share tokenome

# Inspect / manage
uv run tokenome remote status      # enrollment, server health, queue depth
uv run tokenome remote projects    # what you're sharing
uv run tokenome remote push        # flush the offline queue now
uv run tokenome remote unshare tokenome --purge   # stop + retract from server

Once enrolled, tokenome search (and the web Search page) go team-first: queries hit the org server, and if it can't answer within ~1.5s you get your local results with a visible warning instead. Control it with --scope (web: the Scope selector): team (server), mine (local only - includes your private, unshared context), all (both, as two sections, deduplicated so a document you shared never appears twice), or auto (the default: team when enrolled, else mine). Team results show who shared each hit.

Authentication is device-key based (an Ed25519 key generated on your laptop; only the public key is sent). No passwords, no third-party identity provider.

The complete guide - starting the server, admin workflows, connecting laptops, scopes, troubleshooting - is docs/team-server-guide.md; the turnkey docker compose bundle (Typesense + server + automatic TLS, with backup/restore runbook) is in deploy/.

Server Management

# Start Typesense (auto-downloads ~40MB binary on first run)
uv run tokenome start

# Stop Typesense
uv run tokenome stop

# Reset everything - drop and re-create the collection
uv run tokenome clean           # prompts for confirmation
uv run tokenome clean --force   # skips confirmation

The same reset is available in the app under Settings → Danger zone ("Clear all indexed data"), which also resets background-poller tracking so detected sources re-ingest from scratch on the next poll.

The binary is cached in ~/.tokenome/bin/, data in ~/.tokenome/data/.

Incremental ingestion

Ingestion is incremental at the segment level. Every ingest path - the CLI, the web poller, the Claude Code hook, and the Claude Desktop API - reconciles a conversation against what's already indexed rather than rebuilding it:

• Each segment carries a content_hash of its text, stored in the index.
• On (re-)ingest, only segments that are new or whose content changed (e.g. a live session's final turn that grew) are re-embedded and upserted. Unchanged turns are skipped - no wasted embedding work.
• Upserts are per-document, so a growing conversation is never deleted and re-added; there's no window where a search misses it.
• Segments that segmentation no longer produces are pruned as orphans.

Clear all is lossless. Because the hashes live in the index itself, "Clear all indexed data" (tokenome clean / Settings → Danger zone) drops them along with everything else, so the next ingest re-embeds and re-indexes the full corpus from scratch under the current code - verified by an integration test that asserts the rebuilt index is identical to the original.

Searching

CLI

# Basic search
uv run tokenome search "deployment strategies"

# Filter by platform, model, or project
uv run tokenome search "error handling" --platform claude-code
uv run tokenome search "error handling" --model claude-sonnet-4
uv run tokenome search "refactor" --project tokenome

# Filter by the derived AI label (kept for compatibility)
uv run tokenome search "error handling" --ai-label claude-sonnet-4

# Search within a specific conversation
uv run tokenome search "*" --conversation My-Project-Name

# Time-range search
uv run tokenome search "refactor" --after 2025-03-01 --before 2025-03-31
uv run tokenome search "*" --around 2025-03-15T14:30 --window 30

# JSON output (for scripting)
uv run tokenome search "Python" --json | jq .

Cross-referencing code with conversations

Find the conversations that led to a change: tokenome blames a line to learn when it was last edited, then surfaces the conversations from around that time.

# When was this line last edited, and what did I discuss just before?
uv run tokenome context src/config.py 42

# Widen to conversations on both sides of the edit (±30 min by default)
uv run tokenome context src/config.py 42 --symmetric --window 45

By default it shows the conversations immediately before the edit - the discussions most likely to have driven it. Blame is resolved live via git blame (so it reflects the current state and needs no prior ingestion, as long as the file is in a local git repo); if the file isn't in git, it falls back to blame captured at ingest time from a --format cowork-output import.

The same lookup is available in the app's Code context pane.

Web UI

# One command - starts Typesense and the daemon, and opens the browser
uv run tokenome app

The page the daemon serves at http://127.0.0.1:8741/ is the desktop app's UI (DESIGN_7 §10): Dashboard, Search, Code context, Conversations, Journal, FAQ, Team, Sources, and the Settings that make sense in a browser. Shell-only controls (launch at login, updates) disable themselves outside the app. It is the UI for Linux and headless daemons, where there is no desktop app yet.

The page itself carries no credentials. tokenome app opens the browser with a single-use launch link that signs the tab in (an HttpOnly session cookie). To open a signed-in tab on a daemon that is already running (the desktop app's, or tokenome daemon), run:

uv run tokenome open            # --port if the daemon is not on 8741

Typing http://127.0.0.1:8741/ by hand loads the page but not your data; the daemon answers only requests addressed to a loopback host name (127.0.0.1, localhost, [::1]), which blocks DNS-rebinding pages.

From a checkout the UI must be built once (needs Node):

apps/desktop/scripts/install-spa.sh   # stages the build for the daemon

Until it is staged, the page at / says so; the API works regardless.

MCP Server (for LLMs)

The daemon serves MCP over Streamable HTTP at http://127.0.0.1:8741/mcp, guarded by the local API token (~/.tokenome/api.token). Claude Code, Claude Desktop, and other MCP clients query your conversation memory through it - one implementation, with team routing when you are enrolled.

Claude Code: uv run tokenome claude install (see Connect Claude Code). It runs claude mcp add --transport http --scope user --header "X-Tokenome-Token: …" tokenome http://127.0.0.1:8741/mcp for you.

Claude Desktop (stdio only): keep the tokenome-mcp entry in ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "tokenome": {
      "command": "uv",
      "args": ["run", "tokenome-mcp"]
    }
  }
}

tokenome-mcp is a thin proxy to the daemon's /mcp, so the daemon must be running. Without a checkout - the desktop app's Install command-line tool puts tokenome on your PATH - use the equivalent subcommand instead:

{
  "mcpServers": {
    "tokenome": { "command": "tokenome", "args": ["mcp"] }
  }
}

TOKENOME_MCP_DIRECT=1 restores the legacy direct-engine mode until it is removed.

Tools available to the model:

Tool	Description
search_memory(query, top_k, ai_label, platform, model, project, scope)	Hybrid search, routed: scope=auto (team when enrolled, else local), team, mine, all. Team hits carry owner; a warning says when the scope could not be honored
why_was(decision, top_k)	Distilled rationale from journal why-records, with the evidence span
find_related(doc_id, top_k)	Nearest-neighbor segments to a result - walk the memory graph
get_document(doc_id)	Full text of a single segment (expand a search/related excerpt)
get_conversation(conversation_id, full)	A whole thread, compact per-turn excerpts (or verbatim with full=True)
find_conversations_near(file, line, window, symmetric)	Git-blame a line, then the conversations just before that edit
ask_faq(question)	Curated/pinned answers
get_journal(date, days) / get_recent(limit)	The daily journal; the most recently indexed segments
browse_by_label(ai_label, limit)	Browse conversations with a specific AI (compact)

Search-style tools return compact results to keep an agent's context lean: each result is a highlighted excerpt plus doc_id/conversation_id/title. The agent expands what it needs with get_document (one segment) or get_conversation (the whole thread), and explores laterally with find_related.

Configuration

All settings via environment variables with sensible defaults:

Variable	Default	Description
TOKENOME_TYPESENSE_HOST	localhost	Typesense server host
TOKENOME_TYPESENSE_PORT	8108	Typesense server port
TOKENOME_TYPESENSE_API_KEY	xyz	Typesense API key
TOKENOME_TYPESENSE_PROTOCOL	http	http or https
TOKENOME_EMBEDDING_MODEL	BAAI/bge-small-en-v1.5	fastembed model name
TOKENOME_EMBEDDING_BATCH_SIZE	32	Embeddings per batch
TOKENOME_INGESTION_BATCH_SIZE	50	Documents per Typesense import batch
TOKENOME_CLAUDE_SESSION_KEY	(none)	sessionKey cookie for fetch-claude
TOKENOME_CLAUDE_ORG_ID	(auto)	Organization UUID for fetch-claude

Architecture

Transcripts / API → Segmenter → Embedder → Typesense (BM25 + vector)
                                              │
                    ┌─────────────────────────┼──────────────────────┐
                    ▼                         ▼                      ▼
              CLI (tokenome)          MCP Server (tokenome-mcp)   Web UI (tokenome app)
              ingest/search/status    AI-accessible tools         http://127.0.0.1:8741

• Segmentation: Pluggable adapters parse each AI's transcript format into user-query/ai-response pairs, tagged with platform, model, and project for filtering (with a derived ai_label kept for compatibility)
• Embedding: fastembed (ONNX, local, no GPU) generates 384-dim vectors from BAAI/bge-small-en-v1.5. Long turns are split into overlapping chunks, embedded, and mean-pooled so content beyond the model's ~512-token limit is not lost
• Indexing: Typesense stores documents with hybrid search - BM-25 text matching fused with vector similarity at alpha=0.5
• Access: Same search backend serves CLI, web UI, and MCP server

Development

uv sync                             # Install dependencies
uv run pytest -m "not integration"  # Unit tests (fast, no services needed)
uv run pytest -m integration        # Integration tests: starts an isolated,
                                    #   throwaway Typesense (never touches your
                                    #   real index) and uses the real embedding
                                    #   model (downloaded on first run)
uv run pytest -m e2e                # Browser E2E tests over the app's UI (needs Node)
                                    #   and the desktop SPA (needs Node:
                                    #    npm ci in apps/desktop, once)
                                    #   (one-time setup:
                                    #    uv run playwright install chromium)
uv run pytest                       # Every suite
uv run ruff check .                 # Lint
uv run ruff format .                # Format

(cd apps/desktop && npm ci && npm test)   # Desktop app unit tests (Vitest)

# End-to-end smoke test (starts Typesense, ingests, searches, cleans up)
uv run python scripts/smoke_test.py

CI enforces a branch-aware coverage floor on the unit suite (--cov-fail-under, ratchet-up-only - see .github/workflows/ci.yml for the current value); the integration suite reports coverage informationally but is gated only on passing. To see coverage locally, add --cov to either command.

License

The tokenome client, meaning the tokenome-ai and tokenome-core packages and the Claude Code plugin, is licensed under the Business Source License 1.1 (see LICENSE). You may use, modify and redistribute it, including in production, except to offer a competing product. Four years after each version is published, that version converts to the Apache License 2.0. The desktop app is distributed under its own end-user terms from tokenome/releases. The team server is proprietary and licensed separately.