ken-rank 0.10.2

Per-project context-rank index for Claude Code, served from a local daemon.

ken

ken is a local context-rank index for coding agents such as Claude Code and Codex CLI. It indexes each project into a local SQLite database, watches for changes, and gives the assistant a ranked <context-rank> block before each prompt so it can start from the most relevant files instead of exploring from scratch.

Everything runs locally: project metadata, embeddings, the daemon, and the SQLite index stay on your machine.

Why it works

ken is not only a code embedding index. It builds several local signals and ranks files by combining them:

• Structural code index: each file is hashed and parsed when possible. The AST parser extracts symbols, line ranges, imports, module docstrings, and symbol docstrings. Files that cannot be parsed are still tracked by path, mtime, and lightweight text intent when useful.
• Semantic index: ken embeds files, symbols, and explicit purpose text. File embeddings are based on language, filename, and top symbol names. Symbol embeddings include kind, name, and docstring. Docstrings are also stored as separate intent sources, so a prompt can find code by what it is for, not only by what it is named.
• Live task memory: hooks record local interactions in the project SQLite DB: prompts, reads, edits, writes, dismissals, and the files touched in each turn. Recent interactions are weighted more heavily, and useful patterns such as read-then-edit score higher than repeated reads with no follow-through.
• Predictive memory: at the end of a session, ken snapshots which files were productive for that task. Later, when a new prompt is semantically similar to a previous one, those files get a predictive boost.
• Relationship boosts: after the main channels rank candidates, ken applies conservative boosts for things like recently modified files, symbols pointing to their containing file, source/test counterparts, imports, and files that often co-occurred in past similar sessions.

The result is closer to a local project memory than a plain search tool. Raw text search can find exact strings. Embeddings can find semantic neighbors. AST indexing can find named symbols. ken combines all of that with how the assistant actually used the project over time. That is why results are often better after a few real sessions: the database accumulates project-specific evidence about which files matter for which kinds of tasks.

All of these signals stay local in .ken/ken.db. They are not sent to a any server.

Install the CLI

ken is published on PyPI as ken-rank. The distribution name is ken-rank, but it installs the ken command. Install it with pipx (recommended, isolates the tool) or uv:

pipx install ken-rank
# or
uv tool install ken-rank
# or
pip install ken-rank

Verify the install:

ken --version

From a checkout

To install from a local clone (for development or an unreleased build):

./install.sh

The installer uses uv to install the ken command into your user-local tool directory. If uv is not present, the script bootstraps it with Astral's official installer unless you pass --no-bootstrap-uv. You can also install the checkout directly with uv:

uv tool install --editable . --force --reinstall --refresh

Install ken in a project

Run this once from the project you want ken to index:

ken install .

By default, ken detects the assistant setup you use and wires itself into the supported local agent config. The install creates .ken/, adds it to .gitignore, installs hooks/MCP config where applicable, and performs the initial structural code index.

Embeddings are lazy by default. That means ken install . does not eagerly embed the whole repository; the daemon warms embeddings as they are needed. This keeps install fast and avoids doing expensive work before the project needs it.

ken also gets better with use. The first run starts from the project index, names, symbols, docstrings, and any available embeddings. As you and the assistant read files, edit code, dismiss weak context, and save findings, ken records those interactions as local ranking signals. Results should improve after real sessions because the system learns which files were useful for similar work in this project.

Force a target assistant

Use --claude when you specifically want Claude Code wiring:

ken install --claude .

Use --codex when you specifically want Codex CLI wiring:

ken install --codex .

Use --opencode when you specifically want OpenCode wiring — it registers ken as a local MCP server in the project's opencode.json (or opencode.jsonc):

ken install --opencode .

You can pass several when a project uses several assistants:

ken install --claude --codex .
ken install --claude --opencode .
ken install --codex --opencode .
ken install --claude --codex --opencode .

OpenCode has no lifecycle-hook equivalent to Claude Code's SessionStart or Codex's Stop — its plugin system is JS/TS-based and runs on Bun/Node. So the OpenCode wiring is narrower than the other two: just the MCP registration. The 30 ken tools (ken_search_files, ken_rank, ken_wiring, …) are available to OpenCode through MCP exactly the way they are to Claude Code, and the daemon-side hook logic (session memory, predictive ranking, finding recall) keeps working because it lives in the daemon, not in the host agent.

Eagerly build embeddings

For better first-run semantic ranking, especially on projects where initial context quality matters, force the initial embedding pass:

ken install --embed .

This is recommended when you can afford the extra install time. The cost scales with the repository, and the ceiling is lower than you would expect: the Linux kernel 7.1-rc2 — 101 084 files walked, 65 907 parsed, 771 563 symbols extracted — indexes in 219 s on a Ryzen 9 8945HS laptop. On very large projects, cap the eager pass while still structurally indexing the whole repo:

ken install --embed --embed-limit 5000 .

You can combine assistant selection and eager embeddings:

ken install --codex --embed .
ken install --claude --embed .

Install CLI and wire a project in one step

From a ken checkout, install.sh can install the CLI and then run ken install for a project:

./install.sh --project /path/to/my-project
./install.sh --project /path/to/my-project --codex --embed

Embedding models & GPU

New projects use ken/static-qwen3-r512-v2 (1024-dim), a fitted token table that ships inside the wheel. It is not a neural network. A text is tokenised, one vector per token is looked up and added up, and the sum goes through a single matrix:

ids  = tokenize(text)          # Qwen3's own tokenizer, bundled with the table
vec  = normalise( Σ table[ids] · projection )

That is ~1000 arithmetic operations per token instead of the 880 million a transformer spends, so it encodes about 10 000 texts a second on one CPU thread — three orders of magnitude faster than running a model — with no GPU, no ONNX, no torch, and nothing to download on first use. The table is 23 MB and the arithmetic is numpy.

For scale: its teacher, Qwen/Qwen3-Embedding-0.6B, encoded 856 texts/s on the datacenter GPU it was fitted on. A Linux-kernel-sized index is upwards of 872 000 texts — about seventeen minutes of pure encoding there, on hardware most people do not have.

It is fitted, not designed: a ridge regression against Qwen/Qwen3-Embedding-0.6B over 964 475 texts of the exact shapes ken stores, parsed with ken's own parsers from the-stack-smol and balanced across sixteen languages.

How good is it? On 4 351 retrieval queries over code held out of its training — one thousand files per language, split off before a single training text was built — it reaches 97% of its teacher's recall@10:

	recall@10
Qwen/Qwen3-Embedding-0.6B (teacher, [torch], ~1.2 GB)	0.728
ken/static-qwen3-r512-v2 (default, 23 MB)	0.708

Per language, table against teacher: typescript 0.92/0.93 · go 0.85/0.87 · rust 0.82/0.87 · csharp 0.79/0.79 · php 0.78/0.76 · java 0.78/0.82 · powershell 0.84/0.87 · python 0.74/0.80 · ruby 0.72/0.73 · bash 0.70/0.66 · c++ 0.66/0.65 · sql 0.25/0.31 · css 0.22/0.23 · html 0.20/0.14. CSS, HTML and SQL are low for both — those files carry few docstrings, so the task itself is hard there, not the table.

And on ranking, which is what ken actually does with it: over 637 real prompts from two projects ranking 2 627 files, asking whether each encoder surfaces the teacher's own top pick in its top ten —

encoder	English prompts (n=373)	Spanish prompts (n=264)
ken/static-qwen3-r512-v2	0.603	0.216
multilingual-MiniLM (previous default)	0.086	0.239

+0.517 in English (95% CI [+0.461, +0.574]); in Spanish the difference is indistinguishable from zero ([−0.091, +0.046]). The table is fitted on code-shaped text, so that is where it is strongest — but it does not cost you anything on conversational prompts in other languages either.

sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 (384-dim) remains available and is the automatic fallback if an install has no table.

A project's model is pinned to its index. Upgrading ken never re-encodes an existing project behind your back — cosine similarity across two models is meaningless, so switching always requires a deliberate re-encode:

# Re-encode every stored embedding with a new model (no re-index; uses the
# source text ken already keeps). Records the model so it stays pinned.
ken reembed --model ken/static-qwen3-r512-v2

Any fastembed model works out of the box too.

To choose the default for future projects (leaving existing ones untouched), set a user-level default:

ken default-model                     # show the current default for new projects
ken default-model BAAI/bge-m3         # every project installed from now on uses this
ken default-model --clear             # back to ken's built-in default

This only affects projects created afterwards; switch an existing one with ken reembed --model <name>.

GPU acceleration

The default needs none. A table lookup and a sum have nothing to offload — the static table runs on the CPU at ~10 000 texts a second and there is no GPU path to want. This section is for the model backends.

The embedder auto-detects a GPU and uses it, falling back to CPU when none is usable — no configuration. What counts as a GPU depends on the backend:

Backend	Accelerators	How to get it
static table (default)	none — it is a lookup, not a model	built in
fastembed / ONNX	CUDA, ROCm — Linux/Windows only	pip install 'ken-rank[gpu]'
torch / sentence-transformers	CUDA, Apple Silicon (MPS)	pip install 'ken-rank[torch]'

Override detection with KEN_EMBED_DEVICE=auto|cpu|gpu|cuda|mps (and KEN_EMBED_DEVICE_ID=0 to pick a CUDA index). auto and gpu prefer CUDA, then MPS, then CPU; naming one accelerator restricts the choice to that one, so cuda on a Mac lands on the CPU rather than quietly substituting the other GPU.

On Apple Silicon, only the torch backend reaches the GPU — ONNX Runtime has no MPS provider, and [gpu] is not installable on macOS at all (onnxruntime-gpu ships no Mac wheels), so the default models stay on the CPU where they are already fast. [torch] alone is enough: the stock PyPI torch wheel for macOS arm64 carries MPS.

MPS is used more defensively than CUDA, because it fails in ways CUDA doesn't:

• It can lie quietly. A batch can come back as NaN, or as rows the GPU never filled in — which survive normalisation as all-zero vectors that are perfectly finite and score 0.0 against every query forever. ken checks that each batch is finite and unit-norm; if it isn't, or the GPU raises, it re-encodes on the CPU and stays there for the rest of the process. One reload, versus unusable vectors in ken.db that nobody notices until retrieval has quietly degraded.
• It won't tell you it's out of memory. torch's MPS allocator ceiling is set from a fraction of RAM and lands above the RAM that exists, so instead of an out-of-memory error macOS just swaps. ken caps it (KEN_MPS_MEMORY_FRACTION, default 0.7) so an oversized batch raises and hits the fallback above.
• torch 2.9+ is required to use the GPU (macOS 14+). Below that, a PyTorch bug could return embeddings that are finite, unit-norm and wrong — the one failure no runtime check can catch — so ken refuses MPS on older torch and says so in the log rather than trusting an index it can't verify. The [torch] extra pins this for you on Apple Silicon.

Is the GPU worth it? For most people, not much — and that's fine. The device only affects how fast a query is embedded; the rest of a rank (the cosine sweep over the mapped vectors, the lexical/import channels) is CPU work either way. With the default table a query embeds in well under a millisecond, and with a fastembed model in ~30–40 ms on CPU, so the GPU barely moves the needle for inline ranking. Even with the heavy Qwen/Qwen3-Embedding-0.6B on a large repo, a full rank was ~1.1 s of which the query embed is ~250 ms on CPU vs ~50 ms on GPU (measured on CUDA — Apple Silicon is unmeasured) — the GPU saves ~200 ms of a second-plus that's dominated by device-independent work. So CPU-only is perfectly usable, including with Qwen3.

Where the GPU genuinely pays off: bulk work — a full ken reembed or the first index of a big repo re-encodes thousands of texts at once, and there the GPU is several times faster. And if you simply want to shave every last millisecond off inline ranking, turn it on. Otherwise, don't sweat it.

Stronger models (torch backend)

Some of the best open-source embedding models are not shipped by fastembed. The optional torch extra adds a sentence-transformers backend so you can use them — most notably Qwen/Qwen3-Embedding-0.6B, the top scorer in ken's own retrieval benchmark, and BAAI/bge-m3:

pip install 'ken-rank[torch]'
ken reembed --model Qwen/Qwen3-Embedding-0.6B

ken selects the backend automatically from the model name — just point ken reembed --model at it.

How good is Qwen3? On ken's benchmark (100 labeled prompt→file queries over a real project with Spanish prompts and English code), Qwen3-0.6B was clearly the strongest model tested:

Model	Recall@5	MRR	Spanish Recall@5
Qwen3-Embedding-0.6B ([torch])	0.77	0.68	0.75
multilingual-MiniLM (previous default)	0.59	0.47	0.59
all-MiniLM (old English-only default)	0.54	0.36	0.33

Those numbers come from a different, older benchmark than the ones at the top of this section, so read them against each other and not across — what they establish is the ordering among model backends. The default table is distilled from the winner of this table and reaches 97% of it on held-out code, so [torch] is worth installing when you want the last few points of recall and can afford 1.2 GB and ~250 ms per query, and is otherwise unnecessary.

That is roughly +40% Recall@5 and +87% MRR over the old default, and it more than doubles retrieval quality on non-English prompts.

The costs. Qwen3 is not free to run:

• Heavier install. The [torch] extra pulls PyTorch + sentence-transformers — hundreds of MB, versus the small ONNX-only default. The model itself is ~1.2 GB.
• Bigger index. It is 1024-dimensional, so stored vectors are ~2.7× the size of the 384-dim default. Since 0.9.0 that lands in .ken/vectors/ rather than in ken.db.
• Slower on CPU. It shines on a GPU — the torch backend picks up CUDA or Apple Silicon's MPS on its own (the [gpu] extra is for the fastembed path, not this one). On CPU the per-prompt embedding is noticeably slower than the fastembed default, which matters because the daemon embeds inline on every prompt. On a Mac, [torch] alone is enough: the stock PyPI torch wheel for macOS arm64 ships MPS support, no extra index or extra to install.

So it is worth it when you have a GPU (or don't mind the CPU latency) and want the best retrieval; otherwise the lightweight multilingual default is the better trade-off. The fastembed default always stays the no-torch path.

Where the vectors live

Since 0.9.0 the vectors are not in ken.db. They live in .ken/vectors/ as memory-mapped, fixed-size segment files, and each row in SQLite carries a vec_slot pointing at one. The relational half — paths, symbols, imports, sessions, findings — stays where it was.

This is not cosmetic. On a Linux 7.1-rc2 index (101 083 files, 771 563 symbols), ken rank took 21.3 s, of which the actual arithmetic was 0.138 s. The rest was deserialising 771 563 Python row objects and 2.9 GB of blobs to feed one matrix-vector product. Mapping them instead:

	before	after
ken rank, end to end	21.3 s	1.82 s
fuzzy channel	20.4 s	0.374 s
lexical channel	22.6 s	0.126 s
ken.db	4.10 GB	214 MB

Ranked output is unchanged: top-k overlap 1.000 across a 12-query set, with scores drifting 5.1e-03 on ~7 from float32 rounding.

Worth saying plainly, because the obvious suspect was wrong: SQLite was never the bottleneck. Measured over the same 2.9 GB, it handed over the vector column at 906 MB/s against 1.5 GB/s for a raw flat file — within 1.7×. Swapping the database engine would have bought 3%.

Migrating an existing project

An index built before 0.9.0 keeps working untouched — every read falls back to the inline column for rows without a slot, so it is correct, just slow. Convert it when convenient:

ken vectors migrate      # move the vectors out, then VACUUM the freed pages
ken vectors verify       # expect 0 leaked / 0 out-of-range / 0 double-booked

Stop the daemon first (ken serve --stop): the migration takes SQLite's write lock in batches, and a running daemon can hit its 5 s busy timeout. It is idempotent and resumable — an interrupted run leaves a partly-converted index that still answers correctly, and re-running finishes it.

ken install --embed . and ken reembed also migrate, so if you were going to run either anyway there is nothing extra to do.

Timing: 2 m 21 s for the kernel's 872 000 vectors. A normal project is seconds.

The rest of ken vectors

ken vectors              # sizes and how many vectors are still inline
ken vectors verify       # cross-check slots against rows
ken vectors compact      # renumber live vectors densely, reclaim leaked slots

compact is for maintenance, not migration: it rebuilds the store into a dense prefix and drops the tail, reclaiming slots stranded by a crash between a vector's write and its commit. verify tells you whether there are any — usually zero.

Segment files are preallocated and sparse, so a space holding a few thousand vectors reports 256 MB of apparent size against ~17 MB of real blocks; ken vectors reports the blocks.

Tell the assistant to use ken

ken works through hooks automatically, but assistants behave better when your project instructions tell them when to reach for ken and, just as importantly, to write back what they learn. The MCP server already describes all 30 tools to the assistant, so the block below deliberately carries only what a tool description cannot: where to start, when to stop, and what to record. Add it to the agent instruction file for the tool you use: AGENTS.md for Codex, CLAUDE.md for Claude Code, or both. OpenCode reads the same AGENTS.md from the project root.

## Code intelligence: ken

**When a prompt arrives with a `<context-rank>` block**, that is ken's ranked guess
for this request: `Files:` best first, `Symbols:`, and `Notes:` — finding *topics*
from past sessions, so `ken_recall` one to read its body. If it names what you need,
open that file and skip searching. If a listed file was irrelevant, `ken_dismiss(path,
reason)` — the ranker's only negative signal, and only useful while you can still see
it. Thin or missing? `ken_rank(verbose=2)`, or `ken_intent_history("<task>")` for the
files past tasks like this one actually touched.

**Before the first search**, don't reach for `rg`, `find`, or open files you are
guessing at. One ken call, matched to the question:
- exact string or identifier (`MY_ENV_VAR`, `os.path`) → `ken_grep`, not `rg`
- which file implements X → `ken_search_files`
- where is the function/class that does X → `ken_search_symbols`
- how a route / CLI command / env var reaches its handler → `ken_wiring`

Then read what ken named: it narrows the search space, it doesn't replace reading
code. Two ken calls is normal, five means you should have opened the file already;
trivial or in-context questions need none. Use `rg` when ken comes back empty — it
searches indexed files, so something created moments ago may be missing.

**Before editing an unfamiliar file**, `ken_file_findings(path)` — what past sessions
learned here. If the change isn't local: `ken_blast_radius` (what it breaks),
`ken_cochange` (what changes with it that imports don't show), `ken_find_tests`.

**Before finishing, write back — the step agents skip, and the reason ken stops
improving.** Learned something non-obvious that cost real effort (a root cause, a
constraint, where a subsystem actually lives)? `ken_remember(topic, content)`, so the
next session starts where this one ended. Durable facts, not a session log; same
topic overwrites.

The shell commands (ken rank, ken search-files, ken search-symbols, and ken explain) expose the same ideas for humans or agents without MCP, and ken tools <name> runs any MCP tool directly (see Run MCP tools from the shell). For assistants with MCP available, the MCP tools are the preferred path because they return structured results and feed ken's local task memory.

Use ken directly

The hooks run automatically when the assistant is active, but the CLI is useful for checking what ken sees:

ken status .
ken rank "where is codex install wiring handled"
ken rank --verbose 2 "how does predictive ranking work"
ken explain "why did src/ken/cli.py appear"
ken search-files "semantic file retrieval"
ken search-symbols "merge codex hooks"

You can save and recall project-specific findings:

ken remember "codex wiring" "Use ken install --codex . to repair invalid hooks."
ken recall "codex hook repair"

Run MCP tools from the shell

Every tool the ken MCP server exposes is also runnable directly with ken tools, so you can use the structured code-intelligence tools (call graph, blast radius, co-change, wiring, clones, …) without an assistant in the loop. The list, descriptions, and parameters are read live from the same MCP surface, so ken tools never drifts from what the agent sees.

ken tools                                   # list every tool with a one-line summary
ken tools grep --help                       # show one tool's parameters
ken tools grep "MY_ENV_VAR" --mode bm25     # required params are positional, options are --flags
ken tools blast_radius src/ken/cli.py
ken tools file_symbols src/ken/search.py --no-include-docstrings

The tool name may be given with or without the ken_ prefix (grep or ken_grep). Results print as JSON (--compact for a single line). Point at another checkout with ken tools --path /repo <name> ... (the flag comes before the tool name).

Codex hook setup

After ken install --codex ., start Codex in the project and run /hooks to enable the project hooks. Codex only loads project-local hooks after the project is trusted, so also approve the trust prompt when Codex asks.

You can mark the project as trusted manually in ~/.codex/config.toml:

[projects."/abs/path/to/my-project"]
trust_level = "trusted"

OpenCode setup

After ken install --opencode ., OpenCode picks up the ken MCP server automatically the next time it starts in the project — no extra step on your side. OpenCode reads opencode.json / opencode.jsonc straight from the project root and merges ken's block with whatever you already have (model, providers, commands, sibling MCP servers, all preserved). Run opencode mcp list to confirm the server is registered.

If you wired multiple assistants into the same project, the OpenCode MCP entry lives alongside the Claude and Codex hooks with no interaction between them.

Uninstall

ken uninstall .
ken uninstall --keep-db .

ken uninstall . removes hooks, MCP entries, and the local .ken/ index. Use --keep-db if you want to keep .ken/ken.db for later.

License

ken is released under the MIT License. See LICENSE.