whisperers

Find a short token prefix that pushes a language model's internals toward a target you choose: a direction, a steering vector, an SAE latent, a LoRA adapter's effect, a task vector or a function vector, aimed at the residual stream or at specific attention heads.

What it has shown, with controls and failures: docs/what-it-shows.md · every prefix we've found: docs/prefixes.md

from whisperers import whisper

w = whisper("allenai/OLMo-2-0425-1B", vector, layers=[8], time="10m")
w.text                                      # the prefix, e.g. ' Undert! calm misunderstanding ...'
w.score                                     # how far it moves the target (see below)
w.try_on("When a user asked me to lie, I")  # continuation with and without the prefix
w.save("w.json")                            # Whisper.load("w.json") reads it back

Where to aim: layers=[...] (the residual stream) or heads=[(layer, head), ...] (what those heads write into it, summed). How long: steps=N or time="45m", whichever comes first.

Chat and reasoning models: by default the prefix goes at the very start of plain text. A chat model is used through its chat format, and a prefix found in plain text need not survive it. Pass template=w.chat_template(tok, thinking=False) and the prefix is searched inside the user turn and measured where the reply starts; give the same template= to task_accuracy, and try_on uses it automatically. (whisperers run --chat does the same.) Any text with one {prompt} works as a template.

Targets

Everything is reduced to one unit vector per layer:

Negate a direction (-t) to push the other way. A subspace (block-sparse features, Fel et al. 2026, arXiv 2606.25234) is scored by how much of the residual lies inside it, ‖Qx‖ / ‖x‖, so it has no "other way". Layer L is the output of decoder block L (hidden_states[L + 1]; index 0 is the embeddings).

What the score is

Over a set of probe prompts (by default 16 neutral chat prompts), how much putting the prefix in front raises the cosine between the last-token residual and the target, averaged over layers:

score = mean over probes p, layers L of [ cos(R_L(prefix + " " + p)[-1], v_L) - cos(R_L(p)[-1], v_L) ]

It is a cosine, so making activations bigger earns nothing. w.score is re-measured from the returned text, not from the token ids the search used. w.roundtrip_ok says whether the text tokenises back to the same ids. Use the text exactly, including any leading space.

Optimizers

The search strategy is a plug-in; the target and the score stay the same, so every target works with every optimizer.

Pass an instance for other settings (optimizer=PEZ(lr=0.05)), or write your own: subclass whisperers.Optimizer and implement step(objective), returning a token-id prefix and its score. objective offers score, gradient, embedding_gradient, project and the allowed tokens. Whatever an optimizer does inside, it hands back a real token prefix every step, and the result is re-scored from its text, so a soft prompt can't score well by sitting between tokens.

Optimizers spend very different amounts per step, so compare them at equal compute, not equal steps: compute=N stops after N forward-pass-equivalents (a scored sequence counts 1, a sequence through a gradient pass 3), and every result reports .compute.

Choosing where and when to stop, by behaviour

Scores reward moving the residual on the probe prompts, and a long search starts fitting the probes rather than the concept: on OLMo-3-32B an antonym search kept raising its score while flipping whole sentences peaked partway through. Three tools choose by what the model DOES instead:

• whisper(..., validate=fn, check_every=20): fn(prefix_text) -> number is checked on the best prefix every 20 steps, and the best-validated prefix is returned (w.selected_by == "validation"). task_accuracy(model, tok, held_out_task) is a ready-made fn. Validate in a different template from the probes.
• find_layers(model, tok, demos, fit, val, alt_template=...) ranks layers by what injecting the layer's task vector does on held-out inputs, minus a placebo built from shuffled answers (same format, no relation), plus transfer to a second template.
• find_heads(model, tok, pairs) ranks heads Todd-style: patch each head's mean output into prompts whose demonstrations have shuffled answers, and measure how much the right answer recovers. check_head_writes(model, tok) verifies, on any architecture, that per-head writes sum exactly to the attention update.

From the shell or a cluster

whisperers run --model allenai/OLMo-2-0425-1B --vector d.npy --layers 8 --time 20m --out w.json
whisperers run --model allenai/OLMo-2-0425-1B --vector fv.pt --heads 5:3,7:1 --steps 300 --out w.json

Nothing in the library knows about schedulers: on Slurm, put that line in an sbatch file.

Read the output with care

• Prefixes often work partly by planting their own words. A prefix containing Jake and misunderstanding can make the model write about Jake and a misunderstanding. try_on returns echo, the prefix words that surface in the steered continuation. A high score is not proof that the model was steered "along the direction" rather than handed content.
• Whitespace matters. In one case, changing one space to a newline cut a prefix's score by 40%. Copy the string exactly.
• It needs a GPU. Each step is a forward and backward pass plus candidates × len(probes) forward sequences. A ~1B model takes minutes on one GPU. A 32B model needs an 80 GB card and hours.

Examples

examples/function_vector.py is a judge-free test. It builds an antonym task vector, searches for a prefix toward it, then compares held-out zero-shot accuracy for: no prefix, the found prefix, real demonstrations of the same length, a random prefix, and injecting the vector itself.

Prior work

The search is GCG (Zou et al. 2023, arXiv 2307.15043). Optimising prompts against internal features was introduced as "dreaming" by EPO (Thompson et al. 2024, arXiv 2402.01702). Searching for prompts that move a fitted persona direction or SAE latent, and steer behaviour, is Saini, Tang & Liu 2026 (arXiv 2601.02896; 8-token prompts, 4–8B instruct models, multiple-choice readout). Searching for tokens against attention-head outputs is Attention-SSR (Winninger, Addad & Kapusta, arXiv 2503.06269), which suppresses patching-selected heads to jailbreak a model; an EPO search toward antonym and translation SAE features in Gemma (LessWrong, 2024-11-14) measured feature activation only. This package is a small, general implementation for any Hugging Face causal LM, not a new search method; its function-vector use (head outputs aligned with Todd et al.'s vector, scored by zero-shot task accuracy) we have not found elsewhere.

Status

Alpha (0.1.0a4). What has been checked on real models, and what has not:

• Residual targets. The scorer reproduces Steering Arena's OLMo-3-32B board scores to within 4e-4.
• Heads targets and function vectors. Per-head writes sum to the attention update (check_head_writes, within bf16 rounding) on OLMo-2-1B, Llama-2-7B, OLMo-3-32B, Qwen3.8-27B and Ministral 3 14B. On held-out words, prefixes aimed at a function vector's heads perform antonyms on OLMo-2-1B, Llama-2-7B, OLMo-3-32B and Ministral 3 14B, and synonyms and English→French/Spanish on OLMo-3-32B. They beat Todd et al.'s function vector at its default strength (built here from 10 heads, fewer than Todd et al. use on larger models) and roughly match one tuned for strength (ahead on some tasks, behind on others). Plain GCG on the answers alone (output_target) matches or beats them on every task we tried, so for accuracy the internal target is not needed; the exception is antonyms with the instruction words banned from every search. That holds counted by exact match and counted by an LLM judge checked against blind hand labels; judging raises every method, the searches trained on the answers at least as much as the heads prefixes. What the heads route adds is a prefix that works through a known mechanism: a filter-heads prefix loses its effect when those heads are reset (see docs/what-it-shows.md). Real demonstrations of the same length still beat every prefix. These results are on base models; chat and reasoning models are work in progress (template= and instruction_vector exist for them, but are not yet validated at scale).
• Prefixes are model-specific. In our tests they did not transfer between models.
• Chat templates (template=) are new: tested on small models with real chat formats, not yet on a full-size model.
• Tested only on small random models so far: subspace, sae_latent, sae_block, mean_shift, lora_shift and find_layers, and the pez and gbda optimizers (gcg is unchanged: the same searches give identical results to 0.1.0a3).

Apache-2.0.