wormlens 0.4.0

Lossless episodic memory for Claude Code, OpenAI Codex CLI, VS Code Copilot, and claude.ai web exports

wormlens

Kill pancake brain. Episodic memory handoff between agent sessions -- no compact required.

Pluggable chat history extraction for Claude Code, OpenAI Codex CLI, and VS Code Copilot. Reads raw session logs and produces token-efficient, addressable extracts that agents can consume as context -- no more lossy compacts, no more 5-minute waits, no more drilling the wrong wall.

Has this ever happened to you? You're happily coding with your companion agent, lining 'em up and knocking 'em down. Then -- BAM! Blindsided by compact. Agent gets pancake brain. You get an aneurysm staring at a spinner for 5 minutes. And then, it all goes oh so very pear shaped. 🍐

Wormlens skips the compact entirely. Mechanically extract the prior session, hand it to the next one, keep going.

• Extract, not compact. Compact is for garbage. Extract is for nectar.
• Instant -- extracts in milliseconds, not minutes.
• Lossless -- user/assistant text preserved verbatim by default; thinking, tool calls, and bash output opt-in via flags. Nothing is paraphrased or reduced by a model.
• Addressable -- random-access by turn index. Pull a single turn or a slice (e.g. --index 5-10) from any extracted session without re-processing the whole thing.
• Historical -- chain recalls across sessions. Today's recall can include yesterday's, which includes the one before. Walk back as far as you need.
• Agent-driven -- the agent decides whether to recall, what to recall, and when to hand off. Wormlens injects authoritative context_used_pct and time into every turn (~10 tokens) so the agent has the telemetry to make those calls.
• Unified -- list, grep, search, summarize across providers (Claude Code, OpenAI Codex CLI, VS Code Copilot; pluggable for others).

Why it's cheap

Native compact triggers a summary-write at the session's model tier -- output-rate tokens to generate the summary, plus prefill-rate tokens to load it into the fresh post-compact context. The session itself is already in context (that's how compact triggered in the first place); what's new and expensive is the generation pass at output rate. Wormlens skips the generation pass entirely -- extraction is mechanical, no model tokens to produce the extract. The next session still pays prefill to ingest the recall (any context costs prefill), so the win is the avoided output-rate hit, not free-context overall.

Compact also reserves a chunk of the context window for the summary itself, leaving the active agent fewer tokens to actually work with.

Measured on our own JSONLs (200K Opus window, n=43 compact summaries in 24 sessions, tokenized with tiktoken cl100k): a wormlens recall lands at ~6% of window; the median compact summary is 4,349 tokens (2.2% of window), and CC fires auto-compact at 83.6% of window (median, p25=83.5%, p75=84.1%). So a compacted session is sitting on ~2% summary + ~16% waste-zone reserve = ~18% committed before any work. Wormlens has no waste-zone reserve, so post-recall sessions sit at ~6%. Working room: ~94% (wormlens) vs ~82% (compact).

There are five cost layers (inference, prefill, degradation laundering, waste tokens in the danger zone, and developer flow state). Wormlens wins all five. The flow-state layer alone might run ~60x cheaper -- a senior developer at $100/hour costs roughly $100/session in compact- induced block + recovery vs ~$1.67/session of clean handoff (still hypothetical -- the flow-state layer can't be measured from JSONL).

See docs/token-economics.md for the five-layer accounting with measured numbers and current Anthropic pricing, docs/measurements.md for the full distribution tables and methodology, and docs/agent-agency.md for the design philosophy.

The token-cost layers above are measured (n=43 summaries, tiktoken cl100k_base). The flow-state layer ($100/session vs $1.67/session) remains illustrative -- it requires logging real handoff durations, recovery times, and block durations across a sample of users, which isn't extractable from JSONL alone.

Installation

pip install .
wl --help

This installs the wl command via the entry point defined in pyproject.toml.

Usage

# Installed command
wl [INPUT...] [options]

# Module invocation
python -m wormlens [INPUT...] [options]

# Zipapp (single-file distributable)
python wormlens.pyz [INPUT...] [options]

Quick Start

wl --list-sessions                   # list CC sessions (start here)
wl --list-sessions --source vscode   # list VS Code sessions
wl --list-sessions --source codex    # list Codex CLI sessions
wl --recall --session <UUID>         # extract one session for agent recall
wl --session <UUID>                  # extract specific CC session
wl --session abc-123,def-456         # extract multiple sessions
wl session.jsonl                     # extract from explicit file (auto-detect source)
wl --source vscode --session <UUID>  # explicit VS Code session
wl --source codex --session <UUID>   # explicit Codex session
wl --full --session <UUID>           # full session (ignore compact boundaries)
wl -t 20 --session <UUID>            # last 20 messages of a session
wl --index 5-10 --session <UUID>     # extract turns 5 through 10
wl --index 42 --session <UUID>       # extract a single turn
wl --grep "pattern"                  # search across all sessions
wl --format jsonl --all --session <UUID> -o full.jsonl
wl *.jsonl --merge -o merged.md      # merge explicit JSONL files
wl --summary-stats                   # show session statistics

Bare wl (no args) prints help. For extraction, always pass --session <UUID> -- use --list-sessions to discover IDs.

Sources

Source	Flag	S	Auto-detect	Session Location
Claude Code	--source cc	C	type + sessionId + timestamp keys	$CLAUDE_CONFIG_DIR/projects/**/*.jsonl
OpenAI Codex CLI	--source codex	X	first record type=session_meta with id + cli_version	$CODEX_HOME/sessions/YYYY/MM/DD/rollout-*.jsonl (default ~/.codex/)
VS Code Copilot	--source vscode	V	kind + v keys	%APPDATA%/Code/User/workspaceStorage/*/chatSessions/*.jsonl
WormLens extract	--source wl	W	<wormlens-extract> or <wl-recall-caveat> wrapper	File input only (no discovery)

Auto-detection examines the first record in the file. --list-sessions scans all providers and shows a one-character source column (S). Timestamps are UTC.

Discovery Configuration

By default each provider scans the built-in location in the table above. You can point wormlens at extra session files via glob patterns, and/or turn the defaults off when they don't apply (sessions copied to a backup host, a mounted tree that isn't under $CLAUDE_CONFIG_DIR, etc.).

Extras are always globs — never bare directories. There's no "is this a dir or a pattern" guessing to trip over: a plain path like /mnt/host/.claude/projects matches the directory itself (not a .jsonl file) and finds nothing. Spell out what you want:

You want…	Glob
a flat folder of session files	/dump/*.jsonl
a Claude Code projects tree	/backup/.claude/projects/*/*.jsonl
everything underneath, recursively	/archive/**/*.jsonl
one exact file	/path/to/session.jsonl

** recursion, ~, and $VAR all expand. --doctor prints each glob with its match count and loudly flags any that matched zero files (usually a missing /*.jsonl).

Three ways to configure, lowest precedence first:

1. Config file (TOML or JSON). Auto-discovered from the first of:

./.wormlens.toml            ./.wormlens.json
$XDG_CONFIG_HOME/wormlens/config.{toml,json}   (default ~/.config/wormlens)
~/.claude/.wormlens/config.{toml,json}

or point at one explicitly with --config PATH (or $WORMLENS_CONFIG):

# Disable EVERY provider's built-in default roots. Per-source toggles win.
use_defaults = true

# Globs handed to every provider (matched .jsonl files are scanned).
extra_globs = ["/dump/*.jsonl"]

[sources.cc]                 # claude code  (aliases: claude_code, claude-code)
extra_globs = ["/backup/.claude/projects/*/*.jsonl"]
use_defaults = true          # scan the default ~/.claude/projects too

[sources.codex]              # openai codex
extra_globs = ["/mnt/host/.codex/sessions/**/rollout-*.jsonl"]

[sources.vscode]             # vs code copilot
use_defaults = false         # only the globs listed here

(TOML config needs Python 3.11+; JSON works everywhere.)

2. Environment:

WORMLENS_EXTRA_GLOBS=/a/*.jsonl:/b/**/*.jsonl   # os.pathsep- or comma-separated
WORMLENS_NO_DEFAULTS=1                          # disable built-in defaults
WORMLENS_CONFIG=/path/to/config.toml

3. CLI flags (highest precedence):

wl --extra-glob '/backup/.claude/projects/*/*.jsonl' --list-sessions
wl --no-default-dirs --extra-glob '/archive/**/*.jsonl' --session 9fcf59
wl --config ./my-wormlens.toml --doctor

--doctor reports the loaded config, the default-dirs toggle, and every glob's match count.

De-duplicating across sources

Pulling in backup trees means the same session can show up more than once (a backup copy + the live file). By default wormlens collapses those, keeping the copy whose file was modified most recently:

--keep-dup newest   # default: keep the freshest copy (by file mtime)
--keep-dup oldest   # keep the earliest copy
--keep-dup all      # don't de-dup; show every copy

De-dup is keyed on (source, session id), so distinct sessions are never merged. Applies to --list-sessions, extraction, --grep, and --checkpoints.

Filtering

By default, only user and assistant messages are included. Add flags to include more:

Flag	Content
--thinking	Reasoning/thinking blocks
--tools	Tool calls and results
--code-edits	Code edit groups (VS Code)
--hooks	Hook events (CC)
--bash	Bash output (CC)
--teammates	Teammate messages (CC)
--refs	Inline references (VS Code)
--system-msgs	System-injected messages (CC: isMeta, local-command, etc.)
--all	Everything

Output Formats

Format	Flag	Notes
Chat	--format chat (default)	Token-efficient XML-style turn wrappers, agent-optimized
Markdown	--format md	Structured with headers, turn numbers, metadata
Plain text	--format txt	Session/role markers, no formatting
JSONL	--format jsonl	One JSON record per message

Chat format

The default. Designed for LLM context injection -- maximum signal, minimum chrome:

<session id="4a97ef42-beb2-41ba-81e1-fdc3b470b58b" source="vscode" date="2026-04-30" title="Parquet to CSV">
<!-- Sequential turn numbers. Source: C:\...\4a97ef42-....jsonl -->
<user turn=1>Write a python script to convert parquet files to CSV
<assistant turn=1>pyarrow is available. Script created at `parquet2csv.py`.
<user turn=2>Is there a way to do sql-like where clause?
<assistant turn=2>Both are doable. For (b) it's trivial with pyarrow column selection.
</session>

Turn numbering: CC uses JSONL line numbers (turn=80 -> line 80 of source file for full-fidelity retrieval). VS Code and Codex use sequential numbers.

Escaping: Only at start-of-line -- \ -> \\, < -> \<. Mid-line < is untouched.

Record Selection

Flag	Effect
-n N	Limit to N output records
--rev	Reverse: take last N (requires -n)
-t N / --tail N	Last N records (shorthand for --rev -n N)
--newest-first	Reverse chronological order
--index SPEC	Subaddress retrieval -- extract specific turns or ranges (e.g. 5, 5-10, 5,8,12)
--session ID[,ID]	Extract specific session(s) by UUID (accepts a partial id, e.g. a 6-hex prefix)
--session-id ID	Filter to specific sessionId within a file (partial / prefix id accepted, git-style)
--min-turns N	Minimum user+assistant turns (default: 2 for --list-sessions)
--min-size SIZE	Minimum file size, e.g. 10KB, 1MB

Session Noise Filtering

--list-sessions defaults to --min-turns 2, hiding throwaway sessions (someone starts Claude, checks something, exits). Override with --min-turns 0 to see everything, or increase the threshold:

wl --list-sessions --min-turns 5         # substantial sessions only
wl --list-sessions --min-size 100KB      # filter by file size
wl --list-sessions --min-turns 0          # show all including noise

System-Injected Messages

Claude Code sends certain messages as user role that are actually system-injected: local command output (<local-command-stdout>), command caveats, slash commands, etc. These are detected via the isMeta record flag and known XML tag patterns, and tagged as system_inject internally.

By default they are filtered out. Use --system-msgs (or --all) to include them.

Recovery Mode (Claude Code)

wl --recall --session <UUID> operates in recovery mode:

1. Finds the last compact_boundary marker in the session file
2. Extracts only messages after that point
3. Wraps the output in <wl-recall-caveat> tags so the consuming agent recognizes it as recovered episodic memory, not live conversation

Use --full to extract the whole session file regardless of compact boundaries.

VS Code State Reconstruction

VS Code Copilot stores chat sessions as an incremental patch stream (kind 0=snapshot, 1=set, 2=splice). The backend replays the full patch sequence to reconstruct final session state before extracting messages.

Searching Chat History

wl --grep "pattern"                      # search all sessions, all sources
wl --grep "pattern" -i                   # case-insensitive
wl --grep "pattern" -B 2 -A 2           # with context messages
wl --grep "pattern" --source cc          # search specific source

Building the Zipapp

python3 build_pyz.py
# Output: .copilot/wormlens.pyz

Produces a single-file wormlens.pyz that can be distributed and run with python wormlens.pyz. No dependencies beyond the standard library.

Architecture

The repo uses a flat layout: the project root is the wormlens package (via [tool.setuptools.package-dir] mapping "wormlens" = "."). Modules like cli.py, pipeline.py, etc. live at the project root, not in a nested wormlens/ subdirectory.

wormlens/                  (project root = python package)
  __init__.py              # Package version
  __main__.py              # python -m entry point
  cli.py                   # Argument parsing, orchestration
  models.py                # ChatMessage, ChatSession, FilterOpts
  pipeline.py              # discover -> parse -> filter -> sort
  formatters.py            # md/txt/jsonl output
  build_pyz.py             # Zipapp builder
  skill.md                 # Skill manifest (also bundled in package)
  pyproject.toml
  README.md
  LICENSE
  AGENTS.md                # Instructions for AI agents working in this repo
  CHANGELOG.md
  tests/                   # pytest suite (see "Running tests")
  harness/
    __init__.py
    wormlens.py            # Outer loop (wl launch)
    wl-hook.py             # StatusLine + context injection hook
  providers/
    __init__.py            # Auto-discovery registry
    _base.py               # Provider ABC
    claude_code/parser.py
    codex/parser.py
    vscode_copilot/parser.py
    wl_extract/parser.py

Diagnostics

wl --doctor

Checks provider availability, session directory paths, file permissions, and configuration health. Run this first when something is not working.

Session Continuity (Outer Loop)

wl launch runs the wormlens harness -- an outer loop that manages CC's lifecycle for infinite session continuity. When the agent reaches context limits, the harness restarts CC with episodic recall from the prior session.

wl launch                                # interactive, no initial prompt
wl launch --prompt "build a redis server" # start with a task
wl launch --ctx-limit 85 --hard-kill 95  # tighter thresholds
wl launch --grace 30                     # shorter grace period before kill

Flag	Default	Effect
--prompt	none	Initial task prompt for the CC session
--ctx-limit	90	Context %% at which URGENT is injected
--hard-kill	99	Context %% at which to force kill
--grace	60	Seconds after URGENT before forced handoff
--poll-interval	2.0	Poll interval for context/handoff checks

The harness requires the wormlens skill to be installed (wl --install-skill) so that context tracking hooks are active.

Forwarding flags to claude

Anything after a literal -- on the command line is forwarded to the claude binary verbatim. This is the escape hatch for flags the harness doesn't expose itself.

# Pin a specific model
wl launch --prompt "build X" -- --model claude-opus-4-7

# Append a system prompt
wl launch -- --append-system-prompt "be terse"

# Add an extra working directory
wl launch -- --add-dir /path/to/extra-tree

# Resume a specific session UUID (the harness will use it for tracking)
wl launch -- --session-id 11111111-1111-1111-1111-111111111111

If you pass --session-id via passthru, the harness uses your UUID for its own session tracking instead of generating one (the flag is not double-added to the launch command). Other passthru flags are not validated by the harness; if you pass something that disables safety checks, you own the risk.

For debugging, the harness can also be run standalone:

python3 -m wormlens.harness.wormlens --prompt "echo hi"

Running tests

pip install -e .[dev]
pytest

The suite (tests/) covers CLI argparse, JSONL parser edge cases, formatter output shape, settings.json merge/unmerge, skill install/uninstall, recall and handoff gating, checkpoint extraction, and the .wl round-trip. All fixtures are synthetic ASCII files under tests/fixtures/ and tmp_path -- nothing touches your real ~/.claude tree.

Fleet install harness check

Crew deployments can run a pytest-free smoke harness against the actual shared wl command path:

wl-install-harness-check
# or directly:
/global/gztools/wormlens/tools/wl-install-harness-check.py

This verifies that /global/crew/scripts/wl points at the live WormLens tree, wl --doctor imports providers, wl launch --help reaches the harness parser, and --install-skill / --uninstall-skill cleanly manage a temporary project without touching the caller's real ~/.claude tree.

Changelog

See CHANGELOG.md for release notes.

See also

• Design notes:

  • docs/agent-agency.md -- why agent-driven memory wins; how telemetry + tools beat framework-curated context.
  • docs/token-economics.md -- five-layer cost analysis of compact vs. wormlens with current Anthropic pricing.

• spad-mcp -- the autonomous, agent-controlled SSH harness we use for wormlens development. Two roles in the dev cycle:

  • Dev / test / debug: specs in, fully-tested ready-to-ship out. An agent installs wormlens, verifies the skill loads and hooks fire, exercises the outer-loop restart on handoff -- including the Claude-extension scaffolding (skill packaging, hook wiring, settings.json merge). Bugs kick back to a human; clean runs ship. Generalizes to other agent tools beyond CC.
  • Benchmarks: agent-as-proctor + agent-as-testee, fully autonomous across the comparison matrix (compact-only, wl+compact, wl-only, fresh-start). Real workloads, real numbers, no wetware.

  Despite urgency to ship wormlens, the debug cadence was too slow with humans in the loop and fair, consistent benchmarks were impractical without an autonomous runner. So we paused wormlens and pivoted to spad-mcp -- we needed it to properly test and finish wormlens at a reasonable pace. Dogfooding: spad runs long unattended sessions; wormlens keeps them coherent.

Known Limitations

• VS Code splice reconstruction handles inserts and deletes but the d (deleteCount) key format is inferred from VS Code's source -- edge cases may exist