gist: Indexed Regex Search for a Live Working Tree

Note

The tree tells the truth. The persisted index and the resident daemon can save work, but they cannot invent a file set or return stale content.

If an accelerator cannot prove its answer is safe, gist walks the live tree.

• Overview
• Should I Be Using This?
• Support
• Quickstart
• Keeping the Reflex
  • The Default Move
  • Docs, Code, and Data
  • Choices That Prevent a Wrong Search
• The Search Contract
• Improvements
  • Binary Files
  • Indexed PCRE
  • Compressed Input
  • Sorted Output
  • The Type Registry
  • Hyperlinks
  • Line Buffering
  • Block Buffering
  • Adjacent Product Choices
• Three Execution Paths
  • The Cold Subprocess
  • The Resident Session
  • The In-Process ABI
• The Two Indexes
• Ranked Search
• Evidence
• Prior Art
  • Indexed Neighbors
  • Matching Engines
  • Ranking
  • The Codex Subcommand
  • Outside the Claim
• Package Map
• Build and Test
  • Running One Test
• Provenance

Overview

gist is where our bit-level idea became a tool.

Text is bits, and a required trigram is a small proof that most files cannot match.

Instead of reading them all, gist rules those files out, then checks every survivor against current bytes.

We kept ripgrep's useful mental model because that muscle memory is already embedded in how agents search: pattern, paths, familiar flags, stdout results, and 0/1/2 exit codes. Preserving that familiarity is why we treated parity as a product constraint rather than a loose resemblance.

Then we added three things for the agent loop:

1. a persisted candidate index that can prove most files are irrelevant, from required trigrams plus a crest sidecar for the class-repetition patterns trigrams cannot see;
2. a fail-open resident session that avoids cold startup when the request is eligible; and
3. a bounded, definition-biased ranked view for questions where the best hit matters more than every hit.

gist is powered by irregex, the engine we built this tool on. The regex engines, the trigram index and the crest sieve beside it, the corpus walk, the freshness law, and the ranking math all come from there.

What this repository adds is everything with an opinion about a product: the argv grammar, the resident daemon, distribution, and the parity contract. Read the engine for why it is shaped the way it is; read this for what the binary promises.

That product thesis, the competitive ancestry behind it, and the gates that try to falsify it are separated into CLAIM.md, PRIOR_ART.md, and TESTING.md. This README explains the shipped instrument; the dossier explains why its claims deserve belief.

Should I Be Using This?

• To search a repository from a terminal, with the flags you already know – here. Type gist where you typed rg, and start at Quickstart.
• For similarity, repetition, or "what is this file like" – relate, a separate package. Compression kinship is not a pattern question and this binary does not answer it.
• For the blast radius of a symbol, or where a pasted snippet came from – blast, also a separate package. Both of those need current bytes from two engines at once.
• For a linear-time regex to call from Python, Rust, Go, or C – irregex, which ships the bindings and the header. You want the engine, not a command-line tool wrapped around one.
• To embed the search engine itself in a host process – the C ABI here, which streams match records from the same resident engine the daemon holds.
• For structural, semantic, or hosted multi-repository search – not here at all. See Outside the Claim for who does answer that.

The dividing line is whether your question is an exact one. gist finds the bytes you can name, quickly, over a tree that is changing underneath it.

Everything it does to go fast is an accelerator that is allowed to decline, and nothing it returns was decided by anything other than the file's current bytes.

Support

• Bugs and feature requests go through the issue templates, which ask for the pattern, the tree, the exact command line, and whether an index or a resident session was warm. A search bug without its corpus is a bug nobody can reproduce.
• A place where gist and ripgrep disagree has its own template, parity_gap. Divergence outside the improvements bucket is a defect by definition, so report it as one rather than as a feature request.
• Security vulnerabilities never go in a public issue. See SECURITY.md, which also explains why the threat model here treats the corpus as the attacker.
• irregex, relate, and blast are separate repositories with their own trackers. A wrong match or a wrong file set usually belongs to the engine; the argv grammar, the daemon, and the parity contract belong here.
• CONTRIBUTING.md is the entry point for a change, and CODE_OF_CONDUCT.md governs the conversation around it.

Install

The CLI is the product. Three ways in, and none of them needs a Zig toolchain except the last:

pip install gist-search      # the binary, on PATH — no Python needed to use it

Every release also attaches a plain archive per platform (macOS, Linux, and Windows × x86_64 and arm64) with a SHA256SUMS beside them, for a machine with no Python at all:

curl -LO https://github.com/The-Billy-Company/gist/releases/latest/download/gist-<version>-<target>.tar.gz
tar -xzf gist-<version>-<target>.tar.gz && ./gist index

From source, one command produces the binaries, links them onto PATH, and writes the trigram index — this is also what a contributor runs:

zig build                    # ReleaseFast binaries, PATH link, trigram index

The language bindings are published, and each drives that same binary rather than reimplementing it, so the CLI is a prerequisite for all three — the wheel above carries its own copy, and the other two find one on PATH:

	Install	You write
Python	pip install gist-search	import gist
Rust	cargo add gist-search	use gist::…
Go	go get github.com/The-Billy-Company/gist/bindings/go	import ".../bindings/go/exact"

The bare name gist was taken on both PyPI and crates.io and names there are permanent, so the distribution carries the -search suffix while the identifier you type stays gist — the bs4 / PIL split. Per-language detail is in bindings/python, bindings/rust, and bindings/go.

Quickstart

The canonical form is the one you already type, with no verb and no setup:

gist 'SearchRequest'                 # search from the current directory
gist 'SearchRequest' services -n     # explicit scope, line numbers
gist 'SearchRequest' -l              # matching paths only
gist 'SearchRequest' --rank          # best definitions and uses, default top 20
gist 'foo(?=bar)' -P                 # vendored PCRE2: lookaround/backreferences
gist 'foo(?=bar)' --engine auto      # linear first, PCRE2 only if required
gist 'begin.*end' -U                 # multiline mode
gist 'needle' --no-index             # force a pure live walk

Three lifecycle verbs inspect and refresh what the accelerators hold:

gist status --json                  # versioned index/freshness snapshot
gist index                          # rebuild the persisted candidate index
gist serve [ROOT...]                # run the resident UDS service explicitly

The codex is a separate shelf, for exact literal questions answered without opening source files:

gist codex build
gist codex count 'literal'           # exact corpus-wide occurrence count
gist codex tally 'literal' --top 20  # per-file counts, heaviest first
gist codex status

Everything the binary supports can be asked of the binary rather than of this document:

gist --help
gist --schema                        # machine-readable flags and compatibility
gist --generate man                  # gist(1); also complete-{bash,zsh,fish,powershell}

No index is required. Without one, gist scans the live tree; with a covering index, it automatically skips files that cannot contain the query's required trigrams and verifies every candidate against current bytes.

gist rg … and gist search … are aliases for the same search engine. The canonical form is intentionally verbless.

zig build also links the Vim/Neovim plugin into any editor already installed, so :grep becomes gist and --vimgrep output streams into the quickfix list while the search is still running.

The plugin is a client of this CLI and nothing more. It discovers flags from --schema, file types from --type-list, and index state from gist status --json, so a binary upgrade reaches the editor without a plugin release.

The same install places gist(1) and the shell completions, rendered by cli/primer/ from this face's own flag catalog. man gist answers, and gist -<TAB> offers a menu captioned by what each flag changes.

Every closed value set is baked in, so no tab ever forks a process: 241 file types with their globs, 233 encodings, the engines, sort keys, color postures, and hyperlink aliases.

Keeping the Reflex

Gist has two ergonomic lanes, and the first one is the reflex you already have. The muscle-memory lane lets a person or agent replace rg with gist without stopping to translate the search.

The native lane is for an intent ripgrep does not name: rank the best code hit, force the differential oracle, reuse a warm corpus, or query the compressed codex.

Start in the first lane, and cross over only when the question changes.

• Find matching lines – gist PATTERN [PATH...], exactly where you would have typed rg PATTERN [PATH...].
• Narrow the corpus – the same -t, -T, -g, --iglob, and explicit positional paths, spelled the same way.
• Read only the paper trail, or only the source – --docs, --code, and --data, plus their --no- complements, where ripgrep leaves you hand-assembling a dozen -t names.
• Shape familiar output – -n, -l, -c, -o, -A/-B/-C, and --json, on the same output contract.
• Find the best definition or use – --rank[=N], rather than inspecting ordinary grep output and deciding for yourself.
• Use lookaround or backreferences – -P, or --engine auto to escalate only when the pattern needs it.
• Prove acceleration changed nothing – --no-index, whose answer is the oracle for the indexed path, rather than running another scanner.
• Avoid repeated startup – nothing at all, since eligible searches transparently use the resident session where rg needs an external wrapper or a server.
• Count an exact literal without source-file I/O – gist codex count LITERAL on a clean shelf, rather than scanning the tree.
• Ask what this binary supports – gist --schema, generated from the live flag catalog, rather than prose or remembered flags.
• Read the manual, or tab-complete a flag – gist --generate …, rendered from that same catalog, where rg offers man rg and hand-written completions.

The Default Move

For both humans and coding agents, the shortest correct sequence is six steps.

1. Type the search you already know: gist PATTERN [PATH...].
2. Scope early when you know the neighborhood, since a positional path, -t TYPE, or -g GLOB saves output as well as work.
3. Choose the smallest answer that serves the next step: -q for existence, -l for files, -c for per-file counts, ordinary lines for reading, and --rank when one strong code location matters more than completeness.
4. Stay on the linear engine by default. Use --engine auto when a pattern may need PCRE2, and -P when PCRE2 semantics are the requirement.
5. Let Gist choose acceleration. Reach for --no-index only to debug or prove parity, gist status to inspect freshness, and gist index after a large tree change when you want to re-anchor performance.
6. Read stderr after a miss. Suggestions and budget notices never contaminate stdout, so a person can learn from them while a pipeline keeps rg-shaped bytes.

The aliases gist rg and gist search exist for callers that require a verb, not because they unlock a different engine.

Agents should emit the bare form. It is shorter, canonical, and leaves the pattern in the same argv position as ripgrep.

Docs, Code, and Data

-t answers "which language is this?", and that is the wrong grain for the question anyone actually asks. Nobody wonders whether a file is reStructuredText; they wonder "am I reading the paper trail, or am I reading the implementation?"

So that is its own corpus axis:

gist 'SessionStore' --docs      # only prose: what was written ABOUT it
gist 'SessionStore' --no-docs   # only the implementation and its payload
gist 'retry_budget' --data      # only config: json, yaml, toml, lockfiles
gist 'TODO' --code --no-index   # implementation only, no acceleration

Three genera (docs, code, data) are total and disjoint over every path, so --docs and --no-docs are exact complements and no file can fall through the partition. Repeats union, so --docs --data is either.

Each name is also a type name, so -t docs and -T code mean the same thing and compose with --type-add. The aliases prose, doc, and source resolve too, because a name you guess correctly beats one you have to learn.

code is the leftover, never a recognized set. An unfamiliar extension, a generated blob, or a file with no extension at all lands in code, so the worst a gap in the table can do is show --code one line too many.

A fourth unknown genus excluded from --code would turn that same gap into a silent miss instead. Classification is spelling first and location second, which is why docs/conf.py stays code and CMakeLists.txt is a build recipe rather than prose.

That decision is powered by irregex's genus classifier and the totality argument beneath it. What this repository adds is the flag family and the gates below.

A genus narrows what the walk produced and never un-hides. Unlike -t and -g it will not pull a dotfile or a gitignored leaf back in, because code is the default and an un-hiding genus would surface all of .git/.

The whole thing is daemon-eligible. The selection rides the query_ext frame as a two-byte trailer, so a --docs query answers from the resident session at warm speed, byte-identical to the cold run.

Extend it with --type-add 'docs:notes/**' for one run, or types = ["docs:notes/**"] in .irregex.toml for the whole tree.

No grep-class tool ships this axis. ripgrep has prose-adjacent types and no aggregate over them, and its type globs are basename-only, so a docs/ rule is not expressible there even by hand (ripgrep#3339, open).

The rival is therefore what a person types instead: one -t per prose type, hand-assembled, every time. Against that union, derived at run time from gist --type-list --docs ∩ rg --type-list so it can be neither strawmanned nor left to drift, --docs runs 2.9× faster cold and 21× warm (geomean over the needle slate, with the warm arm running with the answer keep disabled, so it is a search and not a memoized recall).

Speed is the smaller half. A basename glob and a genus disagree about what is prose, and the disagreement is proven on a hermetic tree rather than on this repository, so the numbers are the same on your machine.

The union calls three CMakeLists.txt build recipes prose, because *.txt has no way to say "except this one", and it cannot name two extensionless documents that gist promotes by location and by name.

Over this repository's tracked corpus the two rosters land within one file of each other. That is expected, since the rival is derived from gist's own docs types, and it is why the mechanism is measured where it cannot drift.

Both halves are gated permanently. partition_parity.sh proves the set identities over the live tree on every zig build test, and bench/dominance/partition/ holds the speed floors and the classification contract.

The taxonomy is GitHub Linguist's; the classifier names the two deliberate divergences.

Choices That Prevent a Wrong Search

This section teaches selection, not a second flag registry. The checked-in flag_catalog and gist --schema remain the exhaustive, versioned answer.

• Case and character semantics – -i, -s, and -S are last-wins. Unicode folding, classes, properties, and word boundaries are the default, and --no-unicode or a leading (?-u) deliberately selects byte/ASCII semantics. The fold is simple (C+S), matching ripgrep exactly, so ß is not SS on either tool. Under PCRE2, --pcre2-unicode and --no-pcre2-unicode control that backend separately.
• Literal, word, line, and inverse matching – use -F when punctuation should not become regex, -w for a whole Unicode word, -x for a whole line, and -v for non-matching lines. Multiple intents use repeated -e or a pattern file with -f.
• Hidden and ignored files – -u, -uu, and -uuu progressively disable ignores, add hidden paths, and include binary data. An explicit -g or --iglob include can whitelist an ignored path; a type filter can unhide a matching dotfile but does not override gitignore.
• The -rn trap – recursion is already the default, and ripgrep semantics parse bundled -r as replacement. gist -rn PATTERN therefore means --replace=n, not "recursive with line numbers". Spell -n alone; Gist preserves the behavior for parity but emits a diagnostic.
• Zero is sometimes a real value – explicit -m0 means match nothing and exits 1, where omitting -m means unlimited. Likewise -M0 explicitly disables the long-line cap.
• Stable ordering – the default streams in fast parallel discovery order. Use --sort path|modified|accessed|created or --sortr only when stable global order is part of the consumer's contract.
• Unusual input – -z searches compressed files, and --pre CMD searches a preprocessor's stdout and takes precedence over -z, with the command receiving the path as argv[1] and the file's bytes on stdin, ripgrep's exact contract. -E/--encoding accepts auto, none, or the checked-in WHATWG label set. Unknown labels and failing preprocessors exit 2 rather than looking like empty searches.
• Binary intent – -a treats input as text, while --binary and -uuu search a binary file in full and print every matching line, an improvement over ripgrep's one-line binary summary for a code locator (see Binary Files).
• Machine output – use --json for typed records, -0 for NUL-delimited paths, --null-data for NUL-delimited input records, and explicit sorting when downstream comparison requires deterministic file order.
• Who is reading – most of the human posture is already the default a terminal gets, with matches grouped under a filename title and the rows numbered beneath it, exactly as ripgrep lays them out, while a pipe keeps the path:line: prefix and ripgrep's bytes. -p/--pretty adds the remaining piece (color, unconditionally) and --plain is the opposite pole, the piped posture forced onto a terminal, so an interactive run reproduces the bytes a script would see. Decline either half on its own with --no-heading / -N, or request it into a pipe with --heading / -n.
• How fast it arrives – delivery cadence is separate from all of that. --line-buffered is for a consumer that reacts per line, --block-buffered (with --buffer-size) for one that only wants the bytes cheaply, and --buffer-size=0 when nothing may be held at all. Left alone, a pipe blocks and a terminal streams by line, which is almost always right.
• Agent budgets – prefer --rank, -l, -c, a narrower path, or -m N before lifting the soft output guard. --uncap or GIST_UNCAP=1 is the deliberate escape hatch, and GIST_HINTS=0 mutes guidance without changing results.
• Warm and codex paths – the resident session is an invisible, fail-open accelerator, and unsupported shapes simply stay cold. The codex is different: use it only for exact literal count/tally questions, and treat absence as proven only when gist codex status reports a clean shelf.
• Persisted defaults – a committed .irregex.toml at the tree root declares the corpus (roots, skip, types), while a machine-local $XDG_CONFIG_HOME/gist/preferences (on Windows, %LOCALAPPDATA%\gist\preferences, never the roaming %APPDATA%) holds flag lines and applies only when stdout is an interactive terminal. A pipe, a script, --json, and the daemon never inherit them, nor do they open the file, so a typo in one person's preferences cannot fail anybody else's run. gist config reports the resolved stack, gist config check validates both layers without searching, and gist config init writes a charter prefilled from this machine's GIST_ROOTS / skips.list. --no-config / GIST_NO_CONFIG=1 ignores both.

The Search Contract

The cold runtime's flag_catalog is the source of truth for both argv handling and gist --schema. It separates the public surface into four buckets: exact support, improvements (identical-or-superset results that are strictly better, whether faster, more robust, or better for code search, and never a regression), accepted no-ops, and unknown flags that fail with exit 2.

Where gist differs from ripgrep it is an improvement, or it is a bug; there is no third category. We do not claim every option ripgrep ever shipped.

That claim is measured rather than asserted, against a denominator ripgrep owns. surface.py reads rg's documented flag surface at run time (longs from rg --generate complete-bash, shorts and value grammar from its man page) and compares both binaries byte-for-byte on stdout and exit code.

Measured that way, 186 of 186 documented flags conform: 176 are byte-identical and 10 differ only at a declared boundary whose residual check is re-verified on every run, with 0 rejected and 0 undeclared divergences.

Alongside that sit 411/411 of ripgrep's mined integration cases and 27/27 adverse undo pairs, where a negation must actually undo, on a fixture where the two answers differ.

Both of those denominators are ripgrep's own, which is their ceiling as well as their authority. So a third lane, fuzz.py, generates what nobody curated: a random pattern × flag set × a hostile corpus (invalid UTF-8, NUL bytes, a 4 MiB line, a symlink cycle, an unreadable file, catastrophic-backtracking patterns), demanding byte-identical agreement while measuring crash, hang, and peak RSS.

It is the only lane that still finds anything, and it does: a low-single-digit tail per 6,000 iterations, in corners where ripgrep's own three printers do not agree with each other. An empty match at the end of a file with no final newline is counted by --count-matches, dropped by -o --json, and rendered as the whole line by -o.

That tail is published, not excluded. It is classified by root cause in fuzz_baseline.json and in Layer I of the certificate, ratcheted shrink-only, and a missing fuzz record refuses the mint outright.

The implemented surface includes:

• regular, fixed (-F), smart-case (-S), case-insensitive (-i), whole-word (-w), inverted (-v), and multiple (-e/-f) patterns;
• Unicode-by-default case folding, character classes, properties, and word boundaries, with (?-u) or --no-unicode for byte/ASCII semantics. The fold is simple (Unicode C+S), which is ripgrep's posture rather than a shortfall against it: café ⇄ CAFÉ matches on both and ß ⇄ SS on neither, because full (F) folding is one-to-many and neither engine performs it;
• the linear RE2/Pike engine, vendored PCRE2 10.47 with JIT (-P), and --engine auto escalation;
• native multiline search (-U, --multiline-dotall);
• verbose mode ((?x), and the scoped (?x: … )), so a long pattern can carry the whitespace and # comments that make it readable without leaving the linear-time engine, and a commented pattern still composes with -e and -x;
• the by-value escapes in full - \uHHHH, \u{H..H}, \UHHHHHHHH, \U{H..H}, octal \0oo/\ooo, and \N{NAME} by Unicode name - in atom position and inside […], in byte and Unicode mode, either end of a range;
• path, type, glob, hidden-file, symlink, depth, size, filesystem, and the full .gitignore/.ignore/.rgignore control family;
• context, only-match, count, replacement, heading, column, byte-offset, vimgrep, JSON Lines, null-delimited, sorted, and stats output;
• stdin, UTF BOM detection, the WHATWG encoding label set, preprocessing, and compressed-file search.

Normal results go to stdout. Diagnostics, timing, output-budget notices, and search guidance go to stderr, and GIST_HINTS=0 disables guidance without touching results. Guidance is evidence-backed wherever evidence is affordable rather than derived from the pattern's spelling: when the searched bytes are still in hand a suggestion is withheld unless a probe over them backs it, an empty scoped run asks the index for the file that does hold the string and confirms it before naming it, and the one line a matching run can emit is a note that some branch of an A|B|C appears nowhere in the results. A scope too broad to re-read declines the byte probes and falls back to the syntactic guess, so guidance never becomes a cost the search pays for.

Search exit codes follow ripgrep:

• 0 – at least one match;
• 1 – a clean search with no match;
• 2 – invalid argv, unsupported syntax, an unreadable path, or another search error.

An unknown flag or a pattern rejected by the selected engine is therefore an error, never a convincing empty result.

Improvements

Eight flag groups are not bit-identical to ripgrep, plus two pattern-syntax families, and every one of them is an improvement: identical-or-superset results that are strictly better in behavior, performance, or robustness, never a regression.

This is the only category of divergence. If gist ever disagrees with ripgrep outside this list it is a bug, not a design choice, and gist --schema reports the flag groups under the improvements bucket. The two syntax families are not flags and so have no row there; they are By-Value Escapes and Verbose Mode below.

For an exact, versioned answer about a flag, inspect gist --schema rather than relying on a prose list.

Binary Files

--binary (and -uuu) searches a NUL-bearing file in full. ripgrep prints one opaque line, binary file matches (found "\0" byte around offset N), and stops.

A code locator wants the matches, not a shrug, so gist searches past the NUL and prints every matching line, exactly as -a/--text does.

For the source artifacts that carry a stray NUL, such as minified bundles, checked-in fixtures, and mixed-content files, this is strictly more information.

Indexed PCRE

-P / --pcre2 is the only indexed PCRE search. The vendored PCRE2 JIT backend returns ripgrep's exact -P match set, including lookaround, backreferences, and Unicode properties.

It rides the same trigram prefilter as the linear engine, so PCRE queries skip provable non-candidate files instead of scanning the whole tree. Same answers, fewer bytes read.

The gist-native --rank is linear-only.

Compressed Input

-z / --search-zip decompresses in-process. Results are identical to ripgrep across every codec, verified byte-for-byte in bench/conformance/rgsuite, but gzip, zlib, zstd, and xz decode in-process via std.compress.

That means no gzip -dc fork per file, the single biggest speed edge on compressed corpora. bzip2, lz4, Brotli, lzma, and .Z shell the standard external tool exactly as ripgrep does.

Sorted Output

--sort / --sortr reads in parallel and orders after. The final path/modified/accessed/created order is identical to ripgrep's, which single-threads a sorted run.

created additionally falls back to ctime where the platform has no birth time, so a sort ripgrep cannot perform still succeeds.

The Type Registry

--type-list is a strict superset of ripgrep's. It is sorted and framed exactly like ripgrep's, with ripgrep's rows byte-identical, plus richer definitions and gist-only types.

A caller parsing ripgrep's format parses gist's; it just sees more.

Hyperlinks

--hyperlink / --hyperlink-format makes results clickable when they should be. One axis has three spellings: --hyperlink[=auto|always|never|<alias>|<format>], --no-hyperlink, and ripgrep's --hyperlink-format.

The default is auto, so links appear when a person is reading in a terminal known to render OSC-8 and vanish the moment the bytes are going somewhere else, where ripgrep defaults to none.

The deeper difference is that ripgrep's links are a property of its color layer. By its own help, "hyperlinks are only written when a path is also in the output and colors are enabled".

So a link into a pipe costs --color=always, which also forces color into that pipe, and rg's documented escape hatch (--colors path:none --colors line:none …) still wraps every field in ESC[0m resets. There is no rg invocation that yields clean text plus links, and gist --hyperlink=always is that invocation.

Nor does gist need a path in the output to have something to click. Where rg drops the link entirely when the filename is not printed, as with one explicit file argument, gist anchors the line number instead.

A link is navigation, not paint, and NO_COLOR has no opinion about it. Naming a destination on the command line turns links on, because typing --hyperlink=vscode and getting silence is the mystery this flag exists to prevent.

The standing-preference spelling is GIST_HYPERLINK, which may carry a destination alone, leaving the probe to decide, or a WHEN,WHERE pair like always,vscode.

The format grammar is ripgrep's, so a format rg accepts gist accepts and one it rejects gist rejects with the same reason, plus aliases rg lacks (zed, windsurf, vscode-remote, cursor-remote).

A link trace lens says on one line why a run linked or didn't, and always says it, naming the posture (turned off), the reader (output is a byte protocol, machine-shaped output), or the terminal (stdout is not a terminal, terminal does not advertise OSC-8), because a diagnostic that goes quiet reads as "nothing to report". Lighting the lens is enough to keep a run off the warm path, which has no beacon to explain.

Paths fold lexically rather than through a realpath(2) per file, so a click lands in the tree you searched. From /tmp/x gist emits file:///tmp/x where rg emits /private/tmp/x, which resolves outside the workspace folder your editor has open.

Every shape that prints a filename is clickable: match rows, headings, -l/--files lists sorted or not, -c counts, the binary notice, and the --rank view, whose whole point is that its top row is the one to open.

Two shapes refuse every posture, including always: --json records and NUL-framed -0 lists, where the filename's bytes are the payload. So does a filename carrying a control byte, where you cannot see where the click target starts and stops, since a newline in a name splits the anchor across two terminal lines outright.

The URL stays exact either way; it is the text between the escapes that is refused, where rg frames those and emits the two-line link.

Linking 93k matches costs ~5 ms (≈60 ns each), because the URL is split once per file into a prebuilt Waypoint and a row only writes the digits. The output cap counts results rather than escapes, so turning links on never costs you a row.

Line Buffering

--line-buffered keeps the same promise for a fraction of the syscalls. Neither implementation ever holds a finished line, and ripgrep's LineWriter also never writes more than one at a time, while gist emits every finished line already in hand in a single write(2).

-n std src/ here is 1.04 MB of results and the same bytes either way: rg -j1 --line-buffered makes 15,782 writes, and gist makes 342.

The boundary is the run's real terminator, so --null-data records flush on NUL, where rg's line writer only knows \n and holds NUL-delimited output until its buffer fills.

Block Buffering

--block-buffered ramps the block, and the ceiling is one you can name. The first fragment leaves immediately and the threshold then doubles to the ceiling, so | head -1 answers instantly and a closed pipe is discovered within a kilobyte, while a full dump settles into whole-buffer writes.

On the same run ripgrep makes 342 writes, from its 8 KiB BufWriter that holds the first byte as long as the last, and gist makes 23, or 11 at --buffer-size=1M, a knob ripgrep does not have.

This is gist's default posture into a pipe, and it reaches the reader sooner as well as less often: 5 ms to first byte against ripgrep's 9.

Record Anchors

--null-data searches NUL-delimited records, and a record can hold newlines. So ^ and $ are newline assertions inside one, \z is the record's real end, and a record's trailing newline is content rather than a terminator - it opens the empty line after it like any other.

That is not a house opinion; Python's re refereed it. Split a file on NUL by hand, hand each record to re with re.MULTILINE, and compare: over 322 cells of record-mode -c and -o answers, gist agrees with re on every one and ripgrep disagrees on 13. rg misses a record's own start for ^, because it reads ^ as "after a \n" and a record beginning after a NUL is not a line start to it - ^. finds 7 of the 8 first characters in a three-record fixture. It prints whole records as -o rows for matches it rejected, so ^. yields ef, two bytes, for a pattern that can match one. And it matches nothing at all for \z, whose NUL it keeps in the slice it searches. BSD grep -z agrees with gist about ^.

It is also the faster reading. A record is a sequence of lines whenever the pattern cannot see across one - no consuming class admits a \n, no \A/\z is present - so gist splits at the newlines and every piece goes down the ordinary per-line ladder with its DFA, prefilter, and SIMD kernels intact, instead of the whole-record Pike scan an assertion-bearing wide haystack would otherwise force.

Measured on 50 MB of NUL-delimited records, -c, minimum of 15 rounds, all three tools run back to back inside each round so they meet the same machine. before is this same source with the one switch that decides whether a record is decomposed forced off. Counts are identical everywhere.

Pattern	before	current	ripgrep	vs rg
^zzsentinel	7.3 ms / 11 ms	7.4 ms / 8 ms	36.7 ms / 38 ms	5.0x w, 4.5x c
^étop	6.6 ms / 8 ms	7.4 ms / 8 ms	36.2 ms / 38 ms	4.9x w, 5.0x c
^[a-z]+ [a-z]+ [a-z]+	16.8 ms / 104 ms	8.2 ms / 18 ms	43.6 ms / 44 ms	5.3x w, 2.5x c
^\w+ mid	341.9 ms / 2885 ms	16.2 ms / 50 ms	67.7 ms / 67 ms	4.2x w, 1.3x c
^(?:alpha|beta|gamma)	45.7 ms / 387 ms	11.8 ms / 37 ms	73.2 ms / 72 ms	6.2x w, 1.9x c
mid\ntail (-U)	7.2 ms / 10 ms	7.4 ms / 9 ms	35.9 ms / 36 ms	4.9x w, 3.9x c

Two things worth reading twice. ^\w+ mid cost 2885 ms of CPU before and costs 50 now, a 57x cut, because a \w-led program is exactly what gets no DFA and no accelerator tier when the haystack is wide - it was scanning every record with the Pike VM. And the decomposition is neutral where it should be: the three rows that barely move are the ones a required literal already carried, where splitting buys nothing and is asked to cost nothing.

The alternation is the honest one. It was already 1.6x faster than ripgrep on wall clock before any of this - but on 387 ms of CPU against rg's 72, which is a loss on any laptop doing something else with its cores. Ahead on both axes is the only kind of ahead worth shipping.

By-Value Escapes

You can write a character by its value, whichever spelling you already know: \uHHHH, \u{H..H}, \UHHHHHHHH, \U{H..H}, octal \0oo / \ooo, and \N{NAME} by Unicode name - in atom position and inside […], in byte mode and Unicode mode, and at either end of a range ([\u00ab-\u00bb]).

The interesting part is that this is a superset of both incumbents, because the two disagree and each one's gap is the other's feature. ripgrep has the braced spellings Python's re rejects; re has octal and \N{NAME}, which rg refuses outright - it reads \007 as a backreference, says "backreferences are not supported", and points you at -P. Since each engine refuses what the other accepts, accepting both reinterprets nothing: every pattern rg compiles keeps rg's meaning, and every pattern re compiles keeps re's.

Fifteen spellings over a fixture holding each target character, -o, against a real rg process and Python re in the same script:

Spelling	gist	ripgrep	re
\u00e9, \U0001F4A9	match	match	match
\u{1F4A9}, \U{2603}	match	match	rejects
\N{SNOWMAN}, \N{PILE OF POO}	match	rejects	match
\N{NO-BREAK SPACE}, \N{LATIN SMALL LETTER E WITH ACUTE}	match	rejects	match
\N{NBSP}, \N{ALERT} (NameAliases)	match	rejects	match
\N{CJK UNIFIED IDEOGRAPH-4E00}, \N{HANGUL SYLLABLE GA}	match	rejects	match
\007, \01, \0	match	rejects	match

gist agrees with re on 13 of 15 and the two exceptions are the braced forms re rejects and rg accepts, so gist is the union rather than a third opinion. rg cannot run 11 of the 15 at all. The names are the whole Unicode set, not a table of favorites: NameAliases resolve, and the algorithmic ranges are computed, so all 100k-plus CJK ideographs and every Hangul syllable have their names without shipping a name for each.

Octal needed the one real decision, because \1 is ambiguous and re resolves it by position: inside […] every numeric escape is octal ([\1] is U+0001), while at atom position \1 and \12 are group references, so only a leading 0 or a full three digits commits to octal there. We adopt that rule exactly, and a bare \1 at atom position stays an error - not because it is unparseable but because a group reference is the one construct a linear-time engine cannot honor, and answering it with a literal would be a confident wrong answer.

Speed, on the eight of these that rg can run, over 50 MB, -c, minimum of 15 interleaved rounds, counts identical throughout:

Pattern	gist	ripgrep
\u00e9 (dense)	12.7 ms / 39 ms	56.6 ms / 57 ms	4.5x w, 1.4x c
\u{00e9}	17.5 ms / 40 ms	62.2 ms / 58 ms	3.6x w, 1.4x c
\U0001F4A9 (rare)	8.7 ms / 10 ms	37.3 ms / 38 ms	4.3x w, 3.9x c
\U{2603}	7.5 ms / 9 ms	31.7 ms / 32 ms	4.2x w, 3.7x c
\u00e9top	8.8 ms / 19 ms	36.4 ms / 37 ms	4.1x w, 1.9x c
[\u00e9\u00fc]	9.9 ms / 27 ms	58.3 ms / 59 ms	5.9x w, 2.2x c
\u00e9\w+	13.0 ms / 41 ms	44.9 ms / 45 ms	3.5x w, 1.1x c
^\u00e9	10.2 ms / 33 ms	38.1 ms / 39 ms	3.7x w, 1.2x c

An escape is resolved at parse time into the codepoint it names, so it reaches the same DFA, prefilter, and SIMD kernels a literal does - \u00e9 is é, and nothing downstream can tell which way it was typed. That is the reason there is no slow path to fall back to, and the reason the numbers look like the literal ones: they are the literal ones.

Mechanically this was a collapse rather than an addition. The four positions the grammar can reach a character escape from each carried their own \x-shaped prong, which is exactly why \u was missing from all four at once - there was no single place to add it. They now share one decoder, on the principle that what a character's value is cannot depend on where it was written. Two things genuinely do differ, and they are the decoder's only two parameters. One is positional: whether a bare \1 is octal, which is re's rule and turns on […]. The other is the spelling's own promise about its width, and it is the parameter I got wrong first: \xNN and octal are byte syntax, so (?-u)\xe9 is the raw byte 0xE9, while \x{…} \u \U \N{…} name a character, so (?-u)\u00e9 is that character's UTF-8 sequence - the two bytes 0xC3 0xA9, exactly what (?-u)é is. Disabling Unicode changes what a class, a fold, and a boundary mean; it cannot change what a scalar value is. rg draws the line in the same place, and the first cut of this feature narrowed every spelling to one byte, which silently matched 0xE9 where rg matched the character. The record-mode lane found it (bench/conformance/rgsuite/records.py), and a byte-mode […] now refuses a character it cannot hold rather than matching one byte of it, which is rg's judgment too.

A short counted run stays an error, because \u00 is a typo and reading it as U+0000 would match something nobody wrote; surrogates and values past U+10FFFF are refused, since this engine emits well-formed UTF-8 or nothing.

Verbose Mode

(?x) lets a long pattern be written the way a long pattern wants to be written - whitespace to group it, # comments to say what a clause is for - without leaving the linear-time engine. The scoped (?x: … ) works too, so one branch of an alternation can be commented while the rest stays compact.

ripgrep accepts (?x) as well, so unlike the escapes above this is not a family rg refuses. It is a family where rg's own suite never crosses the mode with the two places verbose is not supposed to reach, and rg is wrong in both:

A pattern may end inside a comment. rg wraps every pattern in (?:…) - including a lone one, so there is no spelling that dodges it - and a comment runs to the next newline, so the ) it appends is swallowed:

$ rg -c '(?x) alpha \s+ \d+  # the count' vrb.txt
rg: regex parse error:
    (?:(?x) alpha \s+ \d+  # the count)
    ^
error: unclosed group

$ gist -c '(?x) alpha \s+ \d+  # the count' vrb.txt
1

gist closes each wrap with a newline instead. Under verbose a newline is both insignificant whitespace and a comment terminator, so it cannot change a pattern's meaning; it only gives the wrapper somewhere to close. That is what makes a commented pattern compose with -e and -x the way an uncommented one already did.

A character class is not trivia. re and PCRE2 both stop applying verbose inside […]: a space there is a member and # is a literal. rg does not, so [a b] is [ab] to it, [ ] is an empty class it rejects, and [#] opens a comment that eats the rest of the class. gist follows re.

Both halves, as -c tallies over the 208-line fixture the lane generates (VERBOSE_LINES in records.py, written out as vrb.txt), against a real rg process and Python re in the same script:

Pattern	gist	ripgrep	re
(?x) alpha \s+ \d+ # the count	1	rejects	1
(?x) beta # one word	101	rejects	101
(?x)#only a comment	208	rejects	208
(?x) [a b]	205	202	205
(?x) [0-9 a-f]	205	204	205
(?x) [ ]	205	rejects	205
(?x) alpha [#] filler	0	rejects	0

Both differences are pinned in bench/conformance/rgsuite/records.py as the rg_wrapper and class_trivia boundaries, and neither is a name someone decided to forgive - each re-proves its own mechanism on every run. rg_wrapper holds only if rg answers the same pattern with a newline appended and then answers what gist did, which is what separates a broken wrapper from a missing grammar. class_trivia holds only if rg answers identically for rg's own claimed reading of the pattern (trivia deleted through the class) and gist equals re. If rg fixes either, the lane fails and the boundary gets deleted rather than refreshed.

Speed is the engine's, not the mode's: verbose changes which bytes are a token, never what a token means, so it is resolved entirely in the parse and the same pattern commented and uncommented scans within a fraction of a percent of itself. Over a frozen 11,902-file / 124 MiB tree, -c, median of 11 interleaved rounds, both tools walking the identical file set and agreeing byte-for-byte:

Pattern (wall / CPU)	gist	gist --no-index	ripgrep
(?x) WalletService	29.7 ms / 147 ms	262.6 ms / 1149 ms	718.3 ms / 7651 ms
(?x) [0-9a-f]{8}	32.0 ms / 156 ms	248.8 ms / 1452 ms	577.5 ms / 5446 ms
(?x) ^ func \s+ \w+	37.6 ms / 139 ms	283.0 ms / 1455 ms	669.9 ms / 5478 ms
(?x) \b [A-Z] \w{9,} \b	163.9 ms / 692 ms	167.5 ms / 944 ms	249.7 ms / 2458 ms

That is 18-24x wall and 35-52x CPU on the three selective patterns with the index, and 2.3-2.7x wall / 3.8-6.7x CPU with --no-index, which is engine against engine with every byte read. The last row is the floor rather than the headline, and it is the one worth reading: \b [A-Z] \w{9,} \b matches almost everywhere, so there is nothing for an index to skip and both arms converge - 1.5x wall, and 3.6x CPU because rg spends its cores where gist does not need to. The three patterns rg exits 2 on answer in 22-27 ms.

Adjacent Product Choices

Three more choices are not rg-flag divergences, and it is worth saying so.

--mmap, --no-mmap, --dfa-size-limit, and --regex-size-limit are accepted compatibility no-ops.

Agent-facing output has a soft budget of roughly 25k tokens / 100 KiB and a hard 256 MiB ceiling that --uncap or GIST_UNCAP=1 lifts.

--colors restyles one element at a time, in ripgrep's own spec grammar ({type}:none or {type}:{fg|bg|style}:{value}, over path/line/column/match, with named colors, 0-255, and r,g,b). A spec merges into gist's palette the way rg's merge into its own, so naming a hue keeps the default's bold, and match:none unstyles matches while leaving path color alone, the thing --color=never cannot say since it is all-or-nothing. gist renders one SGR sequence per element where rg emits a separate escape per attribute, and paints column numbers only when a spec asks it to; a malformed spec exits 2, as it does under rg.

Three Execution Paths

There are three ways into the matcher, and they are required to give one answer.

The Cold Subprocess

We keep the normal process as the path that can answer every request:

argv → parse → compile → walk → index read-elision → verify → emit

The walk chooses the files, and the index only removes provable non-candidates.

Files changed since the index anchor are read live, and missing coverage simply reduces acceleration. --no-index is the differential oracle for this invariant.

The Resident Session

To stop paying startup costs, gist serve holds corpus bytes and a trigram index behind a per-repository Unix socket. The CLI may auto-spawn it after an eligible cold miss.

The request classifier deliberately keeps the warm surface small. client.zig remains the executable authority; what follows is a readable snapshot.

Warm-eligible shapes are rootless line output (-n / -N allowed), rootless -l / --files-with-matches, rootless --rank[=N], the -F, -i / -s / -S, and -w modifiers, and existence or caps via -q and -m N.

Authoritative-cold shapes are any explicit path including ., stdin or TTY stdout, context, JSON, replace, multiline, PCRE2, globs, invert, and any malformed or unrepresentable flag value.

The wire contract also defines a count mode, but CLI -c keeps ripgrep's per-file layout and stays cold. Warm I/O has a two-second deadline, and GIST_NO_AUTOSERVE=1 disables automatic session startup.

Eligibility is an optimization decision, never a support boundary.

Freshness is fail-closed. macOS kqueue or Linux inotify can narrow the work, but a reconcile barrier decides whether resident bytes are safe.

Doubt, overflow, an index generation change, or a walk error declines the warm answer and returns to the subprocess. The sockets and the lifecycle are this repository's; the engine they hold hot is the ResidentSession invariant.

The In-Process ABI

For embedders, the in-process FFI path is the C ABI (gist_open / gist_search / gist_close), which streams match records from the same error-returning resident engine.

Python uses it when the shared library and optional cffi are available, then falls open to UDS or subprocess. It is another route to the same matcher, not a second implementation.

The request options are the engine's, in contract/engine.toml, while transports and session rules are this repository's, in contract/surface.toml.

The Two Indexes

We use two indexes because they answer different questions. The ordinary trigram index is a candidate filter: small, mmap-backed, fast, and never authoritative, because every candidate it admits still gets its current bytes checked.

That is the index behind normal regex search. Riding beside it is the crest sidecar, which prunes the literal-free class repetitions ([0-9a-f]{12}, [0-9]{6}) that extract no trigram at all and that every index in this family therefore concedes whole.

Both filters only ever skip reads, and caseless queries, changed files, and a missing sidecar all fall back to reading. Both are powered by irregex: the persisted index family, and the forced-run theorem the sidecar is sound by.

The codex shelf is a compressed self-index for exact literal questions. It can count in O(pattern length), locate occurrences, recover the indexed corpus, and answer without opening source files.

gist codex count is a proof of absence only when the shelf's freshness report is clean, and the command reports files changed since the shelf was built rather than hiding that qualification. See irregex/src/kernel/codex (FM-index math) and irregex/src/corpus/index/shelf (persisted SHLF); the quotation parse over that index is relate/src/kernel/codex (cento).

Ranked Search

Sometimes we need the best hit, not every hit. --rank[=N] keeps the same pattern and path semantics and changes only the shape of the answer: the definition outranks its two hundred call sites, and generated files sink below authored ones.

The view is powered by irregex's rank fusion and the signals it fuses. What this repository adds is the flag, a default top K of 20, and the row:

1. path:line  [def|use|gen|mirror]  ×count  source line

This is heuristic text ranking, not name resolution or semantic code intelligence. It works from the persisted index when possible and has a live walk fallback, and --rank is limited to the linear engine.

Evidence

The idea is ours; the expected answers are not. The ripgrep muscle-memory promise is why we compare gist with a live rg oracle instead of writing expectations by hand.

The gates cover parallel and serial walks, indexed versus --no-index, freshness, line framing, Unicode, multiline and PCRE2 modes, ordering and ignore flags, encodings, preprocessing, compressed input, binary handling, streams, and resident-versus-cold answers.

The tracked ripgrep 15.2.0 snapshot contains 446 invocations per walk engine:

• Mined upstream suite – 411 PASS, 0 ORDER, 0 FAIL, 14 NA, and 21 SKIP. Supported-surface parity is 411/411 = 100%, so every supported-surface case matches ripgrep, with zero deferred divergences.
• Multiline – 30/30 adversarial cases pass for stdout, exit code, and indexed-versus---no-index equality.
• PCRE2 – 30/30 adversarial cases pass the same three-way oracle, including lookaround, backreferences, Unicode toggles, and resource-limit failures.
• Walk, ignore, and message flags – 39/39 cases pass on each engine. The fixtures make path/time ordering, last-wins negations, worker counts, device boundaries, and global git-ignore state observable. The --no-messages / --no-ignore-messages cases live here rather than in the mined suite because rg's own --no-messages tests assert on the exit code, which a gist that merely rejected the flag would also satisfy; these assert the real property, that stderr goes empty while stdout and the exit class do not move, and pin the nesting asymmetry with both lanes firing at once.
• Content transforms – 22/22 cases pass on each engine across preprocessing, binary input, legacy encodings, and the available gzip, bzip2, xz, zstd, lz4, and Brotli decoders.

Every count above shares one denominator ripgrep chose, the tests it wrote and the flags it documents, so each of those 100%s is scoped to cases someone already thought of.

The differential fuzzer is the lane with no such ceiling. It generates invocations nobody wrote down, over corpora built to be hostile, and it is the only one that still finds anything.

It does, a handful per 6,000 iterations, and that tail is published per root-cause class in fuzz_baseline.json and in Layer I of the certificate rather than left out of the scoreboard. A missing fuzz record refuses the mint outright, and the tail is ratcheted shrink-only, so it can fall but never quietly grow.

Parallel and serial results are reported separately because they share a contract but not an implementation path; they are not added together to inflate the case count.

NA is a deliberate product boundary. SKIP is an accounted companion, boundary, or irreplayable obligation. Neither is called a pass, and with zero FAIL rows the strict check_results.py gate is green without --allow-fail.

Reproduce the cited results from bench/conformance/rgsuite:

python3 run.py
python3 modes.py run --mode multiline
python3 modes.py run --mode pcre
python3 flags.py run
python3 transforms.py run
python3 fuzz.py --iterations 6000 --seed 20260727   # the residual lane

The permanent integration order is documented in bench/conformance/gates: correctness gates run before performance gates, so a faster wrong answer cannot earn a benchmark win.

Performance claims come from the committed fail-closed certificate: fresh processes, 20 measured runs after three warmups, bootstrap 95% confidence intervals on medians, and a Mann–Whitney test. A win requires both a lower median and p < 0.05.

On its recorded 20,492-file / 195.8 MiB corpus, gist beat ripgrep in all 12 query classes by 2.10×–7.76×. Those are measurements from the macroscopic end-to-end layer, not universal constants.

The separately minted lower-bound layer covers a 20,696-file / 199.6 MiB corpus, and those single-thread kernel numbers must not be attached to the end-to-end speedups.

The full data, machine description, losses against other indexed tools, and rerun procedure live with the published receipts in bench/certificate/artifact/.

The layers that bound the engine rather than the product, meaning the µarch budget, the memory roof, the candidate-byte floor, and the crest rung, are minted in irregex's own harness.

Prior Art

Most of the pieces are borrowed and cited. We joined them for one specific job: searching a local, constantly changing tree over and over for coding agents.

The positive product case and precise composition claim live in CLAIM.md. The contribution is that measured composition and the contract around it.

The ancestry of the machinery is documented where the machinery lives. Kleene and Thompson through the Pike VM and RE2, PCRE2 for what the linear lane cannot express, Cox's trigram index and the crest sieve that closes its one blind spot, and the Reciprocal Rank Fusion the ranked view is built from are the lineage of the engine gist is powered by, and that lineage is argued in irregex.

What follows is the ancestry of the product: the tools somebody would reach for instead of this one.

Indexed Neighbors

Zoekt is the closest production indexed code-search comparison, with positional trigrams, regex planning, ranking, mmapable shards, and a serving layer.

GitHub's Blackbird extends the same family with sparse variable-length n-grams and global-scale sharding. Gist claims neither distributed search nor organization-wide repository synchronization.

Microsoft tgrep is the nearest public local-agent shape: a persistent trigram index, file watching, client/server operation, and a grep-like CLI.

Gist's distinguishing contract is narrower. Accelerators may decline, while a current-tree subprocess remains capable of answering every supported request.

Matching Engines

The linear lane descends from Thompson's Regular Expression Search Algorithm (CACM 1968), the Pike VM, Cox's Regular Expression Matching Can Be Simple And Fast, and RE2. Unicode range compilation follows the Thompson/Cox UTF-8 decomposition used by RE2 and rust-regex.

Complex constructs use the vendored PCRE2 10.47 engine with JIT and resource caps.

Gist does not claim to make backtracking expressions linear. -P deliberately selects PCRE2 semantics, while --engine auto keeps the linear engine whenever it can express the pattern.

Ranking

The bounded result view uses weighted Reciprocal Rank Fusion from Cormack, Clarke, and Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods (SIGIR 2009). Its inputs are language-agnostic text and path signals.

A declaration-shaped boost is not a symbol table. gist does not resolve types, references, overloads, or call graphs, and it is not an LSP, SCIP, or semantic-retrieval engine.

The Codex Subcommand

gist codex is a thin lifecycle face over the shared compressed self-index (count / find / shelf status).

The Shannon–Manzini / FM-index bibliography and novelty framing live with relate, in relate/research/relate/PRIOR_ART.md § Corpus quotation. The index itself is irregex/src/kernel/codex + irregex/src/corpus/index/shelf; the quotation parse over it is relate/src/kernel/codex.

Outside the Claim

We keep the boundary sharp. Gist is not structural search (Semgrep, ast-grep, Comby), a format-preserving transformation system (OpenRewrite), semantic code intelligence (LSP/SCIP), or a hosted multi-repository platform (Sourcegraph/GitHub Code Search).

It is the exact/regex leg those systems and agents can compose with.

The full landscape, covering unindexed peers, indexed neighbors, matcher/ranking ancestry, and semantic/structural systems, lives in PRIOR_ART.md. The positive product thesis lives in CLAIM.md, and the exact evidence inventory and known losses live in TESTING.md.

Codex / Shannon–Manzini literature stays with Relate in relate/research/relate/PRIOR_ART.md. Where prose lags implementation, gist --schema, the live differential harness, and the committed certificate are authoritative.

Package Map

This repository is the product chassis, and it ships the binary. gist is the indexed, rg-parity search, powered by irregex, where the engines, the index, the corpus walk, the flag grammar, and the warm resident core live.

What is here is everything with an opinion about the product:

• src/exec/session/conduit/ – the daemon wire: protocol, spawn, vigil.
• src/exec/session/daemon/ – the resident session proper: the socket server, request routing, and the client. The answer keep it serves lives in the library's warm core.
• src/exec/session/warden/ – rationing and standdown, so the daemon never taxes the machine it serves.
• src/surface/cli/ – the --generate primer (man page + shell completions). The shared face vocabulary — flags, manifest, grade, reprise — lives in relate.
• src/surface/face/gist/ – the binary face itself.
• src/surface/ffi/ + include/gist.h – the session C ABI (libgist.{a,dylib,so}), which links libirgx for the substrate symbols.
• bindings/ – Go (cgo), Python (cffi), and Rust consumers of that ABI.
• editor/vim/ – the Vim/Neovim plugin (:grep-as-gist, streamed quickfix, :GistRank, :GistBlast).
• shell/ – the generated man page and bash/zsh/fish/pwsh completions, minted from the same flag table argv is parsed with.
• bench/ – the vs-ripgrep dominance certificate and the ratio gates that keep it honest.

Build and Test

Four steps cover everything the package builds:

zig build             # gist binary + libgist → zig-out/ (libirgx installed
                      # beside it, from the irregex dependency)
zig build test        # the unit suite
zig build check       # compile-only
zig build coverage    # per-function coverage

The binaries default to ReleaseFast regardless of the build's own optimize mode, which -Dcli-optimize overrides. The test binary stays ReleaseSafe, so the suite that tries to break the checks keeps them.

Dev model is sibling checkouts. build.zig.zon path-deps on ../irregex and ../relate, and releases pin url + hash; a consuming monorepo may wrap zig build to symlink the binaries onto PATH.

Running One Test

-Dtest-filter=<substring> narrows the suite and -Dtest-shards=1 puts it back into one process. The harness is brigade.zig, which this package takes from the irregex dependency rather than owning, so the trap below is the same one that repository documents at more length. It is restated here because you will hit it here, running these tests.

The trap is that zig build test caches the test run, and the environment is part of the cache key.

The filter reaches the harness as BRIGADE_FILTER, an environment variable set on the run step, and Zig hashes a run step's environment along with its argv.

First run under a given environment executes. Every later run under an environment already used is served from cache, so the step is skipped, nothing executes, and it exits 0 in about the time a no-op build takes (~0.3 s here).

A cache hit still reports a test count, which is what makes it dangerous. --summary all prints 1/1 tests passed either way, and the only token that distinguishes them is cached against success <n>ms:

+- test shard 0/1 success 3ms     # ran
+- test shard 0/1 cached          # did NOT run, still "1/1 tests passed"

So zig build test cannot answer whether the tree is sensitive to an environment variable. The natural probe runs with the variable, then without it to confirm, and the confirming leg revisits an environment it has already seen, making it a replay that is green by construction.

To probe an environment variable, drive the compiled binary directly, since it has no build-cache layer and executes every time:

env FORCE=$RANDOM zig build test -Dtest-filter='<name>' -Dtest-shards=1 --verbose
#   ... BRIGADE_SHARD=0/1 BRIGADE_FILTER=<name> ./.zig-cache/o/<hash>/test

BRIGADE_SHARD=0/1 BRIGADE_FILTER='<name>' BRIGADE_TIMES=1 \
  ./.zig-cache/o/<hash>/test

BRIGADE_TIMES=1 prints one line per test, which is the evidence a run happened. A filter matching nothing fails loudly rather than passing empty, so a stale filter cannot read as a clean run.

Provenance

Extracted from a private monorepo kernel package, cut at ce430bbaab.

The cut line is ripgrep's. What rg-the-binary owns, meaning the daemon, the product vocabulary, distribution, and the certificate, lives here; what the grep-* crates own, meaning engines, walker, index, and argv, lives in the library.

Architecture is machine-checked by charter.zone. Apache-2.0; nothing third-party is bundled here, and the certificate measures competitors by invoking installed binaries.