rgapi

rgapi provides fd-style file discovery and rg-style text search from Python without starting a shell command.

It uses ripgrep's ignore and grep-searcher crates for walking and file scanning, with the pure-Rust fancy-regex engine for regex matching. Walking and searching run in parallel by default. Most expensive work stays in Rust.

Overview

For common file discovery and search:

from rgapi import fd, ls, rg, rg_iter

fd(ext="py", exclude="test_*.py")
ls("src")
for row in rg_iter("TODO", ".", include="*.py", context=2): print(row.asdict())
rg("TODO", ".", ext="py", skip_dir=".venv", paths=True)
rg("TODO", ["src", "tests"], ext="py")

For cell-aware search of Jupyter notebooks (see Notebooks):

from rgapi import nbrg

nbrg("read_csv", ".", cell_context=1)

Walk and search functions have async versions. Streaming versions yield results as the search finds them. See Async:

from rgapi import fda, rga, rga_iter, nbrga, nbrga_iter

await rga("TODO", ".", ext="py", timeout_ms=200)
async for row in rga_iter("TODO", "."): print(row)

Use the lower-level functions to compile a regex, or to search text or a single file:

from rgapi import compile, search_path, search_text

matcher = compile("TODO")
matcher.is_match("TODO")
matcher.finditer("TODO TODO")

search_text(matcher, "alpha\nTODO\nomega\n", path="memory.txt", context=1)
search_path(matcher, "src/lib.rs", display_path="src/lib.rs")

Install

pip install rgapi

File discovery

fd returns absolute pathlib.Path objects. Use them directly with .read_text(), .open(), .stat(), or other filesystem operations. Its collected results display names relative to root. Pass root as a str or Path, or as a list of them. The sync and async APIs expand ~ and accept ., ./, and paths containing ...

A list of roots, such as rg("TODO", ["src", "tests"]), works with every walk and search function. The roots are searched as one walk that returns one result list. timeout_ms and max_results apply to the whole walk. Result paths are relative to the common ancestor of the roots. For example, rows read src/app.py and tests/test_app.py. PathResults displays use the same relative paths. Filters such as include, path_re and skip_dir match these relative paths. A file under more than one root, such as src/app.py with roots [".", "src"], appears once.

Discovery uses the ignore crate with ripgrep's default filters. It reads .gitignore, .ignore, and .rgignore files. .rgignore takes precedence over .gitignore. Pass ignore=False to disable all ignore-file filtering, including .rgignore.

Hidden files are skipped unless hidden=True. Discovery returns symlinks themselves, including explicitly named roots and dangling links. Set follow_links=True to traverse their targets. Paths retain the symlink spelling. Use same_file_system=True to avoid crossing filesystem boundaries.

Traversal runs in parallel without guaranteed result order. Use sorted(...) when order matters.

fd takes pattern first and root second, as rg does. With no pattern, it returns every file under root. pattern is a smart-case regex matched against each basename. Lowercase patterns match case-insensitively. A pattern containing uppercase letters is case-sensitive. Use path_re to match the slash-separated relative path instead.

include and exclude use case-sensitive glob syntax. glob= is an alias for include=. Patterns without / match names at any depth, so *.py matches src/app.py. Patterns containing / are root-relative. * stays within a path component; ** spans directories: src/* matches immediate children, whereas src/** also matches deeper descendants. Matching a directory with exclude prunes its entire subtree. Includes never prune traversal. Excludes always win, and includes do not override ignore rules.

Filter extensions with ext="py" or ext=["py", "rs"]. Extension and glob filters must both match. For example, include="src/*", ext="py" requires src/* and *.py, like combining rg -g with -t.

Set min_depth and max_depth to bound recursion. max_filesize skips files above a byte limit.

ls follows the shell command's listing conventions. It takes root first, as the shell's ls DIR does. It uses fd with max_depth=1, includes directories, disables ignore rules, and sorts by name. Set hidden=True for ls -a behaviour. All fd filters remain available.

fd_iter yields absolute Path objects as the walk finds them. It accepts every fd filter. Stopping iteration ends the walk. It does not accept timeout_ms.

path_re and skip_path_re filter slash-separated relative paths using regexes. They select returned paths or searched files without changing traversal. To skip entire subtrees, use skip_dir with a glob or skip_dir_re with a regex.

Text search

rg and rg_iter return structured SearchLine rows. They accept the same filters as fd: include, exclude, glob, ext, path_re, skip_path_re, skip_dir, skip_dir_re, min_depth, max_depth, max_filesize, follow_links, and same_file_system.

Search is case-sensitive by default, matching rg. Use smart_case=True for rg --smart-case behaviour. Use case_sensitive=False to force case-insensitive matching.

Patterns support backreferences and lookaround everywhere, including filename and path filters:

rg(r"\b(\w+)\s+\1\b", ".")  # repeated words
rg(r"(?<=TODO: )\w+", ".")   # word following TODO:
fd(r"^(\w+)_\1\.py$")  # names such as utils_utils.py

All regex parameters use fancy-regex, with no engine-selection flag and no C library. Ordinary patterns, including word boundaries (\b, \B) outside lookbehinds, use its non-backtracking engine. Patterns requiring backtracking have a step budget that allows lines of millions of characters. A search that exceeds the budget raises ValueError. It never silently skips a file or path.

Each SearchLine has these fields:

kind         'match', 'before', 'after', or 'context'
path         path relative to root
line_number  1-based line number
lnhash       exhash-style `lineno|hash|` address for the line
line         line text without the trailing newline
matches      list of (start, end) byte offsets for match rows

rg, search_text, and search_path return SearchResults by default. This list subclass displays as rg-style multiline text in str() and notebook pretty output. rg_iter yields rows lazily.

SearchLine has a structured repr and an rg-style str. The string display truncates line to 180 characters with a trailing …. repr and asdict() retain the full line. SearchLine.asdict() returns the fields as a plain Python dictionary.

Pass lnhashs=True to rg or rg_iter to display hash addresses instead of line numbers. The line_number field remains available.

For other result forms, use rg(..., paths=True) to return unique matched paths, rg(..., count=True) to count match spans, or rg(..., count="file") for a dict from each file's label to its count, sorted by label. paths and count cannot both be set.

^ and $ match at the start and end of each line. A line ends at \n or \r\n. $ also matches before a \r that has no \n after it. rg, rgstr, and compile reject patterns containing a literal \n or \r. nbrg allows them with multiline=True.

Content searches and compiled content matchers are confined to individual lines: whitespace classes, backreferences and lookaround cannot cross a line ending, and \A/\z refer to each line. With nbrg(..., multiline=True), \A/\z refer to the whole cell, while ^/$ still anchor lines. Filename and path filters match the full name or path instead.

Path results

fd and ls return PathResults, a list of absolute Path objects. So do rg and nbrg with paths=True, and their async equivalents. Indexing or iterating returns ordinary Paths:

for path in fd(root="src", ext="py"):
    text = path.read_text()
    size = path.stat().st_size

Use .name, .suffix, and .relative_to(root) for path components. .stat() follows symlinks; .lstat() inspects the link itself. .readlink() returns a link's target. Discovery preserves native filenames, including literal backslashes on Unix and non-UTF-8 names on filesystems that support them.

PathResults displays as an ls -l-style listing of at most rgapi.MAX_REPR rows. A final … N more line reports omitted rows. The listing uses .lstat() only on displayed rows. show_target=True adds symlink targets. str(res) returns one root-relative name per line. Slices retain the display root and completion status. list(res) returns the absolute Paths without the custom display.

Structured text and notebook search rows retain root-relative string labels in their path fields. Content searches still follow explicitly named root links. This differs from discovery's default of returning the link itself.

The Rust find and find_iter APIs return native PathBuf values relative to the root, or to the common ancestor of several roots in WalkOptions::roots. For an explicitly named file or unfollowed root link, the result is its basename.

Rust callers can set FindOptions::special_files to also discover FIFOs, sockets, and device nodes, for example to reject unsupported entries during archiving. This defaults to false; normal discovery returns regular files, directories when requested, and symlinks.

Limits and timeouts

Set timeout_ms on rg, fd or ls to stop at a deadline and return the results collected so far. Their async versions accept it too. Results report why the operation stopped:

• stop_reason=None means the result is complete.
• stop_reason="max_results" means max_results truncated the result.
• stop_reason="timeout" means the deadline was reached.

complete is true exactly when stop_reason is None. A count is a plain integer or dict, with no completion flag. Counting can't be combined with timeout_ms.

Deadlines stop result collection and request worker cancellation. Workers cannot interrupt a regex evaluation or filesystem call already in progress; a timeout is not a hard per-match execution limit.

Context lines

before_context, after_context, and context correspond to rg -B, rg -A, and rg -C. Files containing NUL bytes or invalid UTF-8 are skipped.

Block summaries

rg(..., summary=True) returns one row per blank-line-delimited block instead of one row per matching line. Empty and whitespace-only lines delimit blocks. A block containing several matching lines appears once and keeps every matching SearchLine in matches.

rg("TODO", ".", summary=True, context=1, maxlen=120)

The result is BlockResults, a list of SearchBlock objects. Each block has path, block_index, start_line, end_line, start_lnhash, end_lnhash, kind, full source, and matches.

Matches display as path:start-end:source. Context displays as path:start-end-source. With lnhashs=True, the range uses copyable boundary addresses such as path:4|Py|,6|HD|:source. Newline runs display as ¶. maxlen limits the displayed text without changing source or asdict().

In summary mode, before_context, after_context, and context count neighbouring blocks. max_results counts matching blocks and retains their context. summary=True cannot be combined with paths or count. It can be combined with lnhash for copyable block boundaries.

Notebooks

nbrg searches cell source in Jupyter .ipynb files and returns matching cells. Each result identifies the cell by its nbformat cell/message id, which stays stable across edits.

Plain rg searches escaped notebook JSON, including outputs and metadata. Its line numbers refer to that JSON file. nbrg searches the reconstructed cell source and identifies the cell you would edit.

from rgapi import nbrg

nbrg("read_csv", ".")                  # cells whose source matches, across all notebooks under "."
nbrg("read_csv", ".", cell_context=1)  # also include neighbouring cells as context

Notebook discovery, parsing, and matching run together in one parallel Rust pass. Matching uses the same fancy-regex engine as rgapi.rg, including backreferences and lookaround, with the same case_sensitive and smart_case behaviour. nbrg accepts the discovery filters from fd and rg, including include, exclude, glob, hidden, max_depth, and skip_dir.

Matching is line-oriented by default. Use nbrg(r"(foo)\n\1", ".", multiline=True) to match across lines within a cell. Matches never cross cell boundaries.

nbrg returns NbResults, a list of NbCell. Each NbCell has:

path         notebook path relative to root
cell_index   0-based position of the cell in the notebook
cell_id      nbformat cell id (falls back to the cell index for notebooks without ids)
cell_type    'code', 'markdown', or 'raw'
kind         'match' or 'context'
source       full cell source
matches      list of SearchLine rows for the matched lines within the cell

NbCell.asdict() returns these fields as a plain dictionary. Its matches field contains SearchLine dictionaries.

str() and pretty display show one line per cell. Matches use path:cell_id:source. Context uses path:cell_id-source. Newline runs display as ¶.

A matching cell's display starts at its first matched line. Earlier lines are replaced by …[Ln], where n is the matched line's one-based number within the cell. A leading #| directive is retained, as in #| export…[L4]needle here.

maxlen limits displayed source and defaults to 120. The full source remains in source and asdict(). Each cell appears once even when it has multiple matches. All hits remain in matches.

cell_context=N includes the N cells before and after each match as kind="context" rows. Context cells are deduplicated within each notebook.

nbrg_iter yields NbCell rows as notebooks are parsed. For collected results, nbrg accepts these limits and result options:

• max_results returns at most that many cells after sorting by path and cell index.
• count=True returns the number of matching cells.
• timeout_ms applies a deadline with the same stop_reason values as rg.

The parser reads only each cell's id, cell_type, and source. It skips outputs and metadata without loading embedded images or plots. search_nb(pattern, path, ...) searches a single notebook file in the same way.

rgapi-nbrg exposes notebook search without requiring a Python kernel:

rgapi-nbrg 'read_csv' .
rgapi-nbrg 'read_csv' . --cell-context 1
rgapi-nbrg 'read_csv' nbs tests
rgapi-nbrg 'read_csv' nbs --glob '*.ipynb' --max-results 20

Run rgapi-nbrg --help for its discovery, matching, and output options.

Async

fda, rga, and nbrga are async versions of fd, rg, and nbrg. The async generators fda_iter, rga_iter, and nbrga_iter yield rows as the search finds them. Async functions accept the same arguments and return the same types as their synchronous equivalents.

from rgapi import fda, rga, rga_iter

await fda(ext="py")
res = await rga("TODO", ".", timeout_ms=200)
if not res.complete: print(f"partial results: {res.stop_reason}")
async for row in rga_iter("TODO", "."): ...

Walking and searching use Rust threads, without asyncio.to_thread or the event loop's executor. A callback uses loop.call_soon_threadsafe to complete the awaited future or supply results to the generator's queue. The event loop remains unblocked, with normal contextvars behaviour.

Cancellation is checked between rows and files; it does not interrupt a regex evaluation or filesystem call already in progress. This includes timeouts from asyncio.wait_for or asyncio.timeout. It also includes task cancellation, such as starlette cancelling a disconnected client's request.

Wrap an async iterator in contextlib.aclosing when leaving its loop early. break alone delays generator finalization until garbage collection. The context manager provides prompt cleanup:

async with aclosing(rga_iter("TODO", ".")) as it:
    async for row in it:
        if enough(row): break

Use streaming results for incremental display, such as sending batches to a browser as they arrive. Collected results with timeout_ms return what was found before the deadline. Use asyncio.wait_for when a timeout should raise instead.

Benchmarks

tools/bench.py compares the rg CLI with in-process rgapi. Run it against a release build. One run on this machine, using best time from seven repeats: