abstract_search

Standalone string search across files and directories with allow/exclude filtering. Stdlib-only at import time — pdf, office, and spreadsheet readers load lazily and are optional extras.

from abstract_search import findContent, get_files_and_dirs, last_report

# 0.1.0: NO implicit filtering — searches EVERY readable file under the roots
hits = findContent("/path/to/project", strings=["needle"])

# opt into the old conveniences with a preset
hits = findContent("/path/to/project", strings=["needle"], preset="code")

# narrow explicitly; exclude a dir (exact segment, pruned)
hits = findContent("/path/to/project", strings=["needle"],
                   allowed_exts="py", exclude_dirs="tests")

# get the report inline (or call last_report())
res, rep = findContent("/path/to/project", strings=["needle"], report=True)
rep["effective_filters"]; rep["skipped"]

NO implicit filtering (0.1.0). With no filter kwargs you get EVERYTHING under the roots — dotfiles, node_modules, .venv, binaries. There is no default allow-whitelist, no default excluded dirs, no default hidden-file skipping. define_defaults / get_file_filters reflect only user input + chosen presets.

Opt in explicitly with presets — preset="code" (CODE_TEXT_EXTS + EXCLUDE_NOISE_DIRS, skip hidden), preset="noise" (EXCLUDE_NOISE_DIRS + BINARY_EXTS, skip hidden), preset="text" — or pass the exported sets CODE_TEXT_EXTS / EXCLUDE_NOISE_DIRS / BINARY_EXTS.

Preset merge = UNION (never be surprised): presets and your kwargs are unioned field-by-field. Good for excludes; but the allow-whitelist is unioned too, so a whitelist preset broadens — preset="code" + allowed_exts=[".rs"] matches code/text AND .rs, not just .rs. To narrow by extension use preset="noise" (excludes only) + your allowed_exts, or no preset.

When you do filter: exclude wins, include narrows, dir excludes are exact path segments (pruned), globs match path AND basename, everything case-insensitive. Content search skips binary files but reports them — abstract_search.engine.last_report() (or report=True) carries the exact effective_filters and skipped counts (binary/unreadable/too_large/ symlink_loop). The reader surface (search_files/read_paths, WalkConfig) follows the same rule: DEFAULT_WALK is empty (searches everything); pass preset=/filters to narrow; its report adds extract_failed for pdf/office read errors. Remote/SSH search was removed in 0.1.0.

Search engine v2 (2026-09-24)

The search core is an isolated, stdlib-only engine in abstract_search/engine.py (a pruned os.scandir walk that replaced the old find-subprocess design). The public modules (filters, find_collect, find_content) are thin compat layers over it. Identify the live engine at runtime:

import abstract_search
abstract_search.ENGINE      # "abstract_search.engine v2 (2026-09-24)"

A byte-for-byte sibling copy lives at hugpy_tools.search and is kept in step (abstract_* never imports hugpy_*).

Include / exclude semantics (now correct)

• Exclude always wins; include narrows. No include ⇒ everything not excluded.
• Directory excludes match an exact path segment and are pruned at walk time (an excluded tree is never entered) — build drops .../build/... but not .../rebuild/....
• Globs (allowed_patterns / exclude_patterns) match the root-relative path AND basename, so **/mct/*.py and *session* work; exact names (session.py) match exactly.
• Case: names/globs/extensions are case-insensitive; literal content search is case-insensitive unless case_sensitive=True, and matches inside comments and URLs (the old normalizer dropped those).

get_files_and_dirs(*args, recursive=True, include_files=True, **kwargs) is the most-used entry point; its (dirs, files) shape and kwargs are unchanged. See its docstring for the full include/exclude contract and examples.

Speed (best-of-2, 2026-09-24)

Count differences are bug fixes: v2 finds the __init__.py files the old default dropped (91 / 818), and old returned .venv / site-packages / hidden-backup files (2235 on /srv/pyit/dev) that v2 prunes. v2's enumerate counts match the hugpy_tools.search sibling exactly.

from abstract_search import make_allowed_predicate, filter_allowed_items, search_files

allowed = make_allowed_predicate(exclude_exts=".log", add=True)
kept = filter_allowed_items(paths, allowed_exts=[".py", ".md"])
hits = search_files("needle", "/path/to/project")   # line-level SearchHit records

Optional readers: pip install abstract_search[pdf] (pdfplumber), [ocr], [xls], [xlsb], [dbf], [parquet], or [all].