abstract_search

Standalone string search across files and directories with allow/exclude filtering. Stdlib-only at import time — pdf, office, and spreadsheet readers load lazily and are optional extras.

from abstract_search import findContent, get_files_and_dirs, last_report

# NO implicit filtering — searches EVERY readable file under the roots
hits = findContent("/path/to/project", strings=["needle"])

# narrow explicitly; exclude a dir (exact segment, pruned)
hits = findContent("/path/to/project", strings=["needle"],
                   allowed_exts="py", exclude_dirs="tests")

# size/depth/hidden/symlink caps are honoured too (never silently dropped)
hits = findContent("/path/to/project", strings=["needle"],
                   max_bytes=1_000_000, max_depth=5)

# a named preset needs an importer-supplied table (no built-ins):
TABLE = {"code": {"allowed_exts": [".py", ".ts"], "exclude_dirs": ["node_modules"]}}
hits = findContent("/path/to/project", strings=["needle"], preset="code", presets=TABLE)

# get the report inline (or call last_report())
res, rep = findContent("/path/to/project", strings=["needle"], report=True)
rep["effective_filters"]; rep["skipped"]

NO implicit filtering. With no filter kwargs you get EVERYTHING under the roots — dotfiles, node_modules, .venv, binaries. There is no default allow-whitelist, no default excluded dirs, no default hidden-file skipping. define_defaults / get_file_filters reflect only user input + chosen presets.

NO opinionated built-in filters (0.1.4). abstract_search is clean and unbiased: it ships no baked-in preset vocabulary and no default exclude/allow sets. PRESETS is empty and CODE_TEXT_EXTS / EXCLUDE_NOISE_DIRS / BINARY_EXTS are empty. A named preset works only against an importer-supplied table passed as presets={name: {filter kwargs}}; with no table, preset= raises ValueError. Which extensions/dirs/types count as code, noise or binary is the importing party's call, not this engine's.

Every keyword is honoured or rejected — never silently dropped (0.1.4). The full engine filter vocabulary (allowed_exts/exclude_exts, allowed_dirs/exclude_dirs, allowed_patterns/exclude_patterns, allowed_types/exclude_types, exts/dir_names/include_globs/ exclude_globs, max_bytes, max_depth, include_hidden, follow_symlinks) reaches the engine through every entry point; an unrecognised keyword raises TypeError naming it.

Preset merge = UNION (never be surprised): an importer preset and your kwargs are unioned field-by-field. Good for excludes; but the allow-whitelist is unioned too, so a whitelist preset broadens. To narrow by extension use an excludes-only preset + your own allowed_exts, or no preset.

When you do filter: exclude wins, include narrows, dir excludes are exact path segments (pruned), globs match path AND basename, everything case-insensitive. Content search skips binary files but reports them — abstract_search.engine.last_report() (or report=True) carries the exact effective_filters and skipped counts (binary/unreadable/too_large/ symlink_loop). The reader surface (search_files/read_paths, WalkConfig) follows the same rule: DEFAULT_WALK is empty (searches everything); pass preset=/filters to narrow; its report adds extract_failed for pdf/office read errors. Remote/SSH search was removed in 0.1.0.

Search engine v2 (2026-09-24)

The search core is an isolated, stdlib-only engine in abstract_search/engine.py (a pruned os.scandir walk that replaced the old find-subprocess design). The public modules (filters, find_collect, find_content) are thin compat layers over it. Identify the live engine at runtime:

import abstract_search
abstract_search.ENGINE      # "abstract_search.engine v2 (2026-09-24)"

A byte-for-byte sibling copy lives at hugpy_tools.search and is kept in step (abstract_* never imports hugpy_*).

Include / exclude semantics (now correct)

• Exclude always wins; include narrows. No include ⇒ everything not excluded.
• Directory excludes match an exact path segment and are pruned at walk time (an excluded tree is never entered) — build drops .../build/... but not .../rebuild/....
• Globs (allowed_patterns / exclude_patterns) match the root-relative path AND basename, so **/mct/*.py and *session* work; exact names (session.py) match exactly.
• Case: names/globs/extensions are case-insensitive; literal content search is case-insensitive unless case_sensitive=True, and matches inside comments and URLs (the old normalizer dropped those).

get_files_and_dirs(*args, recursive=True, include_files=True, **kwargs) is the most-used entry point; its (dirs, files) shape and kwargs are unchanged. See its docstring for the full include/exclude contract and examples.

Speed (best-of-2, 2026-09-24)

Count differences are bug fixes: v2 finds the __init__.py files the old default dropped (91 / 818), and old returned .venv / site-packages / hidden-backup files (2235 on /srv/pyit/dev) that v2 prunes. v2's enumerate counts match the hugpy_tools.search sibling exactly.

from abstract_search import make_allowed_predicate, filter_allowed_items, search_files

allowed = make_allowed_predicate(exclude_exts=".log", add=True)
kept = filter_allowed_items(paths, allowed_exts=[".py", ".md"])
hits = search_files("needle", "/path/to/project")   # line-level SearchHit records

Optional readers: pip install abstract_search[pdf] (pdfplumber), [ocr], [xls], [xlsb], [dbf], [parquet], or [all].