searchbox 0.3.0

A lightweight local document ingestion and retrieval package.

searchbox

searchbox is a lightweight Python package for modular search systems.

It does one thing:

• read local files or folders
• extract text from supported document types
• split long text into chunks
• store chunks through a pluggable repository layer
• fetch candidates through a pluggable source layer
• rank candidates through pluggable search strategies

Install

Install from the current project directory:

pip install .

If you later publish it to PyPI under the same package name:

pip install searchbox

What It Supports

Supported file types:

• .txt, .md, .rst
• .py, .toml, .yaml, .yml
• .csv, .tsv
• .json
• .html, .htm
• .pdf

Default document storage backend:

• SQLite database file on local disk via SqliteDocumentRepository

Default document retrieval behavior:

• documents are stored as chunks internally
• candidates are fetched from a source abstraction
• search is performed on chunks with BM25 ranking by default
• results are returned as document-level hits
• each result includes the best matching chunk text and matched chunk indexes

Architecture:

• BaseSearch defines the shared search workflow
• SearchStrategy defines ranking modes such as BM25, vector, semantic, or generative search
• CandidateSource defines how candidates are fetched
• DocumentRepository defines how chunks are persisted
• DocumentIngestor handles file ingestion independently from search
• this makes it easier to replace local SQLite with another database later

Quick Start

from searchbox import docsearch

engine = docsearch(db_path="docs.db", chunk_size=240, chunk_overlap=40)
engine.add_path("docs")

results = engine.search("how does indexing work", top_k=3)

for item in results:
    print(item["id"], item["score"], item["metadata"]["matched_chunks"])

Main API

Create a search engine:

from searchbox import docsearch

engine = docsearch(
    db_path="docs.db",
    chunk_size=240,
    chunk_overlap=40,
)

Available methods:

• docsearch(db_path="searchbox.db", chunk_size=240, chunk_overlap=40)
• add_path(path, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)
• add_paths(paths, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)
• search(query, top_k=5)
• search_pretty(query, top_k=5, snippet_length=280)
• response(query, top_k=5)
• count()
• clear()
• supported_extensions()

Advanced composition:

• docsearch(..., repository=..., source=..., strategies=..., extractor=...)
• websearch(web_fetcher, strategies=...)
• SqliteDocumentRepository
• RepositoryCandidateSource
• BM25SearchStrategy
• BaseSearch
• SearchFactory
• FileTextExtractor

Constructor Parameters

docsearch(...)

engine = docsearch(
    db_path="searchbox.db",
    chunk_size=240,
    chunk_overlap=40,
)

Parameters:

• db_path: SQLite database path; default is searchbox.db
• chunk_size: number of words per chunk; default is 240
• chunk_overlap: overlapping words between neighboring chunks; default is 40
• repository: optional custom repository implementation
• source: optional custom candidate source implementation
• strategies: optional mapping of search strategies
• extractor: optional custom file text extractor
• default_mode: which strategy name to use by default

Notes:

• chunk_size must be greater than 0
• chunk_overlap must be greater than or equal to 0
• chunk_overlap must be smaller than chunk_size

Ingestion Methods

add_path(path, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)

Index one file or one directory.

If path is:

• a file: only that file is indexed
• a directory: files inside the directory are indexed

Returns:

• a list of inserted chunk ids

Parameters:

• path: file path or directory path
• recursive: when path is a directory, whether to scan subdirectories; default True
• include_hidden: whether to include hidden files and hidden directories; default False
• chunk_size: optional per-call override of constructor chunk_size
• chunk_overlap: optional per-call override of constructor chunk_overlap

Example:

engine.add_path("docs")
engine.add_path("README.md")
engine.add_path("docs", recursive=False)

add_paths(paths, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)

Index multiple files or directories in one call.

Parameters:

• paths: a sequence of file or directory paths
• all other parameters behave the same as add_path(...)

Example:

engine.add_paths(
    [
        "docs",
        "notes/project.md",
        "data/reference",
    ]
)

Search Methods

search(query, top_k=5)

Search the indexed documents and return the most relevant document hits.

Parameters:

• query: search text
• top_k: maximum number of returned documents; default 5

Returns:

• a list of result dictionaries sorted by descending relevance

Current default ranking:

• BM25 over chunk text plus title text
• document results are produced by aggregating chunk-level hits

You can also pass another strategy mode:

results = engine.search("what is rag", mode="bm25")

response(query, top_k=5)

Alias of search(query, top_k=5).

Use whichever name you prefer.

search_pretty(query, top_k=5, snippet_length=280)

Search the indexed documents and return a more display-friendly result shape.

Parameters:

• query: search text
• top_k: maximum number of returned documents; default 5
• snippet_length: max snippet length in characters; default 280

Returns:

• a list of simplified dictionaries with path, file_name, score, matched_chunk_count, matched_chunks, best_chunk_index, and snippet

Examples

Index a single file

from searchbox import docsearch

engine = docsearch(db_path="single.db")
engine.add_path("guide.txt")

results = engine.search("installation guide", top_k=1)
print(results)

Index a directory

from searchbox import docsearch

engine = docsearch(db_path="docs.db")
engine.add_path("docs")

results = engine.search("local retrieval", top_k=3)
for item in results:
    print(item["id"], item["title"], item["score"])

Index multiple locations

from searchbox import docsearch

engine = docsearch(db_path="multi.db")
engine.add_paths(["docs", "notes", "README.md"])

results = engine.search("api design", top_k=5)
for item in results:
    print(item["metadata"]["source_path"])

Override chunk settings for one ingestion call

from searchbox import docsearch

engine = docsearch(db_path="docs.db", chunk_size=240, chunk_overlap=40)

engine.add_path(
    "manuals",
    chunk_size=120,
    chunk_overlap=20,
)

Show supported file types

from searchbox import docsearch

engine = docsearch()
print(engine.supported_extensions())

Use your own repository, source, or strategies

from searchbox import BM25SearchStrategy, DocumentSearch, RepositoryCandidateSource, SqliteDocumentRepository

repository = SqliteDocumentRepository(db_path="docs.db")
source = RepositoryCandidateSource(repository)
strategies = {
    "bm25": BM25SearchStrategy(),
}

engine = DocumentSearch(
    repository=repository,
    source=source,
    strategies=strategies,
    chunk_size=240,
    chunk_overlap=40,
)

Get display-friendly search output

from searchbox import docsearch

engine = docsearch(db_path="docs.db")
engine.add_path("papers")

results = engine.search_pretty("vllm", top_k=3)

for item in results:
    print(item["path"])
    print(item["score"])
    print(item["matched_chunk_count"])
    print(item["snippet"])
    print("---")

Search Result Format

Each search result is a document-level hit.

Example:

[
    {
        "id": "guide.pdf",
        "title": "guide.pdf",
        "content": "best matching chunk text ...",
        "score": 0.82,
        "metadata": {
            "source_path": "guide.pdf",
            "source_name": "guide.pdf",
            "chunk_count": 4,
            "matched_chunks": [0, 1],
            "best_chunk_index": 1,
        },
    }
]

Field meaning:

• id: document identifier; currently the source file path string used by the result layer
• title: display title derived from the file name
• content: the best matching chunk text for that document
• score: relevance score
• metadata["source_path"]: file path stored for the source document
• metadata["source_name"]: file name only
• metadata["chunk_count"]: total number of stored chunks for the file
• metadata["matched_chunks"]: chunk indexes that matched the query
• metadata["best_chunk_index"]: chunk index whose text is returned in content

How Ingestion Works

When you call add_path(...) or add_paths(...):

1. searchbox reads each supported file
2. extracted text is normalized
3. long text is split into overlapping word chunks
4. every chunk is stored through the configured repository
5. later, a candidate source fetches chunks for search
6. the active search strategy scores candidates
7. chunk hits are merged back into document-level results

This means:

• large files can still be searched effectively
• results are returned as documents, not raw chunks
• content in the result is the best matching chunk, not always the full original file

PDF Notes

PDF ingestion is supported.

Current behavior:

• searchbox first tries the system command pdftotext
• if that fails, it uses a lightweight fallback extractor

Recommendation:

• if you care about PDF extraction quality, make sure pdftotext is installed on the machine

Modular Design

searchbox is structured so ingestion, storage, and retrieval are decoupled:

• BaseSearch: shared search flow
• DocSearch: document-specific search based on the shared flow
• WebSearch: web-specific search based on the shared flow
• DocumentSearch: high-level document search facade
• FileTextExtractor: file reading and text extraction
• CandidateSource: candidate fetching interface
• RepositoryCandidateSource: candidate source backed by a repository
• DocumentRepository: storage interface
• SqliteDocumentRepository: current local SQLite implementation
• SearchStrategy: ranking interface
• BM25SearchStrategy: current default strategy
• VectorSearchStrategy, SemanticSearchStrategy, GenerativeSearchStrategy, DiffusionRetrievalStrategy: extension points for more advanced retrieval modes

This means that if you later move from local SQLite to a web or cloud database, you can add a new repository implementation and keep the ingestion and retrieval API stable.

Utility Methods

count()

Return the number of stored chunk rows in the SQLite database.

Example:

count = engine.count()
print(count)

clear()

Delete all stored chunk rows from the database.

Example:

engine.clear()

supported_extensions()

Return the currently supported file extensions.

Example:

print(engine.supported_extensions())

Behavior and Edge Cases

• unsupported file types are skipped
• empty or unreadable extracted content is skipped
• hidden files are skipped by default when scanning directories
• if a given path does not exist, FileNotFoundError is raised
• if query is empty or blank, ValueError is raised
• if no document matches, search(...) returns an empty list

Minimal End-to-End Example

from searchbox import docsearch

engine = docsearch(db_path="example.db")

engine.add_paths(
    [
        "docs",
        "README.md",
        "notes/plan.json",
    ]
)

results = engine.search("sqlite document retrieval", top_k=3)

for item in results:
    print("path:", item["metadata"]["source_path"])
    print("score:", item["score"])
    print("best chunk:", item["content"])
    print("matched chunks:", item["metadata"]["matched_chunks"])
    print("---")