searchbox 0.2.1

A lightweight local document ingestion and retrieval package.

searchbox

searchbox is a lightweight Python package for modular document ingestion and retrieval.

It does one thing:

• read local files or folders
• extract text from supported document types
• split long text into chunks
• store chunks through a pluggable repository layer
• retrieve documents through a pluggable retriever layer

Install

Install from the current project directory:

pip install .

If you later publish it to PyPI under the same package name:

pip install searchbox

What It Supports

Supported file types:

• .txt, .md, .rst
• .py, .toml, .yaml, .yml
• .csv, .tsv
• .json
• .html, .htm
• .pdf

Default storage backend:

• SQLite database file on local disk via SqliteDocumentRepository

Retrieval behavior:

• documents are stored as chunks internally
• search is performed on chunks with BM25 ranking by default
• results are returned as document-level hits
• each result includes the best matching chunk text and matched chunk indexes

Architecture:

• ingestion/extraction is handled independently from storage
• storage is handled by a repository interface
• ranking is handled by a retriever interface
• this makes it easier to replace local SQLite with another database later

Quick Start

from searchbox import docsearch

engine = docsearch(db_path="docs.db", chunk_size=240, chunk_overlap=40)
engine.add_path("docs")

results = engine.search("how does indexing work", top_k=3)

for item in results:
    print(item["id"], item["score"], item["metadata"]["matched_chunks"])

Main API

Create a search engine:

from searchbox import docsearch

engine = docsearch(
    db_path="docs.db",
    chunk_size=240,
    chunk_overlap=40,
)

Available methods:

• docsearch(db_path="searchbox.db", chunk_size=240, chunk_overlap=40)
• add_path(path, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)
• add_paths(paths, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)
• search(query, top_k=5)
• search_pretty(query, top_k=5, snippet_length=280)
• response(query, top_k=5)
• count()
• clear()
• supported_extensions()

Advanced composition:

• docsearch(..., repository=..., retriever=..., extractor=...)
• SqliteDocumentRepository
• BM25Retriever
• FileTextExtractor

Constructor Parameters

docsearch(...)

engine = docsearch(
    db_path="searchbox.db",
    chunk_size=240,
    chunk_overlap=40,
)

Parameters:

• db_path: SQLite database path; default is searchbox.db
• chunk_size: number of words per chunk; default is 240
• chunk_overlap: overlapping words between neighboring chunks; default is 40
• repository: optional custom repository implementation
• retriever: optional custom retriever implementation
• extractor: optional custom file text extractor

Notes:

• chunk_size must be greater than 0
• chunk_overlap must be greater than or equal to 0
• chunk_overlap must be smaller than chunk_size

Ingestion Methods

add_path(path, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)

Index one file or one directory.

If path is:

• a file: only that file is indexed
• a directory: files inside the directory are indexed

Returns:

• a list of inserted chunk ids

Parameters:

• path: file path or directory path
• recursive: when path is a directory, whether to scan subdirectories; default True
• include_hidden: whether to include hidden files and hidden directories; default False
• chunk_size: optional per-call override of constructor chunk_size
• chunk_overlap: optional per-call override of constructor chunk_overlap

Example:

engine.add_path("docs")
engine.add_path("README.md")
engine.add_path("docs", recursive=False)

add_paths(paths, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)

Index multiple files or directories in one call.

Parameters:

• paths: a sequence of file or directory paths
• all other parameters behave the same as add_path(...)

Example:

engine.add_paths(
    [
        "docs",
        "notes/project.md",
        "data/reference",
    ]
)

Search Methods

search(query, top_k=5)

Search the indexed documents and return the most relevant document hits.

Parameters:

• query: search text
• top_k: maximum number of returned documents; default 5

Returns:

• a list of result dictionaries sorted by descending relevance

Current default ranking:

• BM25 over chunk text plus title text
• document results are produced by aggregating chunk-level hits

response(query, top_k=5)

Alias of search(query, top_k=5).

Use whichever name you prefer.

search_pretty(query, top_k=5, snippet_length=280)

Search the indexed documents and return a more display-friendly result shape.

Parameters:

• query: search text
• top_k: maximum number of returned documents; default 5
• snippet_length: max snippet length in characters; default 280

Returns:

• a list of simplified dictionaries with path, file_name, score, matched_chunk_count, matched_chunks, best_chunk_index, and snippet

Examples

Index a single file

from searchbox import docsearch

engine = docsearch(db_path="single.db")
engine.add_path("guide.txt")

results = engine.search("installation guide", top_k=1)
print(results)

Index a directory

from searchbox import docsearch

engine = docsearch(db_path="docs.db")
engine.add_path("docs")

results = engine.search("local retrieval", top_k=3)
for item in results:
    print(item["id"], item["title"], item["score"])

Index multiple locations

from searchbox import docsearch

engine = docsearch(db_path="multi.db")
engine.add_paths(["docs", "notes", "README.md"])

results = engine.search("api design", top_k=5)
for item in results:
    print(item["metadata"]["source_path"])

Override chunk settings for one ingestion call

from searchbox import docsearch

engine = docsearch(db_path="docs.db", chunk_size=240, chunk_overlap=40)

engine.add_path(
    "manuals",
    chunk_size=120,
    chunk_overlap=20,
)

Show supported file types

from searchbox import docsearch

engine = docsearch()
print(engine.supported_extensions())

Use your own repository or retriever

from searchbox import BM25Retriever, SqliteDocumentRepository, docsearch

repository = SqliteDocumentRepository(db_path="docs.db")
retriever = BM25Retriever()

engine = docsearch(
    repository=repository,
    retriever=retriever,
    chunk_size=240,
    chunk_overlap=40,
)

Get display-friendly search output

from searchbox import docsearch

engine = docsearch(db_path="docs.db")
engine.add_path("papers")

results = engine.search_pretty("vllm", top_k=3)

for item in results:
    print(item["path"])
    print(item["score"])
    print(item["matched_chunk_count"])
    print(item["snippet"])
    print("---")

Search Result Format

Each search result is a document-level hit.

Example:

[
    {
        "id": "guide.pdf",
        "title": "guide.pdf",
        "content": "best matching chunk text ...",
        "score": 0.82,
        "metadata": {
            "source_path": "guide.pdf",
            "source_name": "guide.pdf",
            "chunk_count": 4,
            "matched_chunks": [0, 1],
            "best_chunk_index": 1,
        },
    }
]

Field meaning:

• id: document identifier; currently the source file path string used by the result layer
• title: display title derived from the file name
• content: the best matching chunk text for that document
• score: relevance score
• metadata["source_path"]: file path stored for the source document
• metadata["source_name"]: file name only
• metadata["chunk_count"]: total number of stored chunks for the file
• metadata["matched_chunks"]: chunk indexes that matched the query
• metadata["best_chunk_index"]: chunk index whose text is returned in content

How Ingestion Works

When you call add_path(...) or add_paths(...):

1. searchbox reads each supported file
2. extracted text is normalized
3. long text is split into overlapping word chunks
4. every chunk is stored through the configured repository
5. later, search(...) asks the configured retriever to score chunks
6. chunk hits are merged back into document-level results

This means:

• large files can still be searched effectively
• results are returned as documents, not raw chunks
• content in the result is the best matching chunk, not always the full original file

PDF Notes

PDF ingestion is supported.

Current behavior:

• searchbox first tries the system command pdftotext
• if that fails, it uses a lightweight fallback extractor

Recommendation:

• if you care about PDF extraction quality, make sure pdftotext is installed on the machine

Modular Design

searchbox is structured so ingestion, storage, and retrieval are decoupled:

• DocumentSearch: orchestration layer
• FileTextExtractor: file reading and text extraction
• DocumentRepository: storage interface
• SqliteDocumentRepository: current local SQLite implementation
• Retriever: ranking interface
• BM25Retriever: current default retriever

This means that if you later move from local SQLite to a web or cloud database, you can add a new repository implementation and keep the ingestion and retrieval API stable.

Utility Methods

count()

Return the number of stored chunk rows in the SQLite database.

Example:

count = engine.count()
print(count)

clear()

Delete all stored chunk rows from the database.

Example:

engine.clear()

supported_extensions()

Return the currently supported file extensions.

Example:

print(engine.supported_extensions())

Behavior and Edge Cases

• unsupported file types are skipped
• empty or unreadable extracted content is skipped
• hidden files are skipped by default when scanning directories
• if a given path does not exist, FileNotFoundError is raised
• if query is empty or blank, ValueError is raised
• if no document matches, search(...) returns an empty list

Minimal End-to-End Example

from searchbox import docsearch

engine = docsearch(db_path="example.db")

engine.add_paths(
    [
        "docs",
        "README.md",
        "notes/plan.json",
    ]
)

results = engine.search("sqlite document retrieval", top_k=3)

for item in results:
    print("path:", item["metadata"]["source_path"])
    print("score:", item["score"])
    print("best chunk:", item["content"])
    print("matched chunks:", item["metadata"]["matched_chunks"])
    print("---")