searchbox 0.2.0

A lightweight local document ingestion and retrieval package.

searchbox

searchbox is a lightweight Python package for local document ingestion and retrieval.

It does one thing:

• read local files or folders
• extract text from supported document types
• split long text into chunks
• store the chunks in SQLite
• retrieve the most relevant documents for a query

Install

Install from the current project directory:

pip install .

If you later publish it to PyPI under the same package name:

pip install searchbox

What It Supports

Supported file types:

• .txt, .md, .rst
• .py, .toml, .yaml, .yml
• .csv, .tsv
• .json
• .html, .htm
• .pdf

Storage backend:

• SQLite database file on local disk

Retrieval behavior:

• documents are stored as chunks internally
• search is performed on chunks
• results are returned as document-level hits
• each result includes the best matching chunk text and matched chunk indexes

Quick Start

from searchbox import docsearch

engine = docsearch(db_path="docs.db", chunk_size=240, chunk_overlap=40)
engine.add_path("docs")

results = engine.search("how does indexing work", top_k=3)

for item in results:
    print(item["id"], item["score"], item["metadata"]["matched_chunks"])

Main API

Create a search engine:

from searchbox import docsearch

engine = docsearch(
    db_path="docs.db",
    chunk_size=240,
    chunk_overlap=40,
)

Available methods:

• docsearch(db_path="searchbox.db", chunk_size=240, chunk_overlap=40)
• add_path(path, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)
• add_paths(paths, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)
• search(query, top_k=5)
• search_pretty(query, top_k=5, snippet_length=280)
• response(query, top_k=5)
• count()
• clear()
• supported_extensions()

Constructor Parameters

docsearch(...)

engine = docsearch(
    db_path="searchbox.db",
    chunk_size=240,
    chunk_overlap=40,
)

Parameters:

• db_path: SQLite database path; default is searchbox.db
• chunk_size: number of words per chunk; default is 240
• chunk_overlap: overlapping words between neighboring chunks; default is 40

Notes:

• chunk_size must be greater than 0
• chunk_overlap must be greater than or equal to 0
• chunk_overlap must be smaller than chunk_size

Ingestion Methods

add_path(path, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)

Index one file or one directory.

If path is:

• a file: only that file is indexed
• a directory: files inside the directory are indexed

Returns:

• a list of inserted chunk ids

Parameters:

• path: file path or directory path
• recursive: when path is a directory, whether to scan subdirectories; default True
• include_hidden: whether to include hidden files and hidden directories; default False
• chunk_size: optional per-call override of constructor chunk_size
• chunk_overlap: optional per-call override of constructor chunk_overlap

Example:

engine.add_path("docs")
engine.add_path("README.md")
engine.add_path("docs", recursive=False)

add_paths(paths, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)

Index multiple files or directories in one call.

Parameters:

• paths: a sequence of file or directory paths
• all other parameters behave the same as add_path(...)

Example:

engine.add_paths(
    [
        "docs",
        "notes/project.md",
        "data/reference",
    ]
)

Search Methods

search(query, top_k=5)

Search the indexed documents and return the most relevant document hits.

Parameters:

• query: search text
• top_k: maximum number of returned documents; default 5

Returns:

• a list of result dictionaries sorted by descending relevance

response(query, top_k=5)

Alias of search(query, top_k=5).

Use whichever name you prefer.

search_pretty(query, top_k=5, snippet_length=280)

Search the indexed documents and return a more display-friendly result shape.

Parameters:

• query: search text
• top_k: maximum number of returned documents; default 5
• snippet_length: max snippet length in characters; default 280

Returns:

• a list of simplified dictionaries with path, file_name, score, matched_chunk_count, matched_chunks, best_chunk_index, and snippet

Examples

Index a single file

from searchbox import docsearch

engine = docsearch(db_path="single.db")
engine.add_path("guide.txt")

results = engine.search("installation guide", top_k=1)
print(results)

Index a directory

from searchbox import docsearch

engine = docsearch(db_path="docs.db")
engine.add_path("docs")

results = engine.search("local retrieval", top_k=3)
for item in results:
    print(item["id"], item["title"], item["score"])

Index multiple locations

from searchbox import docsearch

engine = docsearch(db_path="multi.db")
engine.add_paths(["docs", "notes", "README.md"])

results = engine.search("api design", top_k=5)
for item in results:
    print(item["metadata"]["source_path"])

Override chunk settings for one ingestion call

from searchbox import docsearch

engine = docsearch(db_path="docs.db", chunk_size=240, chunk_overlap=40)

engine.add_path(
    "manuals",
    chunk_size=120,
    chunk_overlap=20,
)

Show supported file types

from searchbox import docsearch

engine = docsearch()
print(engine.supported_extensions())

Get display-friendly search output

from searchbox import docsearch

engine = docsearch(db_path="docs.db")
engine.add_path("papers")

results = engine.search_pretty("vllm", top_k=3)

for item in results:
    print(item["path"])
    print(item["score"])
    print(item["matched_chunk_count"])
    print(item["snippet"])
    print("---")

Search Result Format

Each search result is a document-level hit.

Example:

[
    {
        "id": "guide.pdf",
        "title": "guide.pdf",
        "content": "best matching chunk text ...",
        "score": 0.82,
        "metadata": {
            "source_path": "guide.pdf",
            "source_name": "guide.pdf",
            "chunk_count": 4,
            "matched_chunks": [0, 1],
            "best_chunk_index": 1,
        },
    }
]

Field meaning:

• id: document identifier; currently the source file path string used by the result layer
• title: display title derived from the file name
• content: the best matching chunk text for that document
• score: relevance score
• metadata["source_path"]: file path stored for the source document
• metadata["source_name"]: file name only
• metadata["chunk_count"]: total number of stored chunks for the file
• metadata["matched_chunks"]: chunk indexes that matched the query
• metadata["best_chunk_index"]: chunk index whose text is returned in content

How Ingestion Works

When you call add_path(...) or add_paths(...):

1. searchbox reads each supported file
2. extracted text is normalized
3. long text is split into overlapping word chunks
4. every chunk is stored in the SQLite database
5. later, search(...) ranks chunks and merges them back into document-level results

This means:

• large files can still be searched effectively
• results are returned as documents, not raw chunks
• content in the result is the best matching chunk, not always the full original file

PDF Notes

PDF ingestion is supported.

Current behavior:

• searchbox first tries the system command pdftotext
• if that fails, it uses a lightweight fallback extractor

Recommendation:

• if you care about PDF extraction quality, make sure pdftotext is installed on the machine

Utility Methods

count()

Return the number of stored chunk rows in the SQLite database.

Example:

count = engine.count()
print(count)

clear()

Delete all stored chunk rows from the database.

Example:

engine.clear()

supported_extensions()

Return the currently supported file extensions.

Example:

print(engine.supported_extensions())

Behavior and Edge Cases

• unsupported file types are skipped
• empty or unreadable extracted content is skipped
• hidden files are skipped by default when scanning directories
• if a given path does not exist, FileNotFoundError is raised
• if query is empty or blank, ValueError is raised
• if no document matches, search(...) returns an empty list

Minimal End-to-End Example

from searchbox import docsearch

engine = docsearch(db_path="example.db")

engine.add_paths(
    [
        "docs",
        "README.md",
        "notes/plan.json",
    ]
)

results = engine.search("sqlite document retrieval", top_k=3)

for item in results:
    print("path:", item["metadata"]["source_path"])
    print("score:", item["score"])
    print("best chunk:", item["content"])
    print("matched chunks:", item["metadata"]["matched_chunks"])
    print("---")