polysearch-engine 0.1.1

Lightweight, pluggable hybrid (keyword + semantic) search over your own documents, backed by Postgres full-text search and Pinecone.

polysearch

Lightweight, pluggable hybrid (keyword + semantic) search over your own documents.

polysearch pairs Postgres full-text search (the keyword engine) with Pinecone (the semantic engine) and fuses their results with Reciprocal Rank Fusion. It is a plain Python library and CLI: no web server, no ORM, no async, and connection details are passed in at runtime. Point it at your Postgres and Pinecone, ingest files or database rows, and search.

Install name vs. import name: the distribution on PyPI is polysearch-engine, but you import it as polysearch:

pip install polysearch-engine

from polysearch import SearchService

Features

• Hybrid retrieval. Keyword and semantic search run together and their rankings are merged, so exact-term lookups and natural-language questions both work well.
• Keyword (sparse): Postgres FTS with a graceful cascade, phraseto_tsquery then websearch_to_tsquery then a :* prefix fallback for search-as-you-type, ranked by ts_rank.
• Semantic (dense): OpenAI or a free local embedding model, stored and queried in Pinecone.
• Fusion: Reciprocal Rank Fusion (K=60) merges the two ranked lists. Each result is tagged content / title / both, with matched_pages (keyword hits) and related_pages (semantic hits).
• Query intent: short exact terms run keyword-only; natural-language questions add the semantic engine. The embedding model is never loaded for a keyword-only query.
• Broad ingestion: PDF, DOCX, HTML, PPTX, .txt, .log, images (via OCR or vision), and rows from an external database.
• Multi-tenant: every operation takes a scope (a namespace / tenant key) used as the Pinecone namespace and a Postgres filter.
• Small core, opt-in extras: the base install is just psycopg + pinecone + openai. The heavy dependencies (the local ML stack, file parsers, OCR) are optional extras you add only if you use them.

How it works

                      query
                        │
             ┌──────────┴───────────┐
   keyword (Postgres FTS)   semantic (embeddings → Pinecone)
             └──────────┬───────────┘
              Reciprocal Rank Fusion (K=60)
                        │
                 ranked SearchResult[]

A query is classified as keyword or semantic with cheap string heuristics (no network). Keyword queries hit Postgres only. Semantic queries also embed the query and search Pinecone, then the two ranked lists are fused. Documents found by both engines rank highest.

Requirements

• Python 3.10+
• A reachable Postgres database (the keyword / full-text store)
• A Pinecone index (the semantic / vector store), cloud or the local emulator
• Optionally an OpenAI API key for embeddings and image understanding

Installation

The core stays small. Everything heavy is an opt-in extra:

pip install polysearch-engine                 # core
pip install 'polysearch-engine[local]'        # free local embeddings (sentence-transformers)
pip install 'polysearch-engine[pdf]'          # PDF (pypdf)
pip install 'polysearch-engine[docx]'         # DOCX (python-docx)
pip install 'polysearch-engine[html]'         # HTML (beautifulsoup4)
pip install 'polysearch-engine[pptx]'         # PPTX (python-pptx)
pip install 'polysearch-engine[ocr]'          # image OCR fallback (rapidocr-onnxruntime)
pip install 'polysearch-engine[db]'           # ingest non-Postgres external DBs (SQLAlchemy)
pip install 'polysearch-engine[all]'          # everything (convenience superset)

Install only the extras for the embedding backend and file types you actually use. The bulky packages are transitive: [local] pulls the full ML stack (torch, transformers, scikit-learn, and friends) via sentence-transformers, and [ocr] pulls onnxruntime / opencv via rapidocr. Using OpenAI embeddings (no [local]) keeps the install small.

Quickstart

You configure Postgres, Pinecone, and (optionally) OpenAI. On construction the service validates each connection, ensures its schema, and selects the embedding backend from what validated:

• OpenAI key present and valid picks text-embedding-3-small (1536-dim), plus OpenAI vision for images.
• No/invalid OpenAI key picks a free local model (BAAI/bge-base-en-v1.5, 768-dim; needs [local]), plus RapidOCR for images (needs [ocr]).

from polysearch import SearchService, PostgresConfig, PineconeConfig, OpenAIConfig

svc = SearchService(
    postgres=PostgresConfig(dsn="postgresql://user:pass@host:5432/db"),
    pinecone=PineconeConfig(api_key="...", index="polysearch", cloud="aws", region="us-east-1"),
    openai=OpenAIConfig(api_key="..."),   # optional; omit to use free local embeddings
)
print(svc.status())

svc.ingest_file("handbook.pdf", scope="docs")      # -> (file_id, chunks_indexed)
svc.ingest_path("./docs", scope="docs")            # a directory, ingested recursively

results = svc.search("database connection pooling", scope="docs")   # -> list[SearchResult]
for r in results:
    print(r.score, r.match_type, r.title, r.matched_pages, r.related_pages)

svc.delete("handbook", scope="docs")
svc.close()

scope is your namespace / tenant key. The same index can hold many scopes; searches and deletes are always scoped.

The per-type ingestors are public too, if you only want to parse (no indexing):

from polysearch import HTMLIngestor, DocxIngestor

parsed = HTMLIngestor().extract("page.html")   # -> ParsedDocument

Search results

As objects

svc.search(query, scope) returns list[SearchResult]. Each result has:

field	meaning
file_id	stable id of the document
title	document title (may be None)
score	fused RRF score (higher is better)
match_type	where it matched: content, title, or both
matched_pages	pages with exact keyword hits ("found on page X")
related_pages	pages with semantic hits ("related content on page X")

svc.search_detailed(query, scope) returns (intent, results) if you also want the classified intent ("keyword" or "semantic").

As a JSON envelope

For a ready-to-serialize response, use svc.search_json(query, scope):

svc.search_json("how do I request VPN access?", scope="docs")
# {
#   "query": "how do I request VPN access?",
#   "scope": "docs",
#   "intent": "semantic",
#   "count": 2,
#   "results": [
#     {"file_id": "...", "title": "...", "score": 0.016,
#      "match_type": "content", "matched_pages": [], "related_pages": [1, 2]}
#   ]
# }

CLI

The polysearch command mirrors the library. Flags fall back to environment variables and a .env file (see .env.example).

polysearch init                                   # validate config, report the chosen backend
polysearch status                                 # show enabled capabilities
polysearch ingest ./docs --scope docs             # a file or a directory
polysearch ingest handbook.pdf --scope docs --file-id handbook --title "Employee Handbook"
polysearch ingest-db --config db.json --scope docs
polysearch search "database connection pooling" --scope docs
polysearch delete handbook --scope docs

Every command accepts connection flags (--database-url, --pinecone-api-key, --pinecone-index, --pinecone-host, --openai-api-key, --embedding-backend) and -v for verbose logging.

Configuration

Config can be passed explicitly (the dataclasses above) or read from the environment (used by the CLI). Defaults are shown below.

Variable	Default	Purpose
POLYSEARCH_DATABASE_URL	(required)	Postgres DSN/URL for the keyword store (DATABASE_URL also accepted)
PINECONE_API_KEY	(required)	Pinecone API key
PINECONE_INDEX	polysearch	Pinecone index name
PINECONE_CLOUD	aws	Pinecone cloud (for index creation)
PINECONE_REGION	us-east-1	Pinecone region (PINECONE_ENVIRONMENT also accepted)
PINECONE_HOST	unset	Control-plane host for Pinecone Local / self-hosted; unset for cloud
OPENAI_API_KEY	unset	Optional; enables OpenAI embeddings and image vision
POLYSEARCH_EMBEDDING_BACKEND	auto	auto (OpenAI if valid, else local), openai, or local
POLYSEARCH_LOCAL_MODEL	BAAI/bge-base-en-v1.5	Local embedding model (needs [local])

Ingesting an external database

Map a table's rows to documents (one row becomes one document). Postgres sources use the bundled psycopg driver; other databases need [db] plus the matching driver (for example pymysql).

{
  "dsn": "postgresql://user:pass@host:5432/src",
  "table": "articles",
  "id_column": "id",
  "text_columns": ["title", "body"],
  "title_column": "title",
  "scope_column": null,
  "batch_size": 200
}

polysearch ingest-db --config db.json --scope docs

Embedding consistency and reindexing

Vectors from different embedding models are not comparable, so the service records which backend built each Pinecone index (in a Postgres embedding_state row). If you later ingest or search with a different backend it raises EmbeddingMismatchError. To switch backends, rebuild the dense side:

polysearch search "..." --scope docs --reindex     # CLI

svc.reindex(force=True)                             # library

Reindex re-embeds the chunk text already stored in Postgres, so no source files are needed. The keyword / FTS side is embedding-independent and untouched.

Idempotency

Ingesting is idempotent per (scope, file_id): re-ingesting a file replaces its chunks and reuses its vector ids. The CLI derives a stable file_id from the file path unless you pass --file-id.

Logging

polysearch is silent by default (it installs a null handler and does not touch your root logger). Opt in from your app:

import polysearch
polysearch.configure_logging("INFO")   # or "DEBUG"; omit the call to stay silent

The CLI is quiet unless you pass -v.

Real-world example

A complete, runnable example lives in tests/e2e/: it starts Postgres and Pinecone Local in Docker, ingests sample PDF / PPTX / HTML / TXT / LOG files, and searches them using the free local embedding backend (no OpenAI key). See tests/e2e/README.md.

Development

pip install -e '.[all,dev]'
python -m pytest                     # unit tests (no network)

The full-text integration tests hit a real Postgres and are gated behind an env flag:

docker compose up -d db
RUN_DB_TESTS=1 DATABASE_URL=postgresql://search:search@localhost:5434/search \
    python -m pytest tests/test_integration_fts.py

License

Released under the MIT License. See LICENSE.