Whoosh-NG

Whoosh-NG is a modern, pure-Python full-text indexing and search library. Version 5.4.0 brings a complete modernization with Python 3.11+ support, strict type annotations, optional feature profiles, and automated semantic releases.

Quick Start

• Full-text search - BM25/BM25F scoring with phrase queries
• Fielded documents - Structured indexing with typed fields
• Query parsing - Flexible parser with boosting and syntax options
• Facets & sorting - Group and sort results by any field
• Highlighting - Snippet extraction with customizable formatters
• Spell checking - Built-in spelling correction
• Event-driven architecture - Plugin system with hooks and middleware
• Optional extensions - Vector search, async, FastAPI, metrics, and more

Installation

Core Installation

pip install whoosh-ng

Optional Profiles

# Vector search with NumPy
pip install "whoosh-ng[vector]"

# Async wrappers
pip install "whoosh-ng[async]"

# FastAPI REST API integration
pip install "whoosh-ng[api]"

# Prometheus metrics
pip install "whoosh-ng[metrics]"

# PostgreSQL backend
pip install "whoosh-ng[postgres]"

# Fuzzy matching
pip install "whoosh-ng[fuzzy]"

# Phonetic search
pip install "whoosh-ng[phonetic]"

# Profiling tools (psutil, re2, pystemmer)
pip install "whoosh-ng[profiling]"

# Fast stemming with PyStemmer C backend
pip install "whoosh-ng[fast-stemming]"

# All optional features
pip install "whoosh-ng[vector,async,api,metrics,postgres,fuzzy,phonetic,profiling,fast-stemming]"

Development Installation

pip install "whoosh-ng[dev]"

Documentation

• Documentation — Full documentation site (English)
• API Reference — Complete module documentation
• Examples — Runnable code examples
• Data Sources — SQL, REST, GraphQL, CSV, JSON, Parquet data sources
• Changelog — Release notes and version history
• French Documentation — Documentation en français
• LLM-Friendly Docs: llms.txt (index) | llms-full.txt (complete API)

Recent Changes in 5.4.0

Performance Highlights

Added

• Plugin System (whoosh.plugins): Plugin base class and PluginManager with entry-point auto-discovery, version validation, conflict detection, enable/disable, and dependency management
• Registry System (whoosh.registry): Generic registry plus StorageRegistry, AnalyzerRegistry, RankingRegistry, SuggestRegistry, VectorRegistry, AutocompleteRegistry, and BackendRegistry
• Middleware Pipeline (whoosh.middleware): Middleware base class (sync + async), MiddlewareContext, MiddlewareChain, MiddlewareRegistry, with official MetricsMiddleware, CacheMiddleware, CompressionMiddleware, EncryptionMiddleware, and PrometheusMiddleware
• Event Bus (whoosh.event_bus): EventBus with subscribe/publish/clear
• Hook System (whoosh.hooks): hookimpl, register_hook, call_hook
• Backends: Backend ABC with lifecycle hooks, FileBackend, and SQLiteBackend
• Provider Architecture: VectorProvider/VectorField, NumpyProvider for vector similarity search
• Autocomplete Plugin (whoosh_modern.autocomplete): Inverted index and edge-ngram autocomplete
• FastAPI Plugin (whoosh_fastapi): REST endpoints for search, autocomplete, vector search, and health checks
• Admin UI Plugin (whoosh_admin): Dashboard for index administration
• Entry Points: Auto-loaded plugins under whoosh.plugins group
• Data Sources (whoosh_modern.data_sources): DataSource protocol with ObservableDataSource, SQLSource (connection pooling, GROUP BY/JOIN/incremental sync), SQLAlchemySource, RESTSource (page/offset/cursor pagination + auth), GraphQLSource, FastCSVSource, JSONSource, ParquetSource, PandasSource, PolarsSource, PeeweeSource, TortoiseSource, PydanticSource, and DataSourceConfig for declarative config from dict/JSON/YAML files
• Schema Discovery (whoosh_modern.schema_discovery): Result-set introspection with duplicate column detection and JSON/JSONB handling
• FacetManager (whoosh_modern.facets): Auto-discovery of facetable fields with manual override support
• Validation Framework (whoosh_modern.validation): 4-level validation (STRICT/WARN/SKIP/NONE) with typed exceptions and field context
• Middleware Pipeline (whoosh_modern.middleware): RetryMiddleware, LoggingMiddleware, CacheMiddleware with chainable pipeline
• SearchView (whoosh_modern.views): Unified interface integrating data sources, schema discovery, facets, validation, and middleware with build(), refresh(), reindex(), validate(), evolve_schema(), and strict mode
• Stemmer Provider System (whoosh_modern.analysis): get_stemmer(), register_stemmer(), auto-detection between internal Porter stemmer and PyStemmer C backend
• Enhanced StemmingAnalyzer (whoosh_modern.analysis): Accepts stemmer="auto"|"internal"|"pystemmer"|provider parameter
• Configuration Engine (whoosh_modern.config): ConfigEngine with hierarchical YAML/JSON merging, Pydantic models (WhooshNGConfig, FieldConfig), and sub-engines (SchemaEngine, AnalyzerEngine, DataSourceEngine, StorageEngine, SearchModelEngine, FacetEngine, PluginEngine, APIEngine)
• Vector Search (whoosh_modern.vector): HnswlibProvider for HNSW approximate nearest neighbor search alongside NumpyProvider
• Embedding Framework (whoosh_modern.embeddings): EmbeddingProvider protocol and SentenceTransformersProvider for sentence-transformers embeddings
• Modern N-Gram Framework (whoosh_modern.analysis): AutoCompleteAnalyzer, EdgeNgramAnalyzer, SEARCH_AS_YOU_TYPE field type, NgramProfiler for performance profiling, and AnalyzerPresets (autocomplete, partial_match, fuzzy, code_search, documentation, ecommerce, blog, multilingual)
• Language Auto-Detection (whoosh_modern.linguistics.detection): StopwordDetector and LangDetectProvider for automatic language detection with configurable supported languages
• Language Registry (whoosh_modern.linguistics.registry): LanguageRegistry, StemmerRegistry, and LanguageProfile for centralized language/analyzer/stemmer resolution with get_default_registry()
• Multi-Language Analyzer (whoosh_modern.linguistics.analyzers): MultiLanguageAnalyzer applies multiple language analyzers simultaneously for multilingual indexing
• Analyzer Presets (whoosh_modern.analysis.stemmer_presets): AnalyzerPresets.documentation(), .ecommerce(), .blog(), .multilingual() for common search scenarios
• Explain Analyzer (whoosh_modern.linguistics.explain): ExplainAnalyzer, AnalysisExplanation, and TokenExplanation expose the tokenization/stemming pipeline for Search Studio
• Cached Stemming Analyzer (whoosh_modern.analysis.cached_stemming_analyzer): CachedStemmingAnalyzer wraps language analyzers with LRU caching (default cache_size=50000) for repeated tokens
• Dictionary Stem Override (whoosh_modern.linguistics.dictionary_stem_override): DictionaryStemOverride allows overriding Snowball stemming with business dictionaries (JSON/Wiktionary)
• Stemmer Profiler (whoosh_modern.profiling.stemmer_profiler): StemmerProfiler and StemmerProfilerReport measure vocabulary reduction, estimated index size reduction, and average stemming time
• Multilingual SearchApplication (whoosh_modern.application): SearchApplication now accepts language_detector and dictionary_stem_overrides parameters; FieldConfig supports language="auto" with detector resolution

Breaking Changes

Re-indexing required. The on-disk posting format in W3TermInfo and position/char encoding in Formats has changed. Indexes created with pre-2.0 versions are not readable by this release. Delete old index directories and re-create them.

• Token now uses __slots__: code that iterates token.__dict__ should use token.copy() or slot introspection instead
• finish_postings() signature changed: allow_compact=True keyword added
• whoosh_modern package structure changed: Import paths for data sources have been updated. For example, from whoosh_modern.data_sources import SQLSource should now be from whoosh_modern.data_sources.sql import SQLSource. Please update your import statements accordingly.
• Import path change: The project was renamed from whoosh-reloaded to whoosh-ng. The modern extension modules previously available under whoosh_reloaded are now importable under whoosh_modern. Existing code using whoosh_reloaded must be updated to whoosh_modern. Core Whoosh components remain available under the whoosh namespace.

Changed

• Distribution is now whoosh-ng (import namespace remains whoosh for core components, whoosh_modern for extensions)
• Documentation links now absolute (GitHub Pages): https://dorel14.github.io/whoosh-ng/en/...
• Python 3.11+ required (dropped Python 3.9/3.10 support)
• Packaging cleaned: consolidated extras in pyproject.toml
• Type annotations modernized: mypy src/whoosh reports 0 errors, py.typed marker included

Example: Simple Search

from whoosh import index
from whoosh.fields import Schema, TEXT, ID
from whoosh.qparser import QueryParser

# Define schema
schema = Schema(
    id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    content=TEXT,
)

# Create index
ix = index.create_in("my_index", schema)

# Index documents
with ix.writer() as w:
    w.add_document(id="1", title="Hello World", content="Welcome to Whoosh-NG")
    w.add_document(id="2", title="Python Search", content="Fast text search library")

# Search
with ix.searcher() as s:
    qp = QueryParser("content", ix.schema)
    q = qp.parse("search library")
    results = s.search(q)
    for hit in results:
        print(hit["title"], hit.score)

Example: FastAPI Integration

from fastapi import FastAPI
from whoosh import index
from whoosh.fields import Schema, TEXT, ID
from whoosh_fastapi import create_app

schema = Schema(id=ID(), title=TEXT(), content=TEXT())
ix = index.create_in("docs", schema)

# Create FastAPI app with Whoosh-NG endpoints
app = create_app(ix, prefix="/api/v1")

# Endpoints available:
# GET  /api/v1/health          - Health check
# POST /api/v1/search          - Full-text search
# GET  /api/v1/autocomplete?q= - Autocomplete suggestions

Example: Data Sources

SQLSource — Index from a SQL database

from whoosh_modern.data_sources.sql import SQLSource
from whoosh import index
from whoosh.fields import Schema, TEXT, NUMERIC
import sqlite3

conn = sqlite3.connect("mydb.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM products",
    incremental_field="updated_at",
    id_field="id",
)

# Discover schema from actual result metadata
schema = source.discover_schema()

# Build index with SearchView
from whoosh_modern.views import SearchView
view = SearchView(name="products", source=source)
ix = view.build("indexdir")

RESTSource — Index from a REST API

from whoosh_modern.data_sources.rest import RESTSource

source = RESTSource(
    url="https://api.example.com/v2/products",
    pagination="page",
    page_size=50,
    headers={"Authorization": "Bearer your_token"},
)

schema = source.discover_schema()
docs = list(source.iter_documents())

SearchView — Full pipeline integration

from whoosh_modern.views import SearchView
from whoosh_modern.data_sources.sql import SQLSource
import sqlite3

conn = sqlite3.connect("mydb.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM reuters_articles",
    incremental_field="article_date",
    id_field="id",
)

view = SearchView(name="reuters", source=source)
ix = view.build("indexdir")

# Incremental refresh
count = view.refresh()

# Full reindex
count = view.reindex()

Example: Vector Search

pip install "whoosh-ng[vector]" numpy

from whoosh import index
from whoosh.fields import Schema, TEXT, ID, VECTOR
from whoosh.vector import VectorField
from whoosh_modern.vector.plugin import VectorPlugin
from whoosh.plugins.manager import PluginManager
import numpy as np

# Create index with vector field
schema = Schema(
    id=ID(stored=True),
    title=TEXT(stored=True),
    embedding=VECTOR(dim=384),
)

# Register vector plugin
VectorPlugin().register(PluginManager())

# Index with embeddings
ix = index.create_in("vector_db", schema)
with ix.writer() as w:
    w.add_document(
        id="doc1",
        title="Python tutorial",
        embedding=np.random.rand(384).astype(np.float32).tobytes()
    )