re-spell-engine 1.1.0

A dual-channel ML-based spelling correction engine for Movies and TV search, trained on TMDB vocabulary.

Re-Spell Engine

A production-ready, high-performance spelling correction engine for Movie and TV Show searches, trained on TMDB content.

This engine implements a Dual-Channel Retrieval Model (character n-gram TF-IDF on full titles and consonant-only skeletons) backed by Damerau-Levenshtein distance alignment and custom phonetic vowel-substitution heuristics to resolve complex structural and phonetic typos (e.g., betmen, batmen, batmn $\rightarrow$ Batman, and bahubli $\rightarrow$ Bāhubali).

---

Installation

Install the package directly from PyPI:

pip install re-spell-engine

---

Using as a Python Library

You can import and integrate the spelling engine directly into other Python applications. The package includes a pre-trained default model binary of over 108,000+ Movies and TV Shows.

from respell_engine import SpellingCorrector

# 1. Initialize the corrector
corrector = SpellingCorrector()

# 2. Load the default pre-trained model shipped with the package
# (You can optionally pass a custom path to corrector.load("/path/to/custom/model.pkl"))
corrector.load()

# 3. Correct a misspelled query
query = "betmen"
result = corrector.correct(query)

if result["is_corrected"]:
    print(f"Original: {result['original_query']}")
    print(f"Corrected Suggestion: {result['corrected_query']}")
    print(f"Confidence: {result['confidence']:.4f}")
    print(f"Autocorrect Recommended: {result.get('is_autocorrect', False)}")
else:
    print("Spelling is correct or no confident correction was found.")

Correction Response Schema

The .correct() method returns a dictionary with the following fields:

• original_query (str): The input query string.
• corrected_query (str): The clean base corrected title (e.g. Bāhubali instead of Bāhubali: The Epic) to maximize the downstream search scope.
• did_you_mean (str | None): Suggestion string if correction is confident; otherwise None.
• is_corrected (bool): True if a spelling correction was determined.
• is_autocorrect (bool): True if correction confidence is extremely high (e.g., $\ge 0.83$ or edit distance $\le 1.0$) recommending immediate query substitution.
• confidence (float): Score representing the quality of match (bounded between 0.0 and 1.0).
• candidate_details (dict | None): Underlying metadata of matched candidate (id, media_type, distance, raw score, etc.).

Web App Integration Example (Streamlit)

For high-performance in-process usage inside web frontends, cache the spelling corrector instance to prevent loading the model on every page refresh:

import streamlit as st
from respell_engine import SpellingCorrector

@st.cache_resource
def get_corrector():
    corrector = SpellingCorrector()
    corrector.load()  # Automatically loads package data corrector.pkl
    return corrector

corrector = get_corrector()
query = st.text_input("Search")

if query:
    result = corrector.correct(query)
    if result["is_corrected"]:
        st.write(f"Showing results for: {result['corrected_query']}")

---

Web API Integration

Start the web server locally using uvicorn (runs CPU-bound NLP logic on a worker thread pool for production concurrency):

uvicorn respell_engine.main:app --host 127.0.0.1 --port 8000

Endpoints

1. Search with Spelling Correction

• Route: GET /api/search
• Parameters:
  • query (str, required): The search input query.
  • autocorrect (bool, optional, default=true): If set to false, disables correction logic and returns query literally.
• Example request: GET /api/search?query=betmen
• Example Response:

  {
      "original_query": "betmen",
      "corrected_query": "Batman",
      "did_you_mean": "Batman",
      "is_corrected": true,
      "is_autocorrect": true,
      "confidence": 0.9035,
      "candidate_details": {
          "movie_id": 2287,
          "display_title": "Batman",
          "media_type": "tv",
          "distance": 1.0,
          "norm_dist": 0.1666,
          "cosine_sim": 0.0787,
          "score": 0.9035
      }
  }
  

2. Get Engine Status

• Route: GET /api/status

3. Trigger Retraining

• Route: POST /api/train

---

Ingestion & Training Workflow

If you want to train on a fresh dataset from TMDB:

1. Set environment variables

Create a .env file at the root of your project directory containing your TMDB Read Access Token (Bearer Token):

TMDB_API_TOKEN=your_bearer_token_here

2. Ingest Content

python3 -m respell_engine.scripts.fetch_data

3. Re-train Model

python3 -m respell_engine.scripts.train

4. Verify Accuracy

python3 -m respell_engine.scripts.generate_evaluation_log

---

Changelog

v1.1.0 (Current Release)

• Enhanced Accuracy: Reached 98.5% overall accuracy on a expanded test suite of 200 synthetic and real-world typos (added keyboard adjacency errors and double-letter insertions).
• Consonant Skeleton Improvements: Added 'y' to the English vowel list in consonant skeleton generation, resolving query matches like the bos -> The Boys.
• Short-Query Adjustments: Relaxed length boundaries on consonant skeleton matching, allowing 2-3 char abbreviations (like hs -> House) to correct safely under exact skeleton matches.
• Space-Omission Fast Match: Added a fast spaceless dictionary index lookup ($O(1)$) to instantly resolve space-omission errors (e.g. laworderspecialvictimsunit -> Law & Order).
• Data Ingestion Expansion: Added weekly trending discovery and multi-genre crawls (Action, Sci-Fi, Horror, etc.) for both Movies and TV Shows. Expanded original languages to Portuguese, Russian, Arabic, Thai, Turkish, and Polish, growing the unique vocabulary to 108,833 items.
• Package Trimming: Cleaned unused packaging requirements, making the core PyPI package lightweight.