TurboText

Lightning-fast, boundary-aware keyword matching for Python
Exact · Fuzzy · Multi-word · Unicode · Pluggable conflict resolution

TurboText finds keywords in text the way a human editor would — enforcing real word boundaries, tolerating typos, handling multi-word phrases, and letting you declare which match wins when keywords overlap, including an optimal (not just greedy) weighted resolver that no other keyword library provides.

Table of Contents

• Why TurboText?
• Features
• Installation
• Quick Start
• Adding Keywords
• Extracting Matches
• Replacing Keywords
• Fuzzy Matching
• Conflict Resolution Policies
• Per-Keyword Metadata
• Word Boundary Rules
• Performance
• API Reference
• Development
• Author
• License

Why TurboText?

Features

• Exact matching — trie-based O(n) scan, constant in vocabulary size
• Fuzzy matching — Levenshtein distance up to k via bounded-edit frontier search
• Word boundary enforcement — Unicode-correct; matches cat but not cats, scat, or concatenate
• Multi-word keywords — "new york", "product manager" work out of the box
• Bulk loading — accepts list, dict[str, str], or dict[str, list[str]]
• Five conflict policies — ALL_OVERLAPS, LEFTMOST_LONGEST, LEFTMOST_FIRST, HIGHEST_PRIORITY, OPTIMAL_WEIGHTED
• Optimal resolution — O(n log n) weighted interval scheduling DP — picks the globally best non-overlapping set
• Rich match objects — canonical form, offsets, edit distance, category, priority, custom metadata
• Cython fast-path — compiled extension ships with every wheel; packed transition-table automaton, 1.5× faster than FlashText on 1 M-word corpora

Installation

pip install turbotext

The Cython extension is built automatically when installing from PyPI (wheels are pre-compiled for CPython 3.10–3.13 on Linux, macOS, and Windows). No extra steps needed.

TurboText falls back to pure Python automatically if the compiled extension is unavailable (e.g. unsupported platform or source install without a C compiler).

git clone https://github.com/nishankmahore/TurboText
cd TurboText
uv sync --group dev
python setup.py build_ext --inplace

Quick Start

from turbotext import KeywordStore, MatchPolicy, FuzzyConfig

store = KeywordStore(
    policy=MatchPolicy.LEFTMOST_LONGEST,
    fuzzy=FuzzyConfig(max_edit_distance=1),
)

store.add_keywords({
    "aspirin":   ["aspirin", "asprin", "aspirin tablet"],
    "ibuprofen": ["ibuprofen", "ibuprofen tablet"],
})

text = "Patient takes asprin and ibuprofen tablet daily"

for m in store.extract(text):
    print(f"{m.canonical:12}  ed={m.edit_distance}  [{m.start}:{m.end}]  '{m.text}'")
# aspirin       ed=1  [14:20]  'asprin'
# ibuprofen     ed=0  [25:42]  'ibuprofen tablet'

print(store.replace(text))
# Patient takes aspirin and ibuprofen daily

Adding Keywords

Single keyword

store = KeywordStore()

# minimal — surface form becomes the canonical
store.add_keyword("aspirin")

# with canonical form
store.add_keyword("aspirin", canonical="Aspirin")

# with full metadata
kid = store.add_keyword(
    "aspirin",
    canonical="Aspirin",
    category="DRUG",
    priority=10.0,
    rxnorm_code="1191",   # any extra kwargs become metadata
    source="rxnorm",
)
print(kid)  # "3f4a2b1c-..."  UUID — useful for tracking

From a list

store.add_keywords(["java", "python", "rust"])
# surface form = canonical for each entry

From a {surface: canonical} dict

store.add_keywords({
    "py":   "Python",
    "js":   "JavaScript",
    "k8s":  "Kubernetes",
})

From a {canonical: [surfaces]} dict (most common bulk shape)

keyword_dict = {
    "java":               ["java", "java_2e", "java programing"],
    "product management": ["PM", "product manager", "prod mgmt"],
    "machine learning":   ["ML", "machine learning", "deep learning"],
}
store.add_keywords(keyword_dict)

Mixed dict

String values follow {surface: canonical}; list values follow {canonical: [surfaces]}.

store.add_keywords({
    "java": ["java_2e", "java programing"],  # canonical → [surfaces]
    "py":   "Python",                         # surface   → canonical
})

Shared category and priority

store.add_keywords(
    {
        "aspirin":   ["aspirin", "asprin"],
        "ibuprofen": ["ibuprofen", "advil"],
    },
    category="DRUG",
    priority=5.0,
)

Extracting Matches

matches = store.extract("take aspirin and ibuprofen daily")

for m in matches:
    print(m.text)           # surface text found in the input
    print(m.canonical)      # normalised keyword name
    print(m.start, m.end)   # character offsets — text[m.start:m.end]
    print(m.edit_distance)  # 0 = exact, 1 = one typo, etc.
    print(m.category)       # "DRUG"
    print(m.priority)       # 10.0
    print(m.keyword_id)     # UUID string
    print(m.metadata)       # {"rxnorm_code": "1191", ...}

Match uses __slots__ for fast bulk creation — all fields are fixed and the object is lightweight.

Replacing Keywords

store = KeywordStore()
store.add_keywords({
    "aspirin":       "Aspirin",
    "tylenol":       "Acetaminophen",   # alias → same canonical
    "ibuprofen":     "Ibuprofen",
})

print(store.replace("take aspirin or tylenol twice daily"))
# take Aspirin or Acetaminophen twice daily

Fuzzy Matching

from turbotext import FuzzyConfig

store = KeywordStore(fuzzy=FuzzyConfig(max_edit_distance=1))
store.add_keyword("aspirin")

store.extract("take asprin")    # substitution  i → r  ✅
store.extract("take aspirn")    # deletion      missing i  ✅
store.extract("take aspirrin")  # insertion     extra r  ✅

Boundary enforcement still applies

store.extract("I see cbt")   # ✅ whole word
store.extract("I see xcbt")  # ❌ left boundary fails
store.extract("I see cbts")  # ❌ right boundary fails

Choosing max_edit_distance

Higher values increase recall but also false positives. Start with 1 and tune.

Conflict Resolution Policies

When keywords overlap, the policy decides which match to keep.

store = KeywordStore(policy=MatchPolicy.ALL_OVERLAPS)
store.add_keywords(["new", "new york", "york"])

store.extract("new york")
# → ["new", "new york", "york"]

store = KeywordStore(policy=MatchPolicy.LEFTMOST_LONGEST)
store.add_keywords(["new", "new york"])

store.extract("new york")
# → ["new york"]   longest match wins

store = KeywordStore(policy=MatchPolicy.LEFTMOST_FIRST)
store.add_keywords(["new", "new york"])

store.extract("new york")
# → ["new"]   first token wins

store = KeywordStore(policy=MatchPolicy.HIGHEST_PRIORITY)
store.add_keyword("new york", canonical="New York", priority=2.0)
store.add_keyword("york",     canonical="York",     priority=5.0)
store.add_keyword("new",      canonical="New",      priority=1.0)

store.extract("new york")
# → ["New", "York"]   priority 5 beats priority 2; "new" doesn't overlap "york"

store = KeywordStore(policy=MatchPolicy.OPTIMAL_WEIGHTED)
store.add_keyword("ab cd", canonical="LONG",   priority=5.0)
store.add_keyword("ab",    canonical="SHORT1", priority=3.0)
store.add_keyword("cd",    canonical="SHORT2", priority=3.0)

store.extract("ab cd")
# Greedy picks "LONG" (5).  Optimal picks "SHORT1"+"SHORT2" (3+3=6).
# → ["SHORT1", "SHORT2"]

Per-Keyword Metadata

store.add_keyword(
    "aspirin",
    canonical="Aspirin",
    category="DRUG",
    priority=10.0,
    rxnorm_code="1191",
    source="rxnorm",
    approved=True,
)

m = store.extract("take aspirin")[0]
print(m.canonical)               # "Aspirin"
print(m.category)                # "DRUG"
print(m.priority)                # 10.0
print(m.metadata["rxnorm_code"]) # "1191"

Match.metadata is a shallow copy — mutating it does not affect the stored keyword.

Word Boundary Rules

TurboText uses Unicode word boundaries — word characters are [a-zA-Z0-9_] and their Unicode equivalents.

store.add_keyword("cat")

# ✅ Accepted
store.extract("cat")          # text edge
store.extract("the cat sat")  # spaces
store.extract("(cat)")        # punctuation
store.extract("cat, sat")     # comma

# ❌ Rejected
store.extract("cats")         # right boundary fails
store.extract("scat")         # left boundary fails
store.extract("cat2")         # digit is a word char
store.extract("concatenate")  # substring

Performance

Apple M-series · best-of-3 runs · Cython extension enabled

1 M-word throughput (1,000 keywords, 7.8 MB corpus)

TurboText's Aho-Corasick engine with inline lowercasing and zero-copy resolve beats FlashText on large documents.

Exact matching vs regex — vocabulary scaling (k=0)

TurboText is O(text) — scan time is flat as vocabulary grows. re alternation degrades linearly with term count because every disjunct is tried at every position.

The k=0 engine is a packed transition table: the Aho-Corasick automaton is flattened into one contiguous int32 array (state*128+char → next_state) instead of a graph of individually heap-allocated trie nodes. Transitions become a single branchless array index — no null-check, no pointer chase through separate node/list/array allocations — so the win grows with automaton size as more of the old pointer graph would have spilled out of cache.

Exact matching vs the Aho-Corasick family (k=0)

Separated from the regex comparison above because it's a different question: every library here is already O(text), so the story isn't complexity class, it's implementation constants — Python-object overhead, boundary handling, and (for the C extension) FFI/allocation cost. pyahocorasick's C core wins on raw scan speed but hands back plain substring spans with no boundary awareness or multi-word phrase support — it's a general-purpose multi-pattern search primitive, not a keyword-extraction library.

TurboText (spans) is extract_spans(): the same FlatAC engine as extract(), but returning bare (start, end, keyword_id) tuples instead of building a Match object per hit (no text-slice, no attribute assignment, no metadata dict). Keyword refs are int-indexed internally (an array lookup) rather than hashed by a 36-char uuid string, so this is close to pyahocorasick's raw-scan speed while still being boundary-aware. extract() pays the difference for Match objects — canonical form, category, priority, metadata — which pyahocorasick doesn't have a concept of at all.

Fuzzy matching vs FuzzyWuzzy — vocabulary scaling (k=1)

TurboText's fuzzy path (bounded-edit frontier search) is a Cython fast path, same as exact matching: a single-pass trie scan regardless of vocabulary size. FuzzyWuzzy tokenises the text and scores every token against every keyword with a pure-Python-Levenshtein-backed ratio — O(tokens × vocab), and it shows: 5,000 keywords take over 11 seconds.

Fuzzy matching vs RapidFuzz, python-Levenshtein, jellyfish (k=1)

A separate chart because these three don't share FuzzyWuzzy's story: RapidFuzz has a genuinely C-accelerated batch primitive (process.extractOne), while python-Levenshtein and jellyfish only expose a single-pair distance function — no batch "find the best match in this list" API — so a token × keyword nested loop calling their distance function is the realistic way to use them here, not an artificial handicap. That nested loop puts jellyfish in the same ballpark as FuzzyWuzzy above (11.1s vs 11.4s at 5,000 terms) despite jellyfish's distance function itself being fast in isolation.

TurboText overtakes RapidFuzz at ~500 keywords and is 8.8× faster at 5,000 terms. Unlike RapidFuzz (a string scorer used with process.extractOne per token), TurboText returns character offsets, enforces word boundaries, and handles multi-word phrases natively — and unlike python-Levenshtein/jellyfish, it never needs an O(tokens × vocab) nested loop in the first place.

API Reference

store = KeywordStore(
    policy=MatchPolicy.LEFTMOST_LONGEST,    # default
    fuzzy=FuzzyConfig(max_edit_distance=0), # default — exact only
)

kid = store.add_keyword(
    "aspirin",
    canonical="Aspirin",
    category="DRUG",
    priority=10.0,
    rxnorm_code="1191",
)

matches = store.extract("Patient takes aspirin daily")

for start, end, keyword_id in store.extract_spans("Patient takes aspirin daily"):
    ...

result = store.replace("take aspirin or tylenol")

FuzzyConfig(max_edit_distance=1)  # int, default 0

Development

uv sync --group dev                        # install dependencies
python setup.py build_ext --inplace        # build Cython extension
uv run pytest                              # run tests
uv run ruff check src tests                # lint
uv run mypy src/turbotext                  # type-check
uv run pytest benches/ --benchmark-only    # throughput benchmarks
uv run python benches/bench_1m_words.py    # 1 M-word TurboText vs FlashText
uv run python benches/scaling_benchmark.py # regenerate scaling charts

src/turbotext/
    __init__.py       public API exports
    trie.py           TrieNode + TrieBuilder
    frontier.py       bounded-edit frontier search (pure Python fallback)
    _fast.pyx         Cython hot-path for exact (k=0) and fuzzy (k>0) search
    _fast.pyi         type stub for the Cython extension
    resolve.py        conflict-resolution policies
    store.py          KeywordStore public class
reference/
    reference_matcher.py    brute-force oracle for differential testing
tests/
    test_m0_smoke.py                   API surface + add_keywords shapes
    test_m1_exact.py                   exact matching + boundary rules
    test_m2_metadata_priorities.py     metadata, HIGHEST_PRIORITY, OPTIMAL_WEIGHTED
    test_m3_fuzzy.py                   fuzzy matching + hypothesis property tests
benches/
    bench_m3.py             pytest-benchmark: k=0 vs k=1 throughput
    bench_comparison.py     pytest-benchmark: TurboText vs FlashText vs re vs RapidFuzz
    bench_1m_words.py       1 M-word throughput: TurboText vs FlashText
    scaling_benchmark.py    vocabulary-sweep scaling charts
assets/
    logo.png

Author

Nishank Mahore
nishankmahore@gmail.com · github.com/nishankmahore

If TurboText is useful to you, feel free to open an issue, suggest a feature, or contribute a pull request.

License

Released under the MIT License — see LICENSE for the full text.

Made with Python · Powered by Cython · Built for speed