turbotext 0.1.0

Boundary-aware fuzzy keyword matching with pluggable conflict resolution

TurboText

Lightning-fast, boundary-aware keyword matching for Python
Exact · Fuzzy · Multi-word · Unicode · Pluggable conflict resolution

---

TurboText finds keywords in text the way a human editor would — enforcing real word boundaries, tolerating typos, handling multi-word phrases, and letting you declare which match wins when keywords overlap, including an optimal (not just greedy) weighted resolver that no other keyword library provides.

---

Table of Contents

• Why TurboText?
• Features
• Installation
• Quick Start
• Adding Keywords
• Extracting Matches
• Replacing Keywords
• Fuzzy Matching
• Conflict Resolution Policies
• Per-Keyword Metadata
• Word Boundary Rules
• Performance
• API Reference
• Development
• Author
• License

---

Why TurboText?

	re	FlashText	RapidFuzz	TurboText
Exact keyword extraction	✅	✅	❌	✅
Fuzzy / typo-tolerant	❌	❌	✅	✅
Word boundary enforcement	✅	✅	❌	✅
Extracts spans from running text	✅	✅	❌	✅
Multi-word phrases (native)	✅	✅	⚠️	✅
O(text) scaling with vocab	❌	✅	❌	✅
Per-keyword metadata	❌	❌	❌	✅
Conflict resolution policies	❌	❌	❌	✅
Optimal weighted resolution	❌	❌	❌	✅

---

Features

• Exact matching — trie-based O(n) scan, constant in vocabulary size
• Fuzzy matching — Levenshtein distance up to k via bounded-edit frontier search
• Word boundary enforcement — Unicode-correct; matches cat but not cats, scat, or concatenate
• Multi-word keywords — "new york", "product manager" work out of the box
• Bulk loading — accepts list, dict[str, str], or dict[str, list[str]]
• Five conflict policies — ALL_OVERLAPS, LEFTMOST_LONGEST, LEFTMOST_FIRST, HIGHEST_PRIORITY, OPTIMAL_WEIGHTED
• Optimal resolution — O(n log n) weighted interval scheduling DP — picks the globally best non-overlapping set
• Rich match objects — canonical form, offsets, edit distance, category, priority, custom metadata
• Cython fast-path — optional compiled extension brings exact-match throughput to FlashText parity

---

Installation

pip install turbotext

With the Cython extension (recommended for exact matching at scale):

pip install cython turbotext
python setup.py build_ext --inplace

The extension is fully optional — TurboText falls back to pure Python automatically when it is not compiled.

Install with uv (development)

git clone https://github.com/nishankmahore/TurboText
cd TurboText
uv sync --group dev
python setup.py build_ext --inplace

---

Quick Start

from turbotext import KeywordStore, MatchPolicy, FuzzyConfig

store = KeywordStore(
    policy=MatchPolicy.LEFTMOST_LONGEST,
    fuzzy=FuzzyConfig(max_edit_distance=1),
)

store.add_keywords({
    "aspirin":   ["aspirin", "asprin", "aspirin tablet"],
    "ibuprofen": ["ibuprofen", "ibuprofen tablet"],
})

text = "Patient takes asprin and ibuprofen tablet daily"

for m in store.extract(text):
    print(f"{m.canonical:12}  ed={m.edit_distance}  [{m.start}:{m.end}]  '{m.text}'")
# aspirin       ed=1  [14:20]  'asprin'
# ibuprofen     ed=0  [25:42]  'ibuprofen tablet'

print(store.replace(text))
# Patient takes aspirin and ibuprofen daily

---

Adding Keywords

Single keyword

store = KeywordStore()

# minimal — surface form becomes the canonical
store.add_keyword("aspirin")

# with canonical form
store.add_keyword("aspirin", canonical="Aspirin")

# with full metadata
kid = store.add_keyword(
    "aspirin",
    canonical="Aspirin",
    category="DRUG",
    priority=10.0,
    rxnorm_code="1191",   # any extra kwargs become metadata
    source="rxnorm",
)
print(kid)  # "3f4a2b1c-..."  UUID — useful for tracking

From a list

store.add_keywords(["java", "python", "rust"])
# surface form = canonical for each entry

From a {surface: canonical} dict

store.add_keywords({
    "py":   "Python",
    "js":   "JavaScript",
    "k8s":  "Kubernetes",
})

From a {canonical: [surfaces]} dict (most common bulk shape)

keyword_dict = {
    "java":               ["java", "java_2e", "java programing"],
    "product management": ["PM", "product manager", "prod mgmt"],
    "machine learning":   ["ML", "machine learning", "deep learning"],
}
store.add_keywords(keyword_dict)

Mixed dict

String values follow {surface: canonical}; list values follow {canonical: [surfaces]}.

store.add_keywords({
    "java": ["java_2e", "java programing"],  # canonical → [surfaces]
    "py":   "Python",                         # surface   → canonical
})

Shared category and priority

store.add_keywords(
    {
        "aspirin":   ["aspirin", "asprin"],
        "ibuprofen": ["ibuprofen", "advil"],
    },
    category="DRUG",
    priority=5.0,
)

---

Extracting Matches

matches = store.extract("take aspirin and ibuprofen daily")

for m in matches:
    print(m.text)           # surface text found in the input
    print(m.canonical)      # normalised keyword name
    print(m.start, m.end)   # character offsets — text[m.start:m.end]
    print(m.edit_distance)  # 0 = exact, 1 = one typo, etc.
    print(m.category)       # "DRUG"
    print(m.priority)       # 10.0
    print(m.keyword_id)     # UUID string
    print(m.metadata)       # {"rxnorm_code": "1191", ...}

Match is a frozen dataclass — hashable and safe to use as a dict key or set member.

---

Replacing Keywords

store = KeywordStore()
store.add_keywords({
    "aspirin":       "Aspirin",
    "tylenol":       "Acetaminophen",   # alias → same canonical
    "ibuprofen":     "Ibuprofen",
})

print(store.replace("take aspirin or tylenol twice daily"))
# take Aspirin or Acetaminophen twice daily

---

Fuzzy Matching

from turbotext import FuzzyConfig

store = KeywordStore(fuzzy=FuzzyConfig(max_edit_distance=1))
store.add_keyword("aspirin")

store.extract("take asprin")    # substitution  i → r  ✅
store.extract("take aspirn")    # deletion      missing i  ✅
store.extract("take aspirrin")  # insertion     extra r  ✅

Boundary enforcement still applies

store.extract("I see cbt")   # ✅ whole word
store.extract("I see xcbt")  # ❌ left boundary fails
store.extract("I see cbts")  # ❌ right boundary fails

Choosing max_edit_distance

Value	Use case
0	Exact matching only (default) — uses Cython fast-path when available
1	Single-character typos — medical terms, product names
2	Two-character errors — longer technical terms

Higher values increase recall but also false positives. Start with 1 and tune.

---

Conflict Resolution Policies

When keywords overlap, the policy decides which match to keep.

ALL_OVERLAPS — return everything, you decide

store = KeywordStore(policy=MatchPolicy.ALL_OVERLAPS)
store.add_keywords(["new", "new york", "york"])

store.extract("new york")
# → ["new", "new york", "york"]

LEFTMOST_LONGEST (default) — FlashText-compatible greedy

store = KeywordStore(policy=MatchPolicy.LEFTMOST_LONGEST)
store.add_keywords(["new", "new york"])

store.extract("new york")
# → ["new york"]   longest match wins

LEFTMOST_FIRST — earliest start wins

store = KeywordStore(policy=MatchPolicy.LEFTMOST_FIRST)
store.add_keywords(["new", "new york"])

store.extract("new york")
# → ["new"]   first token wins

HIGHEST_PRIORITY — priority-based greedy

store = KeywordStore(policy=MatchPolicy.HIGHEST_PRIORITY)
store.add_keyword("new york", canonical="New York", priority=2.0)
store.add_keyword("york",     canonical="York",     priority=5.0)
store.add_keyword("new",      canonical="New",      priority=1.0)

store.extract("new york")
# → ["New", "York"]   priority 5 beats priority 2; "new" doesn't overlap "york"

OPTIMAL_WEIGHTED — globally optimal, not greedy

store = KeywordStore(policy=MatchPolicy.OPTIMAL_WEIGHTED)
store.add_keyword("ab cd", canonical="LONG",   priority=5.0)
store.add_keyword("ab",    canonical="SHORT1", priority=3.0)
store.add_keyword("cd",    canonical="SHORT2", priority=3.0)

store.extract("ab cd")
# Greedy picks "LONG" (5).  Optimal picks "SHORT1"+"SHORT2" (3+3=6).
# → ["SHORT1", "SHORT2"]

---

Per-Keyword Metadata

store.add_keyword(
    "aspirin",
    canonical="Aspirin",
    category="DRUG",
    priority=10.0,
    rxnorm_code="1191",
    source="rxnorm",
    approved=True,
)

m = store.extract("take aspirin")[0]
print(m.canonical)               # "Aspirin"
print(m.category)                # "DRUG"
print(m.priority)                # 10.0
print(m.metadata["rxnorm_code"]) # "1191"

Match.metadata is a shallow copy — mutating it does not affect the stored keyword.

---

Word Boundary Rules

TurboText uses Unicode word boundaries — word characters are [a-zA-Z0-9_] and their Unicode equivalents.

store.add_keyword("cat")

# ✅ Accepted
store.extract("cat")          # text edge
store.extract("the cat sat")  # spaces
store.extract("(cat)")        # punctuation
store.extract("cat, sat")     # comma

# ❌ Rejected
store.extract("cats")         # right boundary fails
store.extract("scat")         # left boundary fails
store.extract("cat2")         # digit is a word char
store.extract("concatenate")  # substring

---

Performance

Apple M-series · 100 → 20,000 keywords · ~20 k-char text · best-of-3 runs

Exact matching (k=0)

TurboText + Cython matches FlashText and stays flat as vocabulary grows. re degrades because its alternation NFA grows with term count.

Library	100 terms	5,000 terms	20,000 terms	Scales?
TurboText	5 ms	5 ms	6 ms	O(text)
FlashText	4 ms	3 ms	3 ms	O(text)
re	6 ms	212 ms	837 ms	O(text × vocab)

Fuzzy matching (k=1)

TurboText does a single-pass trie scan regardless of vocabulary size. RapidFuzz and FuzzyWuzzy tokenise then compare every token against every keyword — O(tokens × vocab).

Library	100 terms	1,000 terms	5,000 terms	Boundary-aware	Extracts spans
TurboText	96 ms	174 ms	239 ms	✅	✅
RapidFuzz	13 ms	120 ms	597 ms	❌	❌
FuzzyWuzzy	222 ms	2,205 ms	10,917 ms	❌	❌

Note: RapidFuzz is a string-similarity scorer, not a text scanner — benchmarks use process.extractOne per token. It does not return character offsets or enforce word boundaries. TurboText does a single trie pass over the full document and natively supports multi-word phrases.

TurboText crosses RapidFuzz at ~2,000 keywords. A Cython extension for the fuzzy frontier is planned for v2.

---

API Reference

KeywordStore(policy, fuzzy)

store = KeywordStore(
    policy=MatchPolicy.LEFTMOST_LONGEST,    # default
    fuzzy=FuzzyConfig(max_edit_distance=0), # default — exact only
)

Parameter	Type	Default	Description
policy	MatchPolicy	LEFTMOST_LONGEST	Conflict-resolution strategy
fuzzy	FuzzyConfig | None	None	Fuzzy config; None = exact only

add_keyword(surface_form, *, canonical, category, priority, **metadata) → str

kid = store.add_keyword(
    "aspirin",
    canonical="Aspirin",
    category="DRUG",
    priority=10.0,
    rxnorm_code="1191",
)

Parameter	Type	Default	Description
surface_form	str	required	Text to search for
canonical	str	surface_form	Normalised name returned on match
category	str | None	None	Grouping label
priority	float	1.0	Weight for priority-based policies
**metadata	Any	—	Arbitrary extra fields

Returns the keyword's UUID string.

add_keywords(keywords, *, category, priority) → list[str]

Shape	Example
list[str]	["java", "python"]
dict[str, str]	{"py": "Python"} — surface → canonical
dict[str, list[str]]	{"Python": ["py", "python3"]} — canonical → surfaces

category and priority apply to every keyword in the call.

extract(text) → list[Match]

Returns resolved matches in span order.

matches = store.extract("Patient takes aspirin daily")

replace(text) → str

Replaces every matched span with its canonical form.

result = store.replace("take aspirin or tylenol")

Match fields

Match is a frozen dataclass — all fields are read-only and it is hashable.

Field	Type	Description
text	str	Matched surface text as it appears in the input
canonical	str	Normalised form from add_keyword
start	int	Start character offset
end	int	End character offset (exclusive)
edit_distance	int	Levenshtein distance (0 = exact)
category	str | None	User-supplied category
priority	float	User-supplied priority
keyword_id	str	UUID from add_keyword
metadata	dict	Copy of extra kwargs

FuzzyConfig and MatchPolicy

FuzzyConfig(max_edit_distance=1)  # int, default 0

MatchPolicy	Description
ALL_OVERLAPS	Return every match
LEFTMOST_LONGEST	Greedy — leftmost, then longest
LEFTMOST_FIRST	Greedy — leftmost, then insertion order
HIGHEST_PRIORITY	Greedy — highest priority wins cluster
OPTIMAL_WEIGHTED	Exact — maximise total priority globally

---

Development

uv sync --group dev                        # install dependencies
python setup.py build_ext --inplace        # build Cython extension (optional)
uv run pytest                              # run tests
uv run ruff check src tests                # lint
uv run mypy src/turbotext                  # type-check
uv run pytest benches/ --benchmark-only    # throughput benchmarks
uv run python benches/scaling_benchmark.py # regenerate scaling charts

Project layout

src/turbotext/
    __init__.py       public API exports
    trie.py           TrieNode + TrieBuilder
    frontier.py       bounded-edit frontier search (pure Python)
    _fast.pyx         Cython hot-path for exact search (k=0)
    _fast.pyi         type stub for the Cython extension
    resolve.py        conflict-resolution policies
    store.py          KeywordStore public class
reference/
    reference_matcher.py    brute-force oracle for differential testing
tests/
    test_m0_smoke.py                   API surface + add_keywords shapes
    test_m1_exact.py                   exact matching + boundary rules
    test_m2_metadata_priorities.py     metadata, HIGHEST_PRIORITY, OPTIMAL_WEIGHTED
    test_m3_fuzzy.py                   fuzzy matching + hypothesis property tests
benches/
    bench_m3.py             pytest-benchmark: k=0 vs k=1 throughput
    bench_comparison.py     pytest-benchmark: TurboText vs FlashText vs re vs RapidFuzz
    scaling_benchmark.py    vocabulary-sweep scaling charts
assets/
    logo.png

---

Author

Nishank Mahore
[e-mail] · github.com/nishankmahore

If TurboText is useful to you, feel free to open an issue, suggest a feature, or contribute a pull request.

---

License

Released under the MIT License — see LICENSE for the full text.

---

Made with Python · Powered by Cython · Built for speed