flashtext2 1.1.0

A package for extracting keywords from large text very quickly (much faster than regex and the original flashtext package

pip install flashtext2

flashtext2

flashtext2 is an optimized version of the flashtext library for fast keyword extraction and replacement. Its orders of magnitude faster compared to regular expressions.

Key Enhancements in flashtext2

• Rewritten for Better Performance: Completely rewritten in Rust, making it approximately 3-10x faster than the original version.
• Unicode Standard Annex #29: Instead of relying on arbitrary regex patterns like flashtext does: [A-Za-z0-9_]+, flashtext2 uses the Unicode Standard Annex #29 to split strings into tokens. This ensures compatibility with all languages, not just Latin-based ones.
• Unicode Case Folding: Instead of converting strings to lowercase for case-insensitive matches, it uses Unicode case folding, ensuring accurate normalization of characters according to the Unicode standard.
• Fully Type-Hinted API: The entire API is fully type-hinted, providing better code clarity and improved development experience.

Usage

Click to unfold usage

Keyword Extraction

from flashtext2 import KeywordProcessor

kp = KeywordProcessor(case_sensitive=False)

kp.add_keyword('Python')
kp.add_keyword('flashtext')
kp.add_keyword('program')

text = "I love programming in Python and using the flashtext library."

keywords_found = kp.extract_keywords(text)
print(keywords_found)
# Output: ['Python', 'flashtext']

keywords_found = kp.extract_keywords_with_span(text)
print(keywords_found)
# Output: [('Python', 22, 28), ('flashtext', 43, 52)]

Keyword Replacement

from flashtext2 import KeywordProcessor

kp = KeywordProcessor(case_sensitive=False)

kp.add_keyword('Java', 'Python')
kp.add_keyword('regex', 'flashtext')

text = "I love programming in Java and using the regex library."
new_text = kp.replace_keywords(text)

print(new_text)
# Output: "I love programming in Python and using the flashtext library."

Case Sensitivity

from flashtext2 import KeywordProcessor

text = 'abc aBc ABC'

kp = KeywordProcessor(case_sensitive=True)
kp.add_keyword('aBc')

print(kp.extract_keywords(text))
# Output: ['aBc']

kp = KeywordProcessor(case_sensitive=False)
kp.add_keyword('aBc')

print(kp.extract_keywords(text))
# Output: ['aBc', 'aBc', 'aBc']

Other Examples

Overlapping keywords (returns the longest sequence)

from flashtext2 import KeywordProcessor

kp = KeywordProcessor(case_sensitive=True)
kp.add_keyword('machine')
kp.add_keyword('machine learning')

text = "machine learning is a subset of artificial intelligence"
print(kp.extract_keywords(text))
# Output: ['machine learning']

Case folding

from flashtext2 import KeywordProcessor

kp = KeywordProcessor(case_sensitive=False)
kp.add_keywords_from_iter(["flour", "Maße", "ᾲ στο διάολο"])

text = "flour, MASSE, ὰι στο διάολο"
print(kp.extract_keywords(text))
# Output: ['flour', 'Maße', 'ᾲ στο διάολο']

Performance

Click to unfold performance

Extracting keywords is usually 2.5-3x faster, and replacing them is about 10x.
There is still room to optimize the code and improve performance.
You can find the benchmarks here.

The words have on average 6 characters, and a sentence has 10k words, so the length is 60k.

TODO

Click to unfold TODO

• Add multiple ways of normalizing strings: simple case folding, full case folding, and locale-aware folding
• Remove all clones in src code

Credit to Vikash Singh, the author of the original flashtext package.