flashtext-i18n 3.0.0

Extract/Replace keywords in sentences. Fork with internationalization fixes for CJK and Unicode.

FlashText i18n

A maintained fork of FlashText with internationalization and Unicode fixes.

Why This Fork?

The original FlashText is no longer actively maintained and has several bugs with international text:

• CJK languages: Adjacent keywords not extracted (Chinese, Japanese, Korean)
• Unicode case folding: Wrong span positions for characters like Turkish İ
• Non-ASCII boundaries: Various edge cases with international characters

This fork aims to fix these issues while maintaining full API compatibility.

Fixed in v3.0.0

CJK Adjacent Keywords

from flashtext import KeywordProcessor

kp = KeywordProcessor()
kp.add_keyword('雅詩蘭黛')  # Estée Lauder
kp.add_keyword('小棕瓶')    # Advanced Night Repair

text = '推薦雅詩蘭黛小棕瓶超好用'
result = kp.extract_keywords(text)
# Original FlashText: ['雅詩蘭黛']  ❌ Missing '小棕瓶'
# FlashText i18n:     ['雅詩蘭黛', '小棕瓶']  ✅ Both extracted!

Installation

pip install flashtext-i18n

Or install from GitHub:

pip install git+https://github.com/termdock/flashtext-i18n.git

Usage

The API is 100% compatible with the original FlashText:

from flashtext import KeywordProcessor

# Create processor
kp = KeywordProcessor()

# Add keywords
kp.add_keyword('Python')
kp.add_keyword('機器學習', 'Machine Learning')

# Extract keywords
text = 'I love Python and 機器學習'
keywords = kp.extract_keywords(text)
# ['Python', 'Machine Learning']

# Extract with span info
keywords_with_span = kp.extract_keywords(text, span_info=True)
# [('Python', 7, 13), ('Machine Learning', 18, 22)]

# Replace keywords
new_text = kp.replace_keywords(text)
# 'I love Python and Machine Learning'

Performance

FlashText uses the Aho-Corasick algorithm with O(n) time complexity, making it extremely fast for keyword extraction from large texts.

Benchmark	FlashText	Regex
1000 keywords, 1M chars	~0.1s	~10s+

Roadmap

See Issues for planned fixes:

• Unicode case folding span fix (Turkish İ, German ß)
• Keywords followed by numbers extraction
• Internationalized word boundary detection
• Indian languages (Devanagari) support

Credits

This project is a fork of FlashText created by Vikash Singh.

The original FlashText algorithm is described in the paper: Replace or Retrieve Keywords In Documents at Scale

License

MIT License - see LICENSE file.

The original copyright belongs to Vikash Singh (2017). This fork is maintained by termdock & Huang Chung Yi.