Blazingly fast Word Matcher
Project description
Matcher Rust Implementation with PyO3 Binding
A high-performance, multi-functional word matcher implemented in Rust.
Designed to solve AND OR NOT and TEXT VARIATIONS problems in word/word_list matching. For detailed implementation, see the Design Document.
Features
- Multiple Matching Methods:
- Simple Word Matching
- Regex-Based Matching
- Similarity-Based Matching
- Text Normalization:
- Fanjian: Simplify traditional Chinese characters to simplified ones.
Example:
蟲艸
->虫草
- Delete: Remove specific characters.
Example:
*Fu&*iii&^%%*&kkkk
->Fuiiikkkk
- Normalize: Normalize special characters to identifiable characters.
Example:
𝜢𝕰𝕃𝙻Ϙ 𝙒ⓞƦℒ𝒟!
->hello world!
- PinYin: Convert Chinese characters to Pinyin for fuzzy matching.
Example:
西安
->/xi//an/
, matches洗按
->/xi//an/
, but not先
->/xian/
- PinYinChar: Convert Chinese characters to Pinyin.
Example:
西安
->xian
, matches洗按
and先
->xian
- Fanjian: Simplify traditional Chinese characters to simplified ones.
Example:
- AND OR NOT Word Matching:
- Takes into account the number of repetitions of words.
- Example:
hello&world
matcheshello world
andworld,hello
- Example:
无&法&无&天
matches无无法天
(because无
is repeated twice), but not无法天
- Example:
hello~helloo~hhello
matcheshello
but nothelloo
andhhello
- Customizable Exemption Lists: Exclude specific words from matching.
- Efficient Handling of Large Word Lists: Optimized for performance.
Installation
Use pip
pip install matcher_py
Install pre-built binary
Visit the release page to download the pre-built binary.
Usage
The msgspec
library is recommended for serializing the matcher configuration due to its performance benefits. You can also use other msgpack serialization libraries like ormsgpack
. All relevant types are defined in extension_types.py.
Explanation of the configuration
Matcher
's configuration is defined by theMatchTableMap = Dict[int, List[MatchTable]]
type, the key ofMatchTableMap
is calledmatch_id
, for eachmatch_id
, thetable_id
inside should but isn't required to be unique.SimpleMatcher
's configuration is defined by theSimpleMatchTableMap = Dict[SimpleMatchType, Dict[int, str]]
type, the valueDict[int, str]
's key is calledword_id
,word_id
is required to be globally unique.
MatchTable
table_id
: The unique ID of the match table.match_table_type
: The type of the match table.word_list
: The word list of the match table.exemption_simple_match_type
: The type of the exemption simple match.exemption_word_list
: The exemption word list of the match table.
For each match table, word matching is performed over the word_list
, and exemption word matching is performed over the exemption_word_list
. If the exemption word matching result is True, the word matching result will be False.
MatchTableType
Simple
: Supports simple multiple patterns matching with text normalization defined bysimple_match_type
.- We offer transformation methods for text normalization, including
Fanjian
,Normalize
,PinYin
···. - It can handle combination patterns and repeated times sensitive matching, delimited by
&
, such ashello&world&hello
will matchhellohelloworld
andworldhellohello
, but nothelloworld
due to the repeated times ofhello
.
- We offer transformation methods for text normalization, including
Regex
: Supports regex patterns matching.SimilarChar
: Supports similar character matching using regex.["hello,hallo,hollo,hi", "word,world,wrd,🌍", "!,?,~"]
will matchhelloworld!
,hollowrd?
,hi🌍~
··· any combinations of the words split by,
in the list.
Acrostic
: Supports acrostic matching using regex (currently only supports Chinese and simple English sentences).["h,e,l,l,o", "你,好"]
will matchhope, endures, love, lasts, onward.
and你的笑容温暖, 好心情常伴。
.
Regex
: Supports regex matching.["h[aeiou]llo", "w[aeiou]rd"]
will matchhello
,world
,hillo
,wurld
··· any text that matches the regex in the list.
Similar
: Supports similar text matching based on distance and threshold.Levenshtein
: Supports similar text matching based on Levenshtein distance.DamerauLevenshtein
: Supports similar text matching based on Damerau-Levenshtein distance.Indel
: Supports similar text matching based on Indel distance.Jaro
: Supports similar text matching based on Jaro distance.JaroWinkler
: Supports similar text matching based on Jaro-Winkler distance.
SimpleMatchType
None
: No transformation.Fanjian
: Traditional Chinese to simplified Chinese transformation. Based on FANJIAN.妳好
->你好
現⾝
->现身
Delete
: Delete all punctuation, special characters and white spaces.hello, world!
->helloworld
《你∷好》
->你好
Normalize
: Normalize all English character variations and number variations to basic characters. Based on SYMBOL_NORM, NORM and NUM_NORM.ℋЀ⒈㈠Õ
->he11o
⒈Ƨ㊂
->123
PinYin
: Convert all unicode Chinese characters to pinyin with boundaries. Based on PINYIN.你好
->␀ni␀␀hao␀
西安
->␀xi␀␀an␀
PinYinChar
: Convert all unicode Chinese characters to pinyin without boundaries. Based on PINYIN_CHAR.你好
->nihao
西安
->xian
You can combine these transformations as needed. Pre-defined combinations like DeleteNormalize
and FanjianDeleteNormalize
are provided for convenience.
Avoid combining PinYin
and PinYinChar
due to that PinYin
is a more limited version of PinYinChar
, in some cases like xian
, can be treat as two words xi
and an
, or only one word xian
.
Delete
is technologically a combination of TextDelete
and WordDelete
, we implement different delete methods for text and word. 'Cause we believe CN_SPECIAL
and EN_SPECIAL
are parts of the word, but not for text. For text_process
and reduce_text_process
functions, users should use TextDelete
instead of WordDelete
.
WordDelete
: Delete all patterns inWHITE_SPACE
.TextDelete
: Delete all patterns in TEXT_DELETE.
Text Process Usage
Here’s an example of how to use the reduce_text_process
and text_process
functions:
from matcher_py import reduce_text_process, text_process
from matcher_py.extension_types import SimpleMatchType
print(reduce_text_process(SimpleMatchType.MatchTextDelete | SimpleMatchType.MatchNormalize, "hello, world!"))
print(text_process(SimpleMatchType.MatchTextDelete, "hello, world!"))
Matcher Basic Usage
Here’s an example of how to use the Matcher
:
import msgspec
import numpy as np
from matcher_py import Matcher
from matcher_py.extension_types import MatchTable, MatchTableType, SimpleMatchType
msgpack_encoder = msgspec.msgpack.Encoder()
matcher = Matcher(
msgpack_encoder.encode({
1: [
MatchTable(
table_id=1,
match_table_type=MatchTableType.Simple(simple_match_type = SimpleMatchType.MatchFanjianDeleteNormalize),
word_list=["hello", "world"],
exemption_simple_match_type=SimpleMatchType.MatchNone,
exemption_word_list=["word"],
)
]
})
)
# Check if a text matches
assert matcher.is_match("hello")
assert not matcher.is_match("hello, word")
# Perform word matching as a dict
assert matcher.word_match(r"hello, world")[1]
# Perform word matching as a string
result = matcher.word_match_as_string("hello")
assert result == """{1:[{\"table_id\":1,\"word\":\"hello\"}]"}"""
# Perform batch processing as a dict using a list
text_list = ["hello", "world", "hello,word"]
batch_results = matcher.batch_word_match(text_list)
print(batch_results)
# Perform batch processing as a string using a list
text_list = ["hello", "world", "hello,word"]
batch_results = matcher.batch_word_match_as_string(text_list)
print(batch_results)
# Perform batch processing as a dict using a numpy array
text_array = np.array(["hello", "world", "hello,word"], dtype=np.dtype("object"))
numpy_results = matcher.numpy_word_match(text_array)
print(numpy_results)
# Perform batch processing as a string using a numpy array
text_array = np.array(["hello", "world", "hello,word"], dtype=np.dtype("object"))
numpy_results = matcher.numpy_word_match_as_string(text_array)
print(numpy_results)
Simple Matcher Basic Usage
Here’s an example of how to use the SimpleMatcher
:
import msgspec
import numpy as np
from matcher_py import SimpleMatcher
from matcher_py.extension_types import SimpleMatchType
msgpack_encoder = msgspec.msgpack.Encoder()
simple_matcher = SimpleMatcher(
msgpack_encoder.encode({SimpleMatchType.MatchNone: {1: "example"}})
)
# Check if a text matches
assert simple_matcher.is_match("example")
# Perform simple processing
results = simple_matcher.simple_process("example")
print(results)
# Perform batch processing using a list
text_list = ["example", "test", "example test"]
batch_results = simple_matcher.batch_simple_process(text_list)
print(batch_results)
# Perform batch processing using a NumPy array
text_array = np.array(["example", "test", "example test"], dtype=np.dtype("object"))
numpy_results = simple_matcher.numpy_simple_process(text_array)
print(numpy_results)
Contributing
Contributions to matcher_py
are welcome! If you find a bug or have a feature request, please open an issue on the GitHub repository. If you would like to contribute code, please fork the repository and submit a pull request.
License
matcher_py
is licensed under the MIT OR Apache-2.0 license.
More Information
For more details, visit the GitHub repository.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for matcher_py-0.4.1-cp38-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b9a0fd526fafd20f47439420eb88fa9e22ba4aefd6cb3fe6e9d3f96ae2c4af1 |
|
MD5 | 2690c8e19fee2e6ff2bcbaa0d88e2ee0 |
|
BLAKE2b-256 | 4e52244b1e4731f248565db0d17bba1c68534e14fea9ead7b003968f3f183460 |
Hashes for matcher_py-0.4.1-cp38-abi3-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 42d88e7fb6e3109f940ecf80bda787e53df104e660daf4eb006b7ce2ca104617 |
|
MD5 | 0691e4213bee8e37c2c7b51317e1f909 |
|
BLAKE2b-256 | 4912b1d4c58437f9c0c132b7c0558cb256427cb0adde7e03e68e5a7058000c35 |
Hashes for matcher_py-0.4.1-cp38-abi3-musllinux_1_2_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c8dad0441c5fb97aec1d9960e376522a144eaf251128d039653e04b5f2f49f5 |
|
MD5 | c3b027ffd8183861aa675a893a40c0c7 |
|
BLAKE2b-256 | ac8442e39197ab9383d06f934442286da87102971ea0c8a3319c87f1fd846126 |
Hashes for matcher_py-0.4.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 90a94f91e74b37a2f4610dcc481e42bc907b7614fae37f54c06556e5166e610b |
|
MD5 | d6df4423b402b0e5befc172280c352fb |
|
BLAKE2b-256 | 2f167b45a165cf8936baf10669b82e5153beb4e5ffc81b6505c415c5b78e467a |
Hashes for matcher_py-0.4.1-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0de010763e8e67e1fc06a7eef7f16e2bf013d4005393089bc5cbca3c87e2e3bd |
|
MD5 | 3b3291f5a7ed324b8ced5113a8317aa9 |
|
BLAKE2b-256 | bfc1580113e772b4042e369895b1229358977c4d444864f40573ce5ca710a01c |
Hashes for matcher_py-0.4.1-cp38-abi3-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | df1634ba4f6b866a9b3e2287f47ecb8d4bb609699367f985fc5dc4d931c62219 |
|
MD5 | 7ddd0757a4787ed6224082137ae370c2 |
|
BLAKE2b-256 | 1ced974162c38acdd56a90fb2b27256e9e3cbb1e2f0f374a00115acccff724d6 |
Hashes for matcher_py-0.4.1-cp38-abi3-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7491645a2a36d0851bd783adeaae6c8a6ee7bc0e6aec5a9ebd4ec58fda06d362 |
|
MD5 | da2ba530d345d86734d4813e2a721c38 |
|
BLAKE2b-256 | 7ecee96a17f1a97d2ef81b781318fa55cfc70b4fd9b7c48725b304c78ab3c98b |