FastText Multithreading Inference
Project description
Multithreading For FastText
An easy to use tool for multithreading fasttext batch inference.
Features
- Fast inference using multithreading utilizing all cpu cores.
- Predicted labels are encoded as i16 numpy array for small memory footprint and easy serialization.
Performance
On macbook Air M2 chip, the performance is about 4 times faster, which is reasonable since it has 4 performance cores.
❯ python -m unittest discover test
fasttext-parallel 1.7130832079565153s
fasttext 6.7440170829650015s
Usage
pip install fasttext-parallel
import fasttext_parallel as ft
model = ft.load_model("./model/lid.176.bin")
# this uses multiple threads
labels, probabilities = model.batch(["你好", "how are you"])
# labels are in a format of numpy.ndarray (i16) format
# to get actual label, call get_label_by_id
assert model.get_label_by_id(labels[0][0]) == "__label__zh"
assert model.get_label_by_id(labels[1][0]) == "__label__en"
# to view all labels (a dict from label_id to label)
print(model.get_labels())
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distributions
Close
Hashes for fasttext_parallel-0.1.4-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 604c2490d45733cdb2402d3c346e215d9195804cb7dd2e5a98ce42c020117903 |
|
MD5 | 9a9539e9007909ea1fe0a30fe8248afd |
|
BLAKE2b-256 | 605ad8f11a49a464f8ca396c4692662dfe714e09dff3d8467486159086e927ef |
Close
Hashes for fasttext_parallel-0.1.4-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5237d01afa9119df0bcd8bd29291d23f95cf0acf9672b0580d773be90a6590f3 |
|
MD5 | 8a0659d39cc5b5c1a612e7b8aec53cad |
|
BLAKE2b-256 | f0c30d5f917f27addae0cc0db6c53368f3de6cb848907ea57e481c819a58fb01 |
Close
Hashes for fasttext_parallel-0.1.4-cp311-cp311-manylinux_2_28_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5e22b12c92b698003cdc8cbd1495bce6c434e6885277e7c09d992170d9ae78f8 |
|
MD5 | 8ae3a6de6734e2787846bdf5384d14ea |
|
BLAKE2b-256 | 0c7d659b0602966a18f7b02d0901a3c9170e11a1ce387607043a6bfc6db72275 |
Close
Hashes for fasttext_parallel-0.1.4-cp311-cp311-manylinux_2_28_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 619b39da581d96ce4dd087c2da89a8f4d6f29ab897de4fab31cf155858f7dbfd |
|
MD5 | 59eb0d2325af24a064b56572e1c25729 |
|
BLAKE2b-256 | 77f1215e331358e57a7b028461fcf2fff3bb25944b8dcadd1116dbf0f12be652 |
Close
Hashes for fasttext_parallel-0.1.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4cd43ee8bbce1c3319c962c7b5996459936fbaee4054a269727e2fc29c463d27 |
|
MD5 | 66d5250ede1d9f9e5513e8212a9a81d5 |
|
BLAKE2b-256 | 01fa4ab8850b1c6f2878e1c1b9acc03713c192dfb5edc8d9ba066807885fc48d |
Close
Hashes for fasttext_parallel-0.1.4-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66f0150ffebf04ed66c8352ee05ad22b43acd648bb5a75c8d59155347b4e7d91 |
|
MD5 | 92e4ca1306eb9ac6a86c4e2a4fe7f0c8 |
|
BLAKE2b-256 | 51bb2993acca4ab1905e114e87eeed14a7aa645f6fc751afa6657300e6212861 |
Close
Hashes for fasttext_parallel-0.1.4-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e4f5f3d5bf8db664638d8847e6feb4e901313627a69f08715be81d48f1d50735 |
|
MD5 | 7239c835b35cb92d1a25fd96d1dca29f |
|
BLAKE2b-256 | 1750481db6fa035e29a96f22a12fb3eebd166e8066d77b2c0ca7de871711e8b1 |
Close
Hashes for fasttext_parallel-0.1.4-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 02fd3d1625c383bf4eef782ff9e5fa1491fc9230b6c7de7a186fe98e13d6be30 |
|
MD5 | 2ecd7b040490113f8e880f6442a3d06b |
|
BLAKE2b-256 | 1730d1c69e5f006daed67209c65eb1444d41f302335f85d545e3a9241571e9a9 |
Close
Hashes for fasttext_parallel-0.1.4-cp310-cp310-manylinux_2_28_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 46b5a0f259034e8baf307f2b2952c8644e57a5a89ad29b5628017957e88c3b53 |
|
MD5 | f507515583e18a1f3f635c4be6ea564c |
|
BLAKE2b-256 | 127788c077775f16744a01bbc4bb89902c9a2d1b4b152ae12b574cb66e9f58f4 |
Close
Hashes for fasttext_parallel-0.1.4-cp310-cp310-manylinux_2_28_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 710431fed042fcde10ce25e4f918a1a21aa3bd3b468f13e48bfbbf5a9446ef52 |
|
MD5 | 608ec14411b8a91c5b7d57d51ca2ade0 |
|
BLAKE2b-256 | 6bb21c368c40d752cc7707b29750d718596bd14d6829a16ee7d54c692eddacb8 |
Close
Hashes for fasttext_parallel-0.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb2891aafe56eeb010506cb4fe9e8959af80dee0ec32b88aa89af425c6a66cd8 |
|
MD5 | 7aa20411cf5035ec358b89e61b5e50dd |
|
BLAKE2b-256 | f3b305bda5db3140e91af10cce7844f1ab45a6386ac19b9fca63cbdc601c54b0 |
Close
Hashes for fasttext_parallel-0.1.4-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 33b806c1ef8035429b59227c5afec5f8c39d4d78207b05ff7d1485cfb753d90c |
|
MD5 | 88c8ddd987de3eaa92b7623e416176f9 |
|
BLAKE2b-256 | c1bd16eb08a7c124cf6c0491ee4a986123428d18aa03b725ab610ca890a57b65 |
Close
Hashes for fasttext_parallel-0.1.4-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3499e5d4542c0ace3110f091c6d0da31a7f67fe2794b98abd2d0ccd4ac8af076 |
|
MD5 | d297f48e56488464b6ac0ef8885640c7 |
|
BLAKE2b-256 | 736358aa111d7f700b7f5c37e8e7731c40d9c562f56e20c213a82e902a688a8e |
Close
Hashes for fasttext_parallel-0.1.4-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec60467af0442dd9c04d067233f046b178c5c3e9ddcfb5bf3b6101773786f84e |
|
MD5 | 24badfdec7d9966f3b6e91adc826e921 |
|
BLAKE2b-256 | a0de59e1cdb804cab23ab949ed9404e4ec5226540fefd5d53de50a3be9728f5a |
Close
Hashes for fasttext_parallel-0.1.4-cp39-cp39-manylinux_2_28_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0a4c7c7c656067edb37427a8bbb4f49d91149240cc7184a54073a7ab62e71b0f |
|
MD5 | 5d0f040fdf485e997d1b6f8fc67037cf |
|
BLAKE2b-256 | 4a49906497d6e1b42b35842ee941bb3734d2a33d2c441f42b5aab3112501b92e |
Close
Hashes for fasttext_parallel-0.1.4-cp39-cp39-manylinux_2_28_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 30b82fbe54088432b63bc237ae54fc07ca6a1f493ecf76b582fc89d934f1a437 |
|
MD5 | ea25decd1156407ee6159a4fd758062d |
|
BLAKE2b-256 | a5fe381c682d96f3fd3abc1e1df62703df98da6679c6e3799b3573be70c4ffdf |
Close
Hashes for fasttext_parallel-0.1.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0d0d6992988d6a43a099d4f016412fe1732265d838e654a7ee490565ab431e72 |
|
MD5 | 5d9ef610d2773a995e1b186043472fa7 |
|
BLAKE2b-256 | ed9f030052ec836346561a5a5e2b9ca5430658c9b642eeae7373a020e5f83e41 |
Close
Hashes for fasttext_parallel-0.1.4-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a0b151d10c6d978d34185075579e15664b437513f9694a88146ddec81e04ca4a |
|
MD5 | 219b09cea029b954be0c406f33e98551 |
|
BLAKE2b-256 | 8d7a5893e6199e3a2662349ac73fe0260358055282597ffe13ff8f06e9080a82 |
Close
Hashes for fasttext_parallel-0.1.4-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8882ee50d183c5a12202d1ccbc78b9cb2a02e8d6b8574be758eea9090a01146a |
|
MD5 | 2bdb9528405d3305fab9d1aa2a21a9d7 |
|
BLAKE2b-256 | beea174519e0b21f44313dfb87de4b2e62dc4d7de9c096a0f998401c5970dbbc |
Close
Hashes for fasttext_parallel-0.1.4-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 97bc4f98be830c10dce6b41bd10fbd6a2ececc0d4fa399ac9612764c6d06392e |
|
MD5 | 996fe26c5ec233464b82dbcc068acdc0 |
|
BLAKE2b-256 | de1146ec299e02a7b7a39b393f9437bb60502f0b6ca3ef2ed77908576a1d22d1 |
Close
Hashes for fasttext_parallel-0.1.4-cp38-cp38-manylinux_2_28_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 648178c886d67d7ad6d3b3008b06c1e99a9b56d94b31f717689bde9f68ea8ad4 |
|
MD5 | 475f6dff6f7cebc375a03ef812de4fc8 |
|
BLAKE2b-256 | a4647c0244c0f216c49dc45c335152b50f93a0481e8e257c951e480a3dfceaef |
Close
Hashes for fasttext_parallel-0.1.4-cp38-cp38-manylinux_2_28_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0039832eb290c8a168dad451a0a937dd1c39ff50454809f27752f4a61fdb7a0d |
|
MD5 | b3dcd62d44ddd213d3c1805bf131bb62 |
|
BLAKE2b-256 | 5a016273742a52ae6cbf96d4e783748fd8769532b11dcfa3d69d560c00431369 |
Close
Hashes for fasttext_parallel-0.1.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c7b423a442d4f56f006c00c532abc7a8cc72719e56ff74fade02320936bd4fba |
|
MD5 | e8b486db429e4ac2442d67006b51015d |
|
BLAKE2b-256 | d7e0624cb60be93c1f1bd4580bccc1116ca28335702cd8d7ea76af40ee5dc7e1 |
Close
Hashes for fasttext_parallel-0.1.4-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f77a7ca4d84f2db05bb8f5959ff8678392c54628b6cfb2732d2faee5131feac |
|
MD5 | 13213f02872af2143b49395e7fd0618f |
|
BLAKE2b-256 | 577d71691cb30b361961a46fe459d0f39114cbb32f3bae1e7b4db7b81a03f351 |
Close
Hashes for fasttext_parallel-0.1.4-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8272aeb83860247cb30a45cf6e06e1acbeee2c44db36f5af0b6a65e562665ceb |
|
MD5 | d36871c1910ece37e3991804d601c21c |
|
BLAKE2b-256 | 3a42a5b732d2609df8608241443caab01610d49a3290b2d38b1e68d1cb475c95 |
Close
Hashes for fasttext_parallel-0.1.4-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0180c81bdc7fc5a68e7ab59730454402467e2942334ee2643dc3c9487e4e65a6 |
|
MD5 | 0a7d1fd156c339d05597ea9dacb5acde |
|
BLAKE2b-256 | 356abbd1062a0d0c9e5a7a596e3c634825a2d91233767f0cbef15bbd094dc685 |
Close
Hashes for fasttext_parallel-0.1.4-cp37-cp37m-manylinux_2_28_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7bffddce5a255699175949cd5736ac0208bc364e61f716c2f5ed0e618ca63dcc |
|
MD5 | ecb7a3d0e4b0f3f9f6d815e9c4b7e595 |
|
BLAKE2b-256 | 39babb2f1b7199d0b7ac86ce2c9b17ebb28936e1d3918367391efd50f32985b4 |
Close
Hashes for fasttext_parallel-0.1.4-cp37-cp37m-manylinux_2_28_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 38edf3bda1194676f2b4c6d53867093c7c349e014c2d736625cf0cd1b00002a2 |
|
MD5 | 7dec4061643d6471b5a83897fb9a32b2 |
|
BLAKE2b-256 | eeeb66462350620822bdbe62f83e371d26de9b3b3f356bb1782cf4d52e0527fb |
Close
Hashes for fasttext_parallel-0.1.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a123e39a23368320efd60d48c5b994ac7009995e6caa852e80a3452209bb63ec |
|
MD5 | 5643e5e5847b82fca15eb067d2b888f7 |
|
BLAKE2b-256 | 9fbb7ef99b008e99605248ec7de91d59242856c2011f8283a29eaa97f0bd5b60 |
Close
Hashes for fasttext_parallel-0.1.4-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7db89e1682ca5f44a382c7094222764f32050d3adb269540592ee5fd141383b4 |
|
MD5 | 982b66bb4a474856f0c23b3317af0ac4 |
|
BLAKE2b-256 | f2e57cc8b814eabb0f18920c25f37746c51b687b26f0eb9bffbfa8a33a052736 |