A spaCy package for the Rust tokenizations library
Project description
spacy-alignments: Align tokenizations for spaCy + transformers
A spaCy package for Yohei Tamura's Rust tokenizations library with Python bindings.
Installation
pip install -U pip setuptools wheel
pip install spacy-alignments
If no binary wheel is available for your platform, you will need to install
Rust in order to build
spacy-alignments
from source.
spacy-alignments vs. pytokenizations
The spacy_alignments
module is a drop-in replacement for tokenizations
:
import spacy_alignments as tokenizations
a2b, b2a = tokenizations.get_alignments(["å", "BC"], ["abc"])
assert a2b == [[0], [0]]
assert b2a == [[0, 1]]
The only difference between this package and the original
pytokenizations
is that it
switches the build system to setuptools-rust
to make it easier for us at
Explosion to build source and binary packages for a wider range of platforms.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
spacy-alignments-0.8.3.tar.gz
(4.6 kB
view hashes)
Built Distributions
Close
Hashes for spacy_alignments-0.8.3-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a12e32cf602d77941c737d182b91142409a417f342b10aa56a8762b47a3545da |
|
MD5 | 0c469c2e4e5d37756340483892fa6089 |
|
BLAKE2b-256 | ae4204523e2cd239c3dd390db4adef81f9475162e53daa51ecef42a04bd4c37f |
Close
Hashes for spacy_alignments-0.8.3-cp39-cp39-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77590f41bfc5a2675578aadf6879ae4d219da084ec6446a40c09baf87f8db95a |
|
MD5 | 0df7683ab18fcf121286bb5372cf8e12 |
|
BLAKE2b-256 | e11c98257cae5292ccfd7e786419752220ea7c3a3bad8a393143044bdd712584 |
Close
Hashes for spacy_alignments-0.8.3-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 75b5f00ed7b4de5c216bb5e4cf2dc4f3d5763a35b3ff80cd1510c29ba8d43c78 |
|
MD5 | f3602c18c3ea1e393ec233db5e421769 |
|
BLAKE2b-256 | 4180b8270ea35c71a0f7b3c7b5c7c0d747b0518439e6b4d659f3a69f509d613b |
Close
Hashes for spacy_alignments-0.8.3-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67a80e0241b1f8ea026febf95a7881265f59740ade4062ac28b652402ffceaff |
|
MD5 | 1a1db2cdb6307e74410b176d40bfbb2e |
|
BLAKE2b-256 | e92fbea4f1ee51c5070fcccccdc523a1d1eefdf3ad25bdb05a13ae8efac0d47f |
Close
Hashes for spacy_alignments-0.8.3-cp38-cp38-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67eaf8a21f1c92fef34fb861a0ca2e79bc8b9fc95ec5071352f5bf5659c193a1 |
|
MD5 | da2bbdccc8f6a96d9c2fcd1ed4225e87 |
|
BLAKE2b-256 | 738a6657b4faefa6322b78cea13a39c96c0eb55b67a461e4b16e33ed69efc300 |
Close
Hashes for spacy_alignments-0.8.3-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6e6fc509528a927371860d52b38481ab9ad268ef840ed8bd46346e9659c67527 |
|
MD5 | 7affdffb15e08b92c6bfb45ac6759396 |
|
BLAKE2b-256 | ca79e110554fb547162902fdb398c84d67cbb739279c22633ec606daa0bcc8e0 |
Close
Hashes for spacy_alignments-0.8.3-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9ae3fa64510999496db9f3745c6eebbaec775d7bf21d97afedf08bd7925be360 |
|
MD5 | c6a6e9dcbbe8c3ba2be2d037bf50317f |
|
BLAKE2b-256 | c650ed9f92d1c3dc6cf6c45b4bebbc8ec7c54ce4ada8ea9d1e52d12f3200cbde |
Close
Hashes for spacy_alignments-0.8.3-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd0fcee1eef4bbc6fa71fa6a3e6318e4b6f2cab08c1122993d2bc92b10d0e595 |
|
MD5 | f76088191745b87725cf21e88f7d2cbf |
|
BLAKE2b-256 | ea86a6786d24d1d8f3a6cff2c60b55a7e845725a94919cd94d270ea49d82e59b |
Close
Hashes for spacy_alignments-0.8.3-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af2a55bff7169ffd6bd6289eec5cdb09bc57fe4fb7417216f84bca138b955e95 |
|
MD5 | 7efa7a4b05299ad48e912cc3c6c4d5e2 |
|
BLAKE2b-256 | 6ef5d027a0a636faead33c38cf20dec6c18d2d4fc0caf54b5644d9f58913a0a7 |
Close
Hashes for spacy_alignments-0.8.3-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2490188847453893255bc6b399f42388981d492581f131b9fab76a58996ecdbb |
|
MD5 | 7004b30bbbf41649ebb707ec80a16b70 |
|
BLAKE2b-256 | 9d13bf8960b4035a4908cd6db5d73d899508560178457ba578e94b3e309466dc |
Close
Hashes for spacy_alignments-0.8.3-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ddb5cdc4915b77e9b0e7a2204fc0705834d5e12d0a0476f43a89e6bad204a3b3 |
|
MD5 | 41e274b7270a3fd91955940e9e803166 |
|
BLAKE2b-256 | f98c76c765717015a0a47501478fce9aa351c2ad14365a8a1bd845f687145881 |
Close
Hashes for spacy_alignments-0.8.3-cp36-cp36m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 729635b150aa6bb5a93a2b7a058e02d7322e224a6a079bbf43b4c630410da245 |
|
MD5 | 7af458d7ccbc83b222aae8c9485f3ef1 |
|
BLAKE2b-256 | 742e9674d0679d72735a49f66994dab8a4a81fe95aa9d639151a2cf0aea7fb0c |