A spaCy package for the Rust tokenizations library
Project description
spacy-alignments: Align tokenizations for spaCy + transformers
A spaCy package for Yohei Tamura's Rust tokenizations library with Python bindings.
Installation
pip install -U pip setuptools wheel
pip install spacy-alignments
If no binary wheel is available for your platform, you will need to install
Rust in order to build
spacy-alignments
from source.
spacy-alignments vs. pytokenizations
The spacy_alignments
module is a drop-in replacement for tokenizations
:
import spacy_alignments as tokenizations
a2b, b2a = tokenizations.get_alignments(["å", "BC"], ["abc"])
assert a2b == [[0], [0]]
assert b2a == [[0, 1]]
The only difference between this package and the original
pytokenizations
is that it
switches the build system to setuptools-rust
to make it easier for us at
Explosion to build source and binary packages for a wider range of platforms.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
spacy-alignments-0.7.2.tar.gz
(4.6 kB
view hashes)
Built Distributions
Close
Hashes for spacy_alignments-0.7.2-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f866d44a4f4d5d96353aef9364eb665baba342267536c694e2794dae2ba6b7f |
|
MD5 | de33822d28e8680bec5b3d563a432cc2 |
|
BLAKE2b-256 | 3cbb167fea215333e0e64305061fc28cc466a2ba15acd5bd7a2b68c7a5201ffd |
Close
Hashes for spacy_alignments-0.7.2-cp39-cp39-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 68cfe59e7358b50e050321e161adc61c8e05f2052fc58a79bbaedc0e5a607616 |
|
MD5 | 1c89b4c78cfe6459eb0995c486c26355 |
|
BLAKE2b-256 | 6d6b309ec78d026b404d7b4d952f771f0a28e29065e8a4a6bf93e3be00e26b0d |
Close
Hashes for spacy_alignments-0.7.2-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0b49d8b5c4448e4d9ad48d14ba443dfa40139c547d1c39c0f2378d0111021329 |
|
MD5 | b1276608dfe3bd6f31ba9daaf8c768db |
|
BLAKE2b-256 | fd682caaa7a87efa6788d0955052d32945c25cd5833f0785c76fb61a37e1798a |
Close
Hashes for spacy_alignments-0.7.2-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 10df6d7da84e3f12e3465cb44a027c6fbcb5a8623f1d503bf92b65f9c9a5f62b |
|
MD5 | 8a097e4548c7c25f235fc77c0713a264 |
|
BLAKE2b-256 | 5fa349e295986668d15c3e9d4e04befd34b2fb63b84e4517cd6b49d2367a7ce9 |
Close
Hashes for spacy_alignments-0.7.2-cp38-cp38-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 60e493fb6a21f7710d80c646df3da59814c3eeb1e3b54065731794bf2e300072 |
|
MD5 | 3027f24350cca64a5ad7d7cbffeed39d |
|
BLAKE2b-256 | 295ab93244e06e36f4b212dfc43ab705aa1e8e17f77a31e00f3b9a6cc35b486a |
Close
Hashes for spacy_alignments-0.7.2-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c8eca36caebd74c4d8b83f6a3865c6d4f91f19cf3f54e537707b7f581b649a52 |
|
MD5 | 63bf83384c127a426842336e70a715da |
|
BLAKE2b-256 | db585be599f7f2820b836cb368b8ffe68e4b921475900860692b251d2f180361 |
Close
Hashes for spacy_alignments-0.7.2-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 969bfe1efe248760eb374ef05f6592fa81bb4ca02279cb31db6f80f520654033 |
|
MD5 | c794a4b3b38892787690a33b01d3bdb8 |
|
BLAKE2b-256 | 6b888463f55f96083df544510e11fe95001e2b02fc763b853a22567830c48cd6 |
Close
Hashes for spacy_alignments-0.7.2-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c8d31ffcce5ea5594fa9dfa2a0f579bd360fafa92ef66887b4f41a75e1ae03aa |
|
MD5 | 31597233436b0b328c7fdbb4dbad5cfe |
|
BLAKE2b-256 | 42861a428af2429609f5a238e96503fc4367efda4caf2b28734903dc9eb1d92c |
Close
Hashes for spacy_alignments-0.7.2-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 044418dcf6e4e0efd350b40ab5643db543de20bb09d980a4baaf8079bdcb95b7 |
|
MD5 | eee7c08bb7cf22150de2dedd2187c42d |
|
BLAKE2b-256 | c9292e82c5cb519b2be9d224430ceff58c2d139632c341adff8cdc5fc09a3115 |
Close
Hashes for spacy_alignments-0.7.2-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c11882416a5953ff685e0aaef39a86260acbb8b953bf918d38eb987491f0c27 |
|
MD5 | bba05ac4025dfcaf47ec131d628bb3f2 |
|
BLAKE2b-256 | 1c9c9e97063b0f545c9a1140abb8e7f270c6f56b50e3f2e500e2cfa3d4c24f9d |
Close
Hashes for spacy_alignments-0.7.2-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 53dd3c2c07e463ff765c1c29d7a291986b3d772eb19bf17b7d0640cbee8052b6 |
|
MD5 | 9e48e024014bd8699e93251ed407d9ef |
|
BLAKE2b-256 | a15207e51cb8e11dab0dcf203131d7ae66606576e71b71faa7eda01d8e299106 |
Close
Hashes for spacy_alignments-0.7.2-cp36-cp36m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2efd7ceb44e65fa0942da0ac8fe13bead0143d650f9b1c384de1dd5d31c857f |
|
MD5 | 83e69379c75ba8df49252066390ed597 |
|
BLAKE2b-256 | 4bb18dde78863186155a49fb563673f44dbe25a932c9447dee7b7fb571d9419d |