spacy-alignments: Align tokenizations for spaCy + transformers
A spaCy package for Yohei Tamura's Rust tokenizations library with Python bindings.
Installation
pip install -U pip setuptools wheel
pip install spacy-alignments
If no binary wheel is available for your platform, you will need to install
Rust in order to build
spacy-alignments from source.
spacy-alignments vs. pytokenizations
The spacy_alignments module is a drop-in replacement for tokenizations:
import spacy_alignments as tokenizations
a2b, b2a = tokenizations.get_alignments(["å", "BC"], ["abc"])
assert a2b == [[0], [0]]
assert b2a == [[0, 1]]
The only difference between this package and the original
pytokenizations is that it
switches the build system to setuptools-rust to make it easier for us at
Explosion to build source and binary packages for a wider range of platforms.
1 maintainer
Metadata
1 maintainer
Release files for spacy-alignments 0.8.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| spacy-alignments-0.8.3.tar.gz | 4.6 kB | Details |
Built distributions (wheels)
Total release size: 5.9 MB
Release files / spacy-alignments-0.8.3.tar.gz
| Download URL | spacy-alignments-0.8.3.tar.gz |
|---|---|
| Size | 4.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ceba818da223b45e9b24c6e6c3b668f81780a4deacc717cb92bcec0e376f0bbf
|
|
BLAKE2b-256 checksum How to use checksums |
2cfd5e5cf8af68b5e2cf39660246463b17aeed67c8b3ab7e1de88d420ecdc4aa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
Release files / spacy_alignments-0.8.3-cp39-cp39-win_amd64.whl
| Download URL | spacy_alignments-0.8.3-cp39-cp39-win_amd64.whl |
|---|---|
| Size | 179.4 kB |
| Tags | CPython 3.9 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
a12e32cf602d77941c737d182b91142409a417f342b10aa56a8762b47a3545da
|
|
BLAKE2b-256 checksum How to use checksums |
ae4204523e2cd239c3dd390db4adef81f9475162e53daa51ecef42a04bd4c37f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
Release files / spacy_alignments-0.8.3-cp39-cp39-manylinux2014_x86_64.whl
| Download URL | spacy_alignments-0.8.3-cp39-cp39-manylinux2014_x86_64.whl |
|---|---|
| Size | 998.3 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
77590f41bfc5a2675578aadf6879ae4d219da084ec6446a40c09baf87f8db95a
|
|
BLAKE2b-256 checksum How to use checksums |
e11c98257cae5292ccfd7e786419752220ea7c3a3bad8a393143044bdd712584
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
Release files / spacy_alignments-0.8.3-cp39-cp39-macosx_10_9_x86_64.whl
| Download URL | spacy_alignments-0.8.3-cp39-cp39-macosx_10_9_x86_64.whl |
|---|---|
| Size | 302.1 kB |
| Tags | CPython 3.9 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
75b5f00ed7b4de5c216bb5e4cf2dc4f3d5763a35b3ff80cd1510c29ba8d43c78
|
|
BLAKE2b-256 checksum How to use checksums |
4180b8270ea35c71a0f7b3c7b5c7c0d747b0518439e6b4d659f3a69f509d613b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
Release files / spacy_alignments-0.8.3-cp38-cp38-win_amd64.whl
| Download URL | spacy_alignments-0.8.3-cp38-cp38-win_amd64.whl |
|---|---|
| Size | 179.4 kB |
| Tags | CPython 3.8 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
67a80e0241b1f8ea026febf95a7881265f59740ade4062ac28b652402ffceaff
|
|
BLAKE2b-256 checksum How to use checksums |
e92fbea4f1ee51c5070fcccccdc523a1d1eefdf3ad25bdb05a13ae8efac0d47f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
Release files / spacy_alignments-0.8.3-cp38-cp38-manylinux2014_x86_64.whl
| Download URL | spacy_alignments-0.8.3-cp38-cp38-manylinux2014_x86_64.whl |
|---|---|
| Size | 998.3 kB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
67eaf8a21f1c92fef34fb861a0ca2e79bc8b9fc95ec5071352f5bf5659c193a1
|
|
BLAKE2b-256 checksum How to use checksums |
738a6657b4faefa6322b78cea13a39c96c0eb55b67a461e4b16e33ed69efc300
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
Release files / spacy_alignments-0.8.3-cp38-cp38-macosx_10_9_x86_64.whl
| Download URL | spacy_alignments-0.8.3-cp38-cp38-macosx_10_9_x86_64.whl |
|---|---|
| Size | 302.1 kB |
| Tags | CPython 3.8 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
6e6fc509528a927371860d52b38481ab9ad268ef840ed8bd46346e9659c67527
|
|
BLAKE2b-256 checksum How to use checksums |
ca79e110554fb547162902fdb398c84d67cbb739279c22633ec606daa0bcc8e0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
Release files / spacy_alignments-0.8.3-cp37-cp37m-win_amd64.whl
| Download URL | spacy_alignments-0.8.3-cp37-cp37m-win_amd64.whl |
|---|---|
| Size | 179.4 kB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Windows x86-64 |
|
SHA-256 checksum How to use checksums |
9ae3fa64510999496db9f3745c6eebbaec775d7bf21d97afedf08bd7925be360
|
|
BLAKE2b-256 checksum How to use checksums |
c650ed9f92d1c3dc6cf6c45b4bebbc8ec7c54ce4ada8ea9d1e52d12f3200cbde
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
Release files / spacy_alignments-0.8.3-cp37-cp37m-manylinux2014_x86_64.whl
| Download URL | spacy_alignments-0.8.3-cp37-cp37m-manylinux2014_x86_64.whl |
|---|---|
| Size | 998.4 kB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
cd0fcee1eef4bbc6fa71fa6a3e6318e4b6f2cab08c1122993d2bc92b10d0e595
|
|
BLAKE2b-256 checksum How to use checksums |
ea86a6786d24d1d8f3a6cff2c60b55a7e845725a94919cd94d270ea49d82e59b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
Release files / spacy_alignments-0.8.3-cp37-cp37m-macosx_10_9_x86_64.whl
| Download URL | spacy_alignments-0.8.3-cp37-cp37m-macosx_10_9_x86_64.whl |
|---|---|
| Size | 302.1 kB |
| Tags | CPython 3.7 CPython 3.7 pymalloc macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
af2a55bff7169ffd6bd6289eec5cdb09bc57fe4fb7417216f84bca138b955e95
|
|
BLAKE2b-256 checksum How to use checksums |
6ef5d027a0a636faead33c38cf20dec6c18d2d4fc0caf54b5644d9f58913a0a7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
Release files / spacy_alignments-0.8.3-cp36-cp36m-win_amd64.whl
| Download URL | spacy_alignments-0.8.3-cp36-cp36m-win_amd64.whl |
|---|---|
| Size | 179.5 kB |
| Tags | CPython 3.6 CPython 3.6 pymalloc Windows x86-64 |
|
SHA-256 checksum How to use checksums |
2490188847453893255bc6b399f42388981d492581f131b9fab76a58996ecdbb
|
|
BLAKE2b-256 checksum How to use checksums |
9d13bf8960b4035a4908cd6db5d73d899508560178457ba578e94b3e309466dc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
Release files / spacy_alignments-0.8.3-cp36-cp36m-manylinux2014_x86_64.whl
| Download URL | spacy_alignments-0.8.3-cp36-cp36m-manylinux2014_x86_64.whl |
|---|---|
| Size | 998.6 kB |
| Tags | CPython 3.6 CPython 3.6 pymalloc Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
ddb5cdc4915b77e9b0e7a2204fc0705834d5e12d0a0476f43a89e6bad204a3b3
|
|
BLAKE2b-256 checksum How to use checksums |
f98c76c765717015a0a47501478fce9aa351c2ad14365a8a1bd845f687145881
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
Release files / spacy_alignments-0.8.3-cp36-cp36m-macosx_10_9_x86_64.whl
| Download URL | spacy_alignments-0.8.3-cp36-cp36m-macosx_10_9_x86_64.whl |
|---|---|
| Size | 302.4 kB |
| Tags | CPython 3.6 CPython 3.6 pymalloc macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
729635b150aa6bb5a93a2b7a058e02d7322e224a6a079bbf43b4c630410da245
|
|
BLAKE2b-256 checksum How to use checksums |
742e9674d0679d72735a49f66994dab8a4a81fe95aa9d639151a2cf0aea7fb0c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.9
|
