spacy-alignments: Align tokenizations for spaCy + transformers
A spaCy package for Yohei Tamura's Rust tokenizations library with Python bindings.
Installation
pip install -U pip setuptools wheel
pip install spacy-alignments
If no binary wheel is available for your platform, you will need to install
Rust in order to build
spacy-alignments from source.
spacy-alignments vs. pytokenizations
The spacy_alignments module is a drop-in replacement for tokenizations:
import spacy_alignments as tokenizations
a2b, b2a = tokenizations.get_alignments(["å", "BC"], ["abc"])
assert a2b == [[0], [0]]
assert b2a == [[0, 1]]
The only difference between this package and the original
pytokenizations is that it
switches the build system to setuptools-rust to make it easier for us at
Explosion to build source and binary packages for a wider range of platforms.
1 maintainer
Metadata
1 maintainer
Release files for spacy-alignments 0.8.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| spacy-alignments-0.8.4.tar.gz | 6.9 kB | Details |
Built distributions (wheels)
Total release size: 7.9 MB
Release files / spacy-alignments-0.8.4.tar.gz
| Download URL | spacy-alignments-0.8.4.tar.gz |
|---|---|
| Size | 6.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d4702997f459d30e537f638fbb540151adfab88aa2969f9d0db3e3ba39f47bdb
|
|
BLAKE2b-256 checksum How to use checksums |
5b322c22e7e4e8ceb0ee04d5b3aa24379f1b30835a11690d883b5a0acc521928
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp310-cp310-win_amd64.whl
| Download URL | spacy_alignments-0.8.4-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 183.8 kB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
3b5cd5beb15dd34d72304b5d07dbedd0676c4cbb018b2ad1994ee8560260810e
|
|
BLAKE2b-256 checksum How to use checksums |
e01df9806cd6249cf04176c788387b6518fec67a73d3ed23aadaf2dd3fc13cd4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | spacy_alignments-0.8.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 1.1 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
32400cc9eff8e4ea9d61d3f6974df979a2c272ca8e9cd65e2811e0aa1b4598ad
|
|
BLAKE2b-256 checksum How to use checksums |
b982e6bfa8f383938325e7d1f90e0873e7fb770b768de37058b9b1958629f897
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp310-cp310-macosx_10_9_x86_64.whl
| Download URL | spacy_alignments-0.8.4-cp310-cp310-macosx_10_9_x86_64.whl |
|---|---|
| Size | 309.1 kB |
| Tags | CPython 3.10 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
fde0b20a9c629b07ce7bd3ccb31d1b2ddd465faee1a6f5c09ed1ef9cf07b8609
|
|
BLAKE2b-256 checksum How to use checksums |
a6476205f947e7439489363586e35bd5321779849a4faa8a0a52ff5c6ecde4f5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp39-cp39-win_amd64.whl
| Download URL | spacy_alignments-0.8.4-cp39-cp39-win_amd64.whl |
|---|---|
| Size | 183.8 kB |
| Tags | CPython 3.9 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
e43ca54c61ab97fc98017c41f6b3b0e016d195a2bfbb3f449cb77d552a842798
|
|
BLAKE2b-256 checksum How to use checksums |
525173dd22dc6ef07c7fe5b1b8c2381db13a6b15f05b89cc20261148dfc7ce31
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | spacy_alignments-0.8.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 1.1 MB |
| Tags | CPython 3.9 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
27f0816db4426dfc4954d506c8855618b6589274edce658b84813d83a2f6ed45
|
|
BLAKE2b-256 checksum How to use checksums |
c0e5605cc8c249160b06ad645513d3cb6aa9645e1dfc796ef964d0fa4c0bb43a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp39-cp39-macosx_10_9_x86_64.whl
| Download URL | spacy_alignments-0.8.4-cp39-cp39-macosx_10_9_x86_64.whl |
|---|---|
| Size | 309.1 kB |
| Tags | CPython 3.9 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
7111b2bca1fcd8e61ef8a508e5b25d1ca4a819063ae61b6ceb50074a8c917848
|
|
BLAKE2b-256 checksum How to use checksums |
f70926f8b344430100ff2d0d8eec594bb794f38db72b9dfc78c612cc7d86824c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp38-cp38-win_amd64.whl
| Download URL | spacy_alignments-0.8.4-cp38-cp38-win_amd64.whl |
|---|---|
| Size | 183.8 kB |
| Tags | CPython 3.8 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
3eb2f35cee5d7cbc5c5b156a0a97379537186911bd9768fbb262b7e6818137a3
|
|
BLAKE2b-256 checksum How to use checksums |
78092dffc5395983d3950db16bfcaa8dd645f4399f94cb77e96fbf67be088ee3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | spacy_alignments-0.8.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 1.1 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
43b11507bb058cb29da454cbe8bb2727dcd3fc83398bcb6bf6bc28da5f1b0bae
|
|
BLAKE2b-256 checksum How to use checksums |
665b7ff4855a27f59721d1d64c5bf1518f2e9e2f52e9ac4a24bab8f7cd6080f4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp38-cp38-macosx_10_9_x86_64.whl
| Download URL | spacy_alignments-0.8.4-cp38-cp38-macosx_10_9_x86_64.whl |
|---|---|
| Size | 309.1 kB |
| Tags | CPython 3.8 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
09ca14a8a43022f067521bb976c3a406edaff7e45ac2c61692a4a84e45d7c630
|
|
BLAKE2b-256 checksum How to use checksums |
5cc0435ee960c1fa899ed531dd396ade5882f2cf1847cb01e47d230763a72cf7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp37-cp37m-win_amd64.whl
| Download URL | spacy_alignments-0.8.4-cp37-cp37m-win_amd64.whl |
|---|---|
| Size | 183.8 kB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Windows x86-64 |
|
SHA-256 checksum How to use checksums |
382f487c74533540ff9f82b6a7369d04b74dec6a3a2ddfc9d6f835eddc40b5f0
|
|
BLAKE2b-256 checksum How to use checksums |
6fefea94624324a4861e155291c215b286a4daecaff0df96faa1f5af674d097e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | spacy_alignments-0.8.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 1.1 MB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
306c2827479ff3b2c377fd8e3cd20c5e9dac2a2d6e6034e62cae117994b56685
|
|
BLAKE2b-256 checksum How to use checksums |
72aeac87fe358846f4e7254b25be3c20904f80832ab7f206fd331cefd28badce
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp37-cp37m-macosx_10_9_x86_64.whl
| Download URL | spacy_alignments-0.8.4-cp37-cp37m-macosx_10_9_x86_64.whl |
|---|---|
| Size | 309.1 kB |
| Tags | CPython 3.7 CPython 3.7 pymalloc macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
327ee5e7c6dc82a0ce915b79f431d3119b74b2e94faa0a0059723e072c915e6d
|
|
BLAKE2b-256 checksum How to use checksums |
25542083f44330ffdb80913e0e62b5d11baa98287a912228d41eb8af26cd728d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp36-cp36m-win_amd64.whl
| Download URL | spacy_alignments-0.8.4-cp36-cp36m-win_amd64.whl |
|---|---|
| Size | 184.3 kB |
| Tags | CPython 3.6 CPython 3.6 pymalloc Windows x86-64 |
|
SHA-256 checksum How to use checksums |
3d0a6c55f130d542efb9c7fccec7674b2ee91c695a1c960e06431614fb7ed7aa
|
|
BLAKE2b-256 checksum How to use checksums |
9e5af9ccf75be9e7bdf4b54f945a1317bf0c00544e525f30440688b87b2730fe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | spacy_alignments-0.8.4-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 1.1 MB |
| Tags | CPython 3.6 CPython 3.6 pymalloc Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
b962d9e72e67203f5d8528d723fcdd467df1935e34ab2a94f720bf96f06526ce
|
|
BLAKE2b-256 checksum How to use checksums |
e8e8f8e2cedf9b1f7b6182a3fc19ba2049005374075dd58a40a53d2d352be8f4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
Release files / spacy_alignments-0.8.4-cp36-cp36m-macosx_10_9_x86_64.whl
| Download URL | spacy_alignments-0.8.4-cp36-cp36m-macosx_10_9_x86_64.whl |
|---|---|
| Size | 309.4 kB |
| Tags | CPython 3.6 CPython 3.6 pymalloc macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
f38834d4f7d161a1f2e32511fa7262b07c79c51ed1c0255bd9ffcf355ed0c111
|
|
BLAKE2b-256 checksum How to use checksums |
92f05bb84b8eac10441fb7d35ce89cf8b2f7ddd9196a247c3bab49149ded2d12
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.9
|
