spacy-alignments: Align tokenizations for spaCy + transformers

A spaCy package for Yohei Tamura's Rust tokenizations library with Python bindings.

Installation

pip install -U pip setuptools wheel
pip install spacy-alignments

If no binary wheel is available for your platform, you will need to install Rust in order to build spacy-alignments from source.

spacy-alignments vs. pytokenizations

The spacy_alignments module is a drop-in replacement for tokenizations:

import spacy_alignments as tokenizations
a2b, b2a = tokenizations.get_alignments(["å", "BC"], ["abc"])
assert a2b == [[0], [0]]
assert b2a == [[0, 1]]

The only difference between this package and the original pytokenizations is that it switches the build system to setuptools-rust to make it easier for us at Explosion to build source and binary packages for a wider range of platforms.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.9.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for explosion from gravatar.com explosion

Credits

Author: Explosion

License

MIT License (MIT)

Requires

Python >=3.6

Classifiers

Report project as malware

Metadata

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.9.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for explosion from gravatar.com explosion

Credits

Author: Explosion

License

MIT License (MIT)

Requires

Python >=3.6

Classifiers

Report project as malware

Release files for spacy-alignments 0.7.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for spacy-alignments 0.7.2
File Size Uploaded
spacy-alignments-0.7.2.tar.gz 4.6 kB Details

Built distributions (wheels)

Table of built distributions (wheels) for spacy-alignments 0.7.2
File
spacy_alignments-0.7.2-cp39-cp39-win_amd64.whl 182.8 kB CPython 3.9 CPython 3.9 Windows x86-64 Details
spacy_alignments-0.7.2-cp39-cp39-manylinux2014_x86_64.whl 977.7 kB CPython 3.9 CPython 3.9 Linux glibc 2.17+ x86-64 Details
spacy_alignments-0.7.2-cp39-cp39-macosx_10_9_x86_64.whl 285.9 kB CPython 3.9 CPython 3.9 macOS 10.9+ x86-64 Details
spacy_alignments-0.7.2-cp38-cp38-win_amd64.whl 182.8 kB CPython 3.8 CPython 3.8 Windows x86-64 Details
spacy_alignments-0.7.2-cp38-cp38-manylinux2014_x86_64.whl 977.7 kB CPython 3.8 CPython 3.8 Linux glibc 2.17+ x86-64 Details
spacy_alignments-0.7.2-cp38-cp38-macosx_10_9_x86_64.whl 285.9 kB CPython 3.8 CPython 3.8 macOS 10.9+ x86-64 Details
spacy_alignments-0.7.2-cp37-cp37m-win_amd64.whl 182.8 kB CPython 3.7 CPython 3.7 pymalloc Windows x86-64 Details
spacy_alignments-0.7.2-cp37-cp37m-manylinux2014_x86_64.whl 977.7 kB CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ x86-64 Details
spacy_alignments-0.7.2-cp37-cp37m-macosx_10_9_x86_64.whl 285.9 kB CPython 3.7 CPython 3.7 pymalloc macOS 10.9+ x86-64 Details
spacy_alignments-0.7.2-cp36-cp36m-win_amd64.whl 183.1 kB CPython 3.6 CPython 3.6 pymalloc Windows x86-64 Details
spacy_alignments-0.7.2-cp36-cp36m-manylinux2014_x86_64.whl 978.3 kB CPython 3.6 CPython 3.6 pymalloc Linux glibc 2.17+ x86-64 Details
spacy_alignments-0.7.2-cp36-cp36m-macosx_10_9_x86_64.whl 286.1 kB CPython 3.6 CPython 3.6 pymalloc macOS 10.9+ x86-64 Details

Total release size: 5.8 MB

Release files / spacy-alignments-0.7.2.tar.gz

Download URL spacy-alignments-0.7.2.tar.gz
Size 4.6 kB
Tags Source
SHA-256 checksum
How to use checksums 		3a48f8c640f0b593bedb09cf2c158778490db4b781f6260736f5a34bfaad9cbd
BLAKE2b-256 checksum
How to use checksums 		3424cc56e214c9dd155209afae5c6c28e77ade38eecf7eec98cf80b218916525
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

Release files / spacy_alignments-0.7.2-cp39-cp39-win_amd64.whl

Download URL spacy_alignments-0.7.2-cp39-cp39-win_amd64.whl
Size 182.8 kB
Tags CPython 3.9 Windows x86-64
SHA-256 checksum
How to use checksums 		9f866d44a4f4d5d96353aef9364eb665baba342267536c694e2794dae2ba6b7f
BLAKE2b-256 checksum
How to use checksums 		3cbb167fea215333e0e64305061fc28cc466a2ba15acd5bd7a2b68c7a5201ffd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

Release files / spacy_alignments-0.7.2-cp39-cp39-manylinux2014_x86_64.whl

Download URL spacy_alignments-0.7.2-cp39-cp39-manylinux2014_x86_64.whl
Size 977.7 kB
Tags CPython 3.9 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums 		68cfe59e7358b50e050321e161adc61c8e05f2052fc58a79bbaedc0e5a607616
BLAKE2b-256 checksum
How to use checksums 		6d6b309ec78d026b404d7b4d952f771f0a28e29065e8a4a6bf93e3be00e26b0d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

Release files / spacy_alignments-0.7.2-cp39-cp39-macosx_10_9_x86_64.whl

Download URL spacy_alignments-0.7.2-cp39-cp39-macosx_10_9_x86_64.whl
Size 285.9 kB
Tags CPython 3.9 macOS 10.9+ x86-64
SHA-256 checksum
How to use checksums 		0b49d8b5c4448e4d9ad48d14ba443dfa40139c547d1c39c0f2378d0111021329
BLAKE2b-256 checksum
How to use checksums 		fd682caaa7a87efa6788d0955052d32945c25cd5833f0785c76fb61a37e1798a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

Release files / spacy_alignments-0.7.2-cp38-cp38-win_amd64.whl

Download URL spacy_alignments-0.7.2-cp38-cp38-win_amd64.whl
Size 182.8 kB
Tags CPython 3.8 Windows x86-64
SHA-256 checksum
How to use checksums 		10df6d7da84e3f12e3465cb44a027c6fbcb5a8623f1d503bf92b65f9c9a5f62b
BLAKE2b-256 checksum
How to use checksums 		5fa349e295986668d15c3e9d4e04befd34b2fb63b84e4517cd6b49d2367a7ce9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

Release files / spacy_alignments-0.7.2-cp38-cp38-manylinux2014_x86_64.whl

Download URL spacy_alignments-0.7.2-cp38-cp38-manylinux2014_x86_64.whl
Size 977.7 kB
Tags CPython 3.8 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums 		60e493fb6a21f7710d80c646df3da59814c3eeb1e3b54065731794bf2e300072
BLAKE2b-256 checksum
How to use checksums 		295ab93244e06e36f4b212dfc43ab705aa1e8e17f77a31e00f3b9a6cc35b486a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

Release files / spacy_alignments-0.7.2-cp38-cp38-macosx_10_9_x86_64.whl

Download URL spacy_alignments-0.7.2-cp38-cp38-macosx_10_9_x86_64.whl
Size 285.9 kB
Tags CPython 3.8 macOS 10.9+ x86-64
SHA-256 checksum
How to use checksums 		c8eca36caebd74c4d8b83f6a3865c6d4f91f19cf3f54e537707b7f581b649a52
BLAKE2b-256 checksum
How to use checksums 		db585be599f7f2820b836cb368b8ffe68e4b921475900860692b251d2f180361
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

Release files / spacy_alignments-0.7.2-cp37-cp37m-win_amd64.whl

Download URL spacy_alignments-0.7.2-cp37-cp37m-win_amd64.whl
Size 182.8 kB
Tags CPython 3.7 CPython 3.7 pymalloc Windows x86-64
SHA-256 checksum
How to use checksums 		969bfe1efe248760eb374ef05f6592fa81bb4ca02279cb31db6f80f520654033
BLAKE2b-256 checksum
How to use checksums 		6b888463f55f96083df544510e11fe95001e2b02fc763b853a22567830c48cd6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

Release files / spacy_alignments-0.7.2-cp37-cp37m-manylinux2014_x86_64.whl

Download URL spacy_alignments-0.7.2-cp37-cp37m-manylinux2014_x86_64.whl
Size 977.7 kB
Tags CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums 		c8d31ffcce5ea5594fa9dfa2a0f579bd360fafa92ef66887b4f41a75e1ae03aa
BLAKE2b-256 checksum
How to use checksums 		42861a428af2429609f5a238e96503fc4367efda4caf2b28734903dc9eb1d92c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

Release files / spacy_alignments-0.7.2-cp37-cp37m-macosx_10_9_x86_64.whl

Download URL spacy_alignments-0.7.2-cp37-cp37m-macosx_10_9_x86_64.whl
Size 285.9 kB
Tags CPython 3.7 CPython 3.7 pymalloc macOS 10.9+ x86-64
SHA-256 checksum
How to use checksums 		044418dcf6e4e0efd350b40ab5643db543de20bb09d980a4baaf8079bdcb95b7
BLAKE2b-256 checksum
How to use checksums 		c9292e82c5cb519b2be9d224430ceff58c2d139632c341adff8cdc5fc09a3115
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

Release files / spacy_alignments-0.7.2-cp36-cp36m-win_amd64.whl

Download URL spacy_alignments-0.7.2-cp36-cp36m-win_amd64.whl
Size 183.1 kB
Tags CPython 3.6 CPython 3.6 pymalloc Windows x86-64
SHA-256 checksum
How to use checksums 		3c11882416a5953ff685e0aaef39a86260acbb8b953bf918d38eb987491f0c27
BLAKE2b-256 checksum
How to use checksums 		1c9c9e97063b0f545c9a1140abb8e7f270c6f56b50e3f2e500e2cfa3d4c24f9d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

Release files / spacy_alignments-0.7.2-cp36-cp36m-manylinux2014_x86_64.whl

Download URL spacy_alignments-0.7.2-cp36-cp36m-manylinux2014_x86_64.whl
Size 978.3 kB
Tags CPython 3.6 CPython 3.6 pymalloc Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums 		53dd3c2c07e463ff765c1c29d7a291986b3d772eb19bf17b7d0640cbee8052b6
BLAKE2b-256 checksum
How to use checksums 		a15207e51cb8e11dab0dcf203131d7ae66606576e71b71faa7eda01d8e299106
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

Release files / spacy_alignments-0.7.2-cp36-cp36m-macosx_10_9_x86_64.whl

Download URL spacy_alignments-0.7.2-cp36-cp36m-macosx_10_9_x86_64.whl
Size 286.1 kB
Tags CPython 3.6 CPython 3.6 pymalloc macOS 10.9+ x86-64
SHA-256 checksum
How to use checksums 		f2efd7ceb44e65fa0942da0ac8fe13bead0143d650f9b1c384de1dd5d31c857f
BLAKE2b-256 checksum
How to use checksums 		4bb18dde78863186155a49fb563673f44dbe25a932c9447dee7b7fb571d9419d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

Release history Release notifications | RSS feed

0.9.2

36 release files

0.9.1

30 release files

0.9.0

25 release files

0.8.6

28 release files

0.8.5

24 release files

0.8.4

16 release files

0.8.3

13 release files

This release

0.7.2 This release

13 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page