Skip to main content

This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is a regular-expression based, extensible, and advanced tokeniser written in C++ (https://languagemachines.github.io/ucto).

Project description

The author of this package has not provided a project description

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

python-ucto-0.6.7.tar.gz (104.7 kB view hashes)

Uploaded Source

Built Distributions

python_ucto-0.6.7-cp312-cp312-musllinux_1_1_x86_64.whl (26.0 MB view hashes)

Uploaded CPython 3.12 musllinux: musl 1.1+ x86-64

python_ucto-0.6.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (22.3 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

python_ucto-0.6.7-cp312-cp312-macosx_10_9_x86_64.whl (16.1 MB view hashes)

Uploaded CPython 3.12 macOS 10.9+ x86-64

python_ucto-0.6.7-cp311-cp311-musllinux_1_1_x86_64.whl (26.0 MB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.1+ x86-64

python_ucto-0.6.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (22.3 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

python_ucto-0.6.7-cp311-cp311-macosx_11_0_arm64.whl (15.4 MB view hashes)

Uploaded CPython 3.11 macOS 11.0+ ARM64

python_ucto-0.6.7-cp311-cp311-macosx_10_9_x86_64.whl (16.1 MB view hashes)

Uploaded CPython 3.11 macOS 10.9+ x86-64

python_ucto-0.6.7-cp310-cp310-musllinux_1_1_x86_64.whl (26.0 MB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.1+ x86-64

python_ucto-0.6.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (22.3 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

python_ucto-0.6.7-cp310-cp310-macosx_11_0_arm64.whl (15.4 MB view hashes)

Uploaded CPython 3.10 macOS 11.0+ ARM64

python_ucto-0.6.7-cp310-cp310-macosx_10_9_x86_64.whl (16.1 MB view hashes)

Uploaded CPython 3.10 macOS 10.9+ x86-64

python_ucto-0.6.7-cp39-cp39-musllinux_1_1_x86_64.whl (26.0 MB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.1+ x86-64

python_ucto-0.6.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (22.3 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

python_ucto-0.6.7-cp39-cp39-macosx_11_0_arm64.whl (15.4 MB view hashes)

Uploaded CPython 3.9 macOS 11.0+ ARM64

python_ucto-0.6.7-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB view hashes)

Uploaded CPython 3.9 macOS 10.9+ x86-64

python_ucto-0.6.7-cp38-cp38-musllinux_1_1_x86_64.whl (26.0 MB view hashes)

Uploaded CPython 3.8 musllinux: musl 1.1+ x86-64

python_ucto-0.6.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (22.3 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

python_ucto-0.6.7-cp38-cp38-macosx_10_9_x86_64.whl (16.1 MB view hashes)

Uploaded CPython 3.8 macOS 10.9+ x86-64

python_ucto-0.6.7-cp37-cp37m-musllinux_1_1_x86_64.whl (26.0 MB view hashes)

Uploaded CPython 3.7m musllinux: musl 1.1+ x86-64

python_ucto-0.6.7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (22.3 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

python_ucto-0.6.7-cp37-cp37m-macosx_10_9_x86_64.whl (16.1 MB view hashes)

Uploaded CPython 3.7m macOS 10.9+ x86-64

python_ucto-0.6.7-cp36-cp36m-musllinux_1_1_x86_64.whl (26.0 MB view hashes)

Uploaded CPython 3.6m musllinux: musl 1.1+ x86-64

python_ucto-0.6.7-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (22.3 MB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

python_ucto-0.6.7-cp36-cp36m-macosx_10_9_x86_64.whl (16.1 MB view hashes)

Uploaded CPython 3.6m macOS 10.9+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page