This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is a regular-expression based, extensible, and advanced tokeniser written in C++ (https://languagemachines.github.io/ucto).

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for proycon from gravatar.com proycon

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: GNU General Public License v3 (GPLv3) (GPLv3)
  • Author: Maarten van Gompel
  • Tags tokenizer , tokenization , tokeniser , tokenisation , nlp , computational_linguistics , ucto
Classifiers
Report project as malware

Project description

The author of this package has not provided a project description

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for proycon from gravatar.com proycon

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: GNU General Public License v3 (GPLv3) (GPLv3)
  • Author: Maarten van Gompel
  • Tags tokenizer , tokenization , tokeniser , tokenisation , nlp , computational_linguistics , ucto
Classifiers

Release history Release notifications | RSS feed

This version

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.3

0.5.2

0.5.1

0.5.0

0.4.7

0.4.5

0.4.4

0.4.3

0.4.2

0.3.0

0.2.4

0.2.2

0.2.1

0.2

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

python_ucto-0.6.9.tar.gz (110.3 kB view details)

Uploaded Source

Built Distributions

python_ucto-0.6.9-cp313-cp313-musllinux_1_1_x86_64.whl (27.0 MB view details)

Uploaded CPython 3.13musllinux: musl 1.1+ x86-64

python_ucto-0.6.9-cp313-cp313-manylinux_2_28_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.28+ x86-64

python_ucto-0.6.9-cp313-cp313-macosx_14_0_arm64.whl (16.0 MB view details)

Uploaded CPython 3.13macOS 14.0+ ARM64

python_ucto-0.6.9-cp312-cp312-musllinux_1_1_x86_64.whl (27.0 MB view details)

Uploaded CPython 3.12musllinux: musl 1.1+ x86-64

python_ucto-0.6.9-cp312-cp312-manylinux_2_28_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.28+ x86-64

python_ucto-0.6.9-cp312-cp312-macosx_14_0_arm64.whl (16.0 MB view details)

Uploaded CPython 3.12macOS 14.0+ ARM64

python_ucto-0.6.9-cp311-cp311-musllinux_1_1_x86_64.whl (27.0 MB view details)

Uploaded CPython 3.11musllinux: musl 1.1+ x86-64

python_ucto-0.6.9-cp311-cp311-manylinux_2_28_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.28+ x86-64

python_ucto-0.6.9-cp311-cp311-macosx_14_0_arm64.whl (16.0 MB view details)

Uploaded CPython 3.11macOS 14.0+ ARM64

python_ucto-0.6.9-cp310-cp310-musllinux_1_1_x86_64.whl (27.0 MB view details)

Uploaded CPython 3.10musllinux: musl 1.1+ x86-64

python_ucto-0.6.9-cp310-cp310-macosx_14_0_arm64.whl (16.0 MB view details)

Uploaded CPython 3.10macOS 14.0+ ARM64

python_ucto-0.6.9-cp39-cp39-musllinux_1_1_x86_64.whl (27.0 MB view details)

Uploaded CPython 3.9musllinux: musl 1.1+ x86-64

python_ucto-0.6.9-cp39-cp39-manylinux_2_28_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.9manylinux: glibc 2.28+ x86-64

python_ucto-0.6.9-cp39-cp39-macosx_14_0_arm64.whl (16.0 MB view details)

Uploaded CPython 3.9macOS 14.0+ ARM64

python_ucto-0.6.9-cp38-cp38-musllinux_1_1_x86_64.whl (27.0 MB view details)

Uploaded CPython 3.8musllinux: musl 1.1+ x86-64

python_ucto-0.6.9-cp38-cp38-manylinux_2_28_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.8manylinux: glibc 2.28+ x86-64

File details

Details for the file python_ucto-0.6.9.tar.gz.

File metadata

  • Download URL: python_ucto-0.6.9.tar.gz
  • Upload date:
  • Size: 110.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.0.1 CPython/3.12.7

File hashes

Hashes for python_ucto-0.6.9.tar.gz
Algorithm Hash digest
SHA256 11fa5f8211842d7e06d1df978904c8263abb8d14f63b285d0020acfc2aa073ae
MD5 1c384d35c799b5faa09977dcd0c3c066
BLAKE2b-256 1faf4389468a5d4a0492469f088f9bc559898994cbe1423f8f6f9d60bf683d5c

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp313-cp313-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp313-cp313-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 240234ee0ca0757b9ec743dc9076619d2b9c3f0b66799cb58b1af5b6dcce755f
MD5 aa640d54a7d26d56bae80a8579e31ddd
BLAKE2b-256 16a519fb930e0ad74bd3ae156685b63ad49728ff06a0455fff51ca10ddecb9f1

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp313-cp313-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp313-cp313-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 6904263bdf1c7a5f8fc069118fdda55ac3d61026d5279895f95d0b50e1b2b8bf
MD5 6158012fe39ca5e0efb2d0c8eb7ee022
BLAKE2b-256 bcb973f9c492df11d35bf58ef5b2e0858cb6b4bb0e63780e84cf0cc11f685623

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp313-cp313-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp313-cp313-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 b1274855d64f4345468d2f90a558b31ba1b30a06973210727c659bd276c3242c
MD5 aea56e20e3f192a36d52b576c4749a99
BLAKE2b-256 624173a8bc3fe41d94cf2607301f6bf7c80df25353c46c6594c79d3b60c650e3

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp312-cp312-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 33a5f7adcba4f320430bbab98c12431f25e872eaf3ee9d3c0a0f96123fa6ae78
MD5 e89f9b1f27ffb04742224db220ee24a3
BLAKE2b-256 808f483f68ed7e677a86fbd01cc661985fc4db9d923bf266d33f4de4f9aae2b6

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp312-cp312-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp312-cp312-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 d4611983c6d2932b8f9cfa82b32235664399d6fd134c8a5986a92000b8b68c1c
MD5 c94d30385b3d6510a256eb8cbcce940b
BLAKE2b-256 54bbd484d5ae755567f1f57207db2be7bf9c760d89969bb07e8344b79da85155

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp312-cp312-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp312-cp312-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 b0302be43a5c10cff34c383c1ad60636c0da44baf2fee8b455e9efd4d4589532
MD5 6643ee2581898f5dcd1cdff77fc6782f
BLAKE2b-256 4284694cbabbfe490d797168c8d3fe79fbb15ecb9a3a8e77d2f7f28267d555e0

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp311-cp311-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 e07ae25c31d3340c523f5f3d373dc4c8c1a5cf2f3783b4e3d454984f9b6cbe5c
MD5 e7357edfe2efcedf2b6eb78ab8e36e9a
BLAKE2b-256 6e51ed1974dacdd61876c6149b1f9c07c33fc9b1139cc4af9b8f0a1508127fb9

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp311-cp311-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp311-cp311-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 c81a4ad97fe02049e39cb06a47db6bb652a20f6708efb130f8f75b336a9721fe
MD5 9c6da2b05127fba376f6238330b7bf9f
BLAKE2b-256 ba2f80fffd8b83ab9e0fe0983f770cf9633b81a0756bb945a75acb22e6e4aa45

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp311-cp311-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp311-cp311-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 664fa27b6c55200debdfef1cc6a552f037794796dd06cdfcb5b16be97da4f182
MD5 04e7e4f0f667c21e32fae707a152f07d
BLAKE2b-256 764bd442afb2d7239926718af79ff4d63c449f959a780e922fc1e6c87a8cba33

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp310-cp310-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 3f56bba3d900b074042a5ec0856695137ca64389bcdf4a22980b61c5b035fe65
MD5 e9114bfb10d53b3e726a6fbea9a6d304
BLAKE2b-256 a3bb3a29a4370d7bf685ce603067b20a1078ed290199fe6790ed255ab9f3503f

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp310-cp310-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp310-cp310-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 6c8e9899b453999c7cea3ccaea5319ba0486150005dd1dbe9b0108502d057c74
MD5 8b3a83b6f7f18823b46fe6ae719928c7
BLAKE2b-256 dadc1e7a93fe79c206448cda884c333c3c932b7ef4147702c1cdb763fd600fc6

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp39-cp39-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 bf692d0819fe3e2d82707f8e6000f17e15df3714e61c657e99f19d24b1ca4309
MD5 2ce7776257e51a08d80dad5d59295acd
BLAKE2b-256 af146c707c30eb82b046e832aae1b99d109e8f8cca854cbcd9cbaea995c6ac25

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp39-cp39-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp39-cp39-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 56a681aaadf7a15c223682eb950a36e2c2e54a7f0b4a0ef41294c1920cfda08a
MD5 70439255fe30cc9df3cad0f769beada8
BLAKE2b-256 60f55a154396a00374c9a4834490e32a665fc7b5333b489495ce2a6300cdff33

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp39-cp39-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp39-cp39-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 53415b668092d8d1d11c19c60f0cbd13b1fbd227da05017bdda0f64af600a2d3
MD5 da917d9bd114172a9d8ece7445a7baa0
BLAKE2b-256 abef8912c8c7f28352636449b9151ba6edb95ba4ad2f4e7bbdafb8d92e985134

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp38-cp38-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 bfbdc816edef7b87bf7f2317e464a0b30e2510475638520db425ff95a8431ea8
MD5 d7e0d969bf8537f099e1d8dc975df0cf
BLAKE2b-256 ad41b65b831360c439cc22da102168f523338ea076f15772ea8c7c151511feb3

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.9-cp38-cp38-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.9-cp38-cp38-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 bcdc57963569823a1d045aa2b06f3f8b33482451e93e7daa9312938c409b75a0
MD5 f76f6d14ca3bf28e630f622cb59c5e31
BLAKE2b-256 435c55830da67f12322d60dbdb190b6c65e2a197dd04004b5511fd5c3d0f6137

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page