This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is a regular-expression based, extensible, and advanced tokeniser written in C++ (https://languagemachines.github.io/ucto).

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for proycon from gravatar.com proycon

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: GPL-3.0-only
  • Author: Maarten van Gompel
  • Tags tokenizer , tokenization , tokeniser , tokenisation , nlp , computational_linguistics , ucto
Classifiers
Report project as malware

Project description

The author of this package has not provided a project description

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for proycon from gravatar.com proycon

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: GPL-3.0-only
  • Author: Maarten van Gompel
  • Tags tokenizer , tokenization , tokeniser , tokenisation , nlp , computational_linguistics , ucto
Classifiers

Release history Release notifications | RSS feed

This version

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.3

0.5.2

0.5.1

0.5.0

0.4.7

0.4.5

0.4.4

0.4.3

0.4.2

0.3.0

0.2.4

0.2.2

0.2.1

0.2

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

python_ucto-0.6.10.tar.gz (122.8 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

python_ucto-0.6.10-cp314-cp314-manylinux_2_28_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.28+ x86-64

python_ucto-0.6.10-cp314-cp314-macosx_14_0_arm64.whl (16.4 MB view details)

Uploaded CPython 3.14macOS 14.0+ ARM64

python_ucto-0.6.10-cp313-cp313-musllinux_1_1_x86_64.whl (27.1 MB view details)

Uploaded CPython 3.13musllinux: musl 1.1+ x86-64

python_ucto-0.6.10-cp313-cp313-manylinux_2_28_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.28+ x86-64

python_ucto-0.6.10-cp313-cp313-macosx_14_0_arm64.whl (16.4 MB view details)

Uploaded CPython 3.13macOS 14.0+ ARM64

python_ucto-0.6.10-cp312-cp312-musllinux_1_1_x86_64.whl (27.1 MB view details)

Uploaded CPython 3.12musllinux: musl 1.1+ x86-64

python_ucto-0.6.10-cp312-cp312-manylinux_2_28_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.28+ x86-64

python_ucto-0.6.10-cp312-cp312-macosx_14_0_arm64.whl (16.4 MB view details)

Uploaded CPython 3.12macOS 14.0+ ARM64

python_ucto-0.6.10-cp311-cp311-musllinux_1_1_x86_64.whl (27.1 MB view details)

Uploaded CPython 3.11musllinux: musl 1.1+ x86-64

python_ucto-0.6.10-cp311-cp311-manylinux_2_28_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.28+ x86-64

python_ucto-0.6.10-cp311-cp311-macosx_14_0_arm64.whl (16.4 MB view details)

Uploaded CPython 3.11macOS 14.0+ ARM64

python_ucto-0.6.10-cp310-cp310-musllinux_1_1_x86_64.whl (27.1 MB view details)

Uploaded CPython 3.10musllinux: musl 1.1+ x86-64

python_ucto-0.6.10-cp310-cp310-manylinux_2_28_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.10manylinux: glibc 2.28+ x86-64

python_ucto-0.6.10-cp310-cp310-macosx_14_0_arm64.whl (16.4 MB view details)

Uploaded CPython 3.10macOS 14.0+ ARM64

python_ucto-0.6.10-cp39-cp39-musllinux_1_1_x86_64.whl (27.1 MB view details)

Uploaded CPython 3.9musllinux: musl 1.1+ x86-64

python_ucto-0.6.10-cp39-cp39-manylinux_2_28_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.9manylinux: glibc 2.28+ x86-64

python_ucto-0.6.10-cp39-cp39-macosx_14_0_arm64.whl (16.4 MB view details)

Uploaded CPython 3.9macOS 14.0+ ARM64

python_ucto-0.6.10-cp38-cp38-musllinux_1_1_x86_64.whl (27.1 MB view details)

Uploaded CPython 3.8musllinux: musl 1.1+ x86-64

python_ucto-0.6.10-cp38-cp38-manylinux_2_28_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.8manylinux: glibc 2.28+ x86-64

File details

Details for the file python_ucto-0.6.10.tar.gz.

File metadata

  • Download URL: python_ucto-0.6.10.tar.gz
  • Upload date:
  • Size: 122.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.2.0 CPython/3.14.2

File hashes

Hashes for python_ucto-0.6.10.tar.gz
Algorithm Hash digest
SHA256 886139d9c92aac2985929150c19a80a6b85ab27a0ce3321d2cda48483a8b99c5
MD5 e2d65bf6c9556487c6c9ac27a60213c9
BLAKE2b-256 eaa8f933b62cf349ffb0b62aab992db555a481c3c7256606ca56abfb3c3a9dbe

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp314-cp314-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp314-cp314-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 dedd9579ab029931f688ac915f00ee4ff74f24f20ca55b060ebe232b8e104aba
MD5 37771d6eec0a9bbff72d0da30ed3aaa0
BLAKE2b-256 fdd50e1b7ccda6a88d344384b061e03b11db9a2d2744bc880bb3d7407582b40e

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp314-cp314-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp314-cp314-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 dd22755d35a1fab916dc0795387cbe6babe8885104a19c35e3f22fb8f001bcc9
MD5 f41c20c2d24da81ba16cb5c599b4fa86
BLAKE2b-256 96e7b10a144e80d67626c0c2893347a938d4230626d968355a885994eaea7544

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp313-cp313-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp313-cp313-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 e39f3726adfdc962daa9e347638a152d5c5c132258e16509de267b14348f56a1
MD5 366fa9eea84e6ef1c0f2c3a47d8b4990
BLAKE2b-256 6c28da56aa97ad1dff3e1a1fc7344463d997b3a1ceed51e6968cadad71b0c947

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp313-cp313-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp313-cp313-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 7df4b3b7a6355fcce5fe1f02d6c8e36ec3bf97470aa76492f0c7a9a587f95bf6
MD5 d36cd1d56f8e53a35c13d8c436ccd61d
BLAKE2b-256 0a8d1b9f1834205e630adec7bcf73c425f691e0921d8d0a78ce09a93654b34db

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp313-cp313-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp313-cp313-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 9a27e39e4d47dcab769679d5a5cd2dcfa633f53249ce59a3b86f5513ff41cf59
MD5 0efb4bdd70aa4790cdf8e69141e8a6da
BLAKE2b-256 4174d9d25820aae5248e6520b4fd167627ff3307aa273f4bdb5882a4f4e89164

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp312-cp312-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 da8e662c997eb71b6ee72ada34c65979ca3ed940b6ff5d5d5143771c9d179550
MD5 016e16d9a500cd8c8b07af745c289f1a
BLAKE2b-256 7be1dcdbf6af3635f8e85b2e852edc6eb2b3d5b692c9649f4a13d0ef8dce577d

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp312-cp312-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp312-cp312-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 42b2c9971516c1f98de393e31aa6071fcd6e1f9be69a614f9dc701153b299566
MD5 012402f19554a72645997a322c405cfe
BLAKE2b-256 554b9a786ac591ac44aba2b64e6d1d8f600d9fb0c85634fcf401c83f098a17e0

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp312-cp312-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp312-cp312-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 affb4292a5a141e69ebafb0291e9b79505ee336075f934941e8c2401c3366e5d
MD5 6c284453cb9ec3a3cd9986f607591118
BLAKE2b-256 6525133ec105c01da87e1c083c1f687f35d0e198a490294aae67bb91c2df5772

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp311-cp311-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 02f421509fc3742f0a86fdf89cc97b7e46adde6e5b6a3ea8d9e0a25e1962a7e0
MD5 5582bec4cf35d7400ff0d772f4572fee
BLAKE2b-256 491666386a8b6d2041b2e3a55d3d9660ec38cd356f2aa0e027b018221ec100d9

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp311-cp311-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp311-cp311-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 33c36e46ab76ebdee45c9ee82d8366ceb39e96ae1f99fa06ec23ccb969d52d38
MD5 94f5b1ec3a7ad28d62f1b53b72151289
BLAKE2b-256 2b4c3dc95d43a8e5e2aecee33ebd27c77be801a00845f6ee070e647e359f86d5

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp311-cp311-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp311-cp311-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 2a7014f1a51b3e4d6e232d64549fdab55df86c88f016f05ff3c124af9d7c0e03
MD5 5c362382452cb6af45fec6c6fdbe64aa
BLAKE2b-256 af5a5d8dc074249da3b125454a32a7c2b105997008b0f51b6198076c3ce421d0

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp310-cp310-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 5e6676fefe24fa9b4890b3d1809ba92f968c001037e9d5e9146db076717ef752
MD5 7170f8b0da69a0bba4a0a46e30ccc4dc
BLAKE2b-256 4d1c155b9c3cb5af60d1952f33cc6f05be9d4dbe77ea99838d888a9f41d95499

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp310-cp310-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp310-cp310-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 01d3087645cc70401cdf840b6248b6924b239bac0079a14c61d5384eaecb1976
MD5 591722c3b105934c11cf5b08a2143c50
BLAKE2b-256 28c7fe8f203bd18ff2b34b5f7180ac12ab92a06dc477a982efb3534a7e40c69a

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp310-cp310-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp310-cp310-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 4120d1c0ac7e18fd11cab494a6a84873d902017abcc6920a5c52b7dd6159d89a
MD5 52b65f1446e72521c168cd8a3bb11572
BLAKE2b-256 d45c6b3cf49a22fa3fa8230c7d642b5f4a8f6248c630ddb6ea2329ea358908ff

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp39-cp39-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 a27db40bc94e1b07bb02148a738f321cef0a45f7035498a6d7dab9f51b693943
MD5 09b808bc7e2c82346c74f0875459dc53
BLAKE2b-256 954828cc8e32a6e4b944a1281a0a7f225928ebee0697d028ca2e46d81d2bc6ab

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp39-cp39-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp39-cp39-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 2770edbc0568d054eb35f86396845cecd60bf48260361b3fcb350e66b24af028
MD5 f644c9407cf4b911bbd46dc9b6b3a30f
BLAKE2b-256 ae92465ab3095a19e7ae4a028f0d742e221d7266c54f3c37e6d6f6302f5882a2

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp39-cp39-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp39-cp39-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 f4681f6dd6443ca5bb0b7cb11dff18906254c8966c7b6567c80af95b27709a0e
MD5 d21c6e7ea6d9e3fdd78b24d995aac177
BLAKE2b-256 6d53f8b4f3543a84031ba0a72c495590de71465423bb27abceef00f654cfbb44

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp38-cp38-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 085a31921675693f9c46ddfc5cf1019bea9430c69b87936446ee84a2980f900f
MD5 2ffc7596c9c5481441db69bf535558b1
BLAKE2b-256 4fa5382935fcfd44a24c649004d82ef2e06f100a0b91a029983317f0f20beb63

See more details on using hashes here.

File details

Details for the file python_ucto-0.6.10-cp38-cp38-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for python_ucto-0.6.10-cp38-cp38-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 c06ce2cd54406d276c9be3922ec5663ee5af28d351ca399b1c79995399d083b8
MD5 3727dc74b2fd0b9f3fecd7b08e6e30f8
BLAKE2b-256 dff2c1cd146b96efb5cd7db0ef8d3f4f5fb4bd18775ab3c1c1671da0fb6ff2f3

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page