Skip to main content
Yanked

This release has been yanked by its maintainers, and will be ignored by installers, except when explicitly specified.
Consider using release 0.1.11 instead.

mbpe

BPE tokenizer — Mojo implementation with Python bindings.

import mbpe
tok = mbpe.get_encoding("gpt2")
tok.encode("hello world")

Supports gpt2, cl100k, o200k encodings plus custom training via GPreTokenizer, GPT2Tokenizer, GPT4Tokenizer.

Metadata

Release files for mbpe 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for mbpe 0.1.0
File Interpreter ABI Platform Actions
mbpe-0.1.0-py3-none-manylinux_2_35_x86_64.whl Python 3 none Linux glibc 2.35+ x86-64 Details for mbpe-0.1.0-py3-none-manylinux_2_35_x86_64.whl
mbpe-0.1.0-py3-none-any.whl Python 3 none any Details for mbpe-0.1.0-py3-none-any.whl

Total release size: 25.3 MB

Release files / mbpe-0.1.0-py3-none-manylinux_2_35_x86_64.whl

Download URL mbpe-0.1.0-py3-none-manylinux_2_35_x86_64.whl
Size 22.2 MB
Tags Linux glibc 2.35+ x86-64 Python 3
SHA-256 checksum
How to use checksums
74692f7d39978f848708b2cb5050f54ec1609520b4558da5f7f53c65ea179bf9
BLAKE2b-256 checksum
How to use checksums
a71cb705608e12e1b912d7730f510399aef8c76de12bb45c29242d69ba3f40f6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.14

Release files / mbpe-0.1.0-py3-none-any.whl

Download URL mbpe-0.1.0-py3-none-any.whl
Size 3.0 MB
Tags Python 3
SHA-256 checksum
How to use checksums
5ec0e732f24fd6af7892368190d89cc1d865c8fdb9477ee17b077dcfb6df0cb3
BLAKE2b-256 checksum
How to use checksums
b696474bbfa32d2c1c0eb878af7e8e890aeffa2a9cf023cf5c73cc10b91a84be
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.14

Security for mbpe 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page