Skip to main content

mbpe

BPE tokenizer — Mojo implementation with Python bindings.

import mbpe
tok = mbpe.get_encoding("gpt2")
tok.encode("hello world")

Supports gpt2, cl100k, o200k encodings plus custom training via GPreTokenizer, GPT2Tokenizer, GPT4Tokenizer.

Metadata

Release files for mbpe 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for mbpe 0.1.1
File Interpreter ABI Platform Actions
mbpe-0.1.1-py3-none-manylinux_2_35_x86_64.whl Python 3 none Linux glibc 2.35+ x86-64 Details for mbpe-0.1.1-py3-none-manylinux_2_35_x86_64.whl
mbpe-0.1.1-py3-none-any.whl Python 3 none any Details for mbpe-0.1.1-py3-none-any.whl

Total release size: 25.3 MB

Release files / mbpe-0.1.1-py3-none-manylinux_2_35_x86_64.whl

Download URL mbpe-0.1.1-py3-none-manylinux_2_35_x86_64.whl
Size 22.2 MB
Tags Linux glibc 2.35+ x86-64 Python 3
SHA-256 checksum
How to use checksums
175ab176a86d73359df9a9d4b77cbb58616a11215aad38c2ab2e7df4f31901a4
BLAKE2b-256 checksum
How to use checksums
504c035d0832708ea40a57f310e9bc5d5bacb3235a91c6b6322985414c1e839a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.14

Release files / mbpe-0.1.1-py3-none-any.whl

Download URL mbpe-0.1.1-py3-none-any.whl
Size 3.0 MB
Tags Python 3
SHA-256 checksum
How to use checksums
591db2ae5707d1b213ac8b044a4dfdffc81cdacb225f0735920c9b65b007f202
BLAKE2b-256 checksum
How to use checksums
9fae5de1bf6c21ad10ea87c429af3550fec14fd179cb31882c5cc3e81239cc06
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.14

Security for mbpe 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page