PieceTokenizer
BPE/BBPE + BytePiece 的 C++ 实现，多种训练/推理算法，带 Python 绑定。
三段式流水线：Normalizer（NFKC）→ PreTokenizer（Split / Num / Cn 三个正交参数）→ Tokenize（可训练）。文本格式 .pt，可读可编辑，UTF-8 字节回退。
安装
PyPI wheel 自带 BERTc 与 Summer 模型及 Summer 中文词典，无需另外下载或现场编译：
pip install piece-tokenizer
import piece_tokenizer as pt
# BERTc：SentencePiece，12,535 词
bertc = pt.BERTcTokenizer()
ids = bertc.encode_as_ids("你好，PieceTokenizer")
print(bertc.decode(ids))
# Summer：Piece BPE，81,903 词；配套中文词典会自动加载
summer = pt.SummerTokenizer()
print(summer.encode("中华人民共和国"))
# 等价的按名称加载方式
tok = pt.Tokenizer("BERTc") # 或 "Summer"
构建
需要 CMake 3.14+、C++17。
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
uv pip install . # 从源码构建 Python 包
快速开始
# 数据：下载中英文维基、分句 → cn_sentences.txt / en_sentences.txt
cd data
make PYTHON=python
# 训练：输出 scripts/output/{method}.model
cd scripts && make bytepiece # 或 make 训练全部；VOCAB_SIZE=16000 自定义
# 分词 / 编码 / 解码（读 stdin）
echo "你好世界" | ./build/piece-tokenizer tokenize --model output/bytepiece.model
echo "你好世界" | ./build/piece-tokenizer encode --model output/bytepiece.model
echo "897 411" | ./build/piece-tokenizer decode --model output/bytepiece.model
数据流程依赖 datasets 和 opencc-python-reimplemented。例如：
python -m pip install datasets opencc-python-reimplemented
data/Makefile 会下载完整的 FineWiki 中英文训练集，可能占用较多网络流量、磁盘空间和处理时间；已有语料时可跳过下载，直接覆盖训练脚本的 CN_INPUT、EN_INPUT。下载数据、中间文本和分句结果均被 Git 忽略。
PreTokenizer 三个参数
Normalizer → PreTokenizer 两段，不需要模型。三个正交参数（任意组合，互不强制），各 1:1 对应模型里持久化的字段：
| 参数 | 取值 | 含义 |
|---|---|---|
--split |
word（默认）/ isolate |
word=GPT-4 式（▁ 依附后词，don't→don+'t）；isolate=空格与每个标点各自独立（don't 整体保留） |
--num |
keep（默认）/ split |
数字串整段 / 逐码点切开 |
--dict |
空 / no / 词典路径 |
连续汉字：不切 / 逐字 / 按词典（见 CN 模式） |
旧参数 --digit 和 Python 关键字 digit= 继续作为兼容别名支持；新代码建议使用 --num 和 num=。模型文件仍使用原有的 split_digits 字段，因此无需转换已有模型。
S="Hello, World! don't 你好，世界。123abc"
echo "$S" | ./build/piece-tokenizer pretokenize # Hello , ▁World ! ▁don 't ▁ 你好 ， 世界 。 123 abc
echo "$S" | ./build/piece-tokenizer pretokenize --split isolate # Hello , ▁ World ! ▁ don't ▁ 你好 ， 世界 。 123 abc
echo "$S" | ./build/piece-tokenizer pretokenize --num split # ... 世界 。 1 2 3 abc
echo "$S" | ./build/piece-tokenizer pretokenize --dict no # ... ▁ 你 好 ， 世 界 。 123 abc
另有 --normalize <name>（如 NFKC_CF）、--reconstruct（保留所有空格）属 Normalizer 阶段。raw-count 用同一套参数做预分词并输出 word\tfreq（频率降序）。
训练方法
| 方法 | 说明 |
|---|---|
piece |
索引链表优化 BPE（类 NanoChat RustBPE） |
sentencepiece |
Symbol-cache BPE + character_coverage 保字 |
bytepiece |
Trie 最长匹配 + byte fallback |
naive |
基础字节级 BPE |
与 SentencePiece 的区别
SentencePiece 通常将规范化后的整句文本直接交给 BPE 或 Unigram，由统计模型自行学习子词边界。PieceTokenizer 则采用显式的三阶段流水线：
Normalizer → PreTokenizer → Tokenizer
其中 PreTokenizer 在子词算法之前确定不可跨越的基础边界，并由所有 Counter 和 Tokenizer 共用。它提供三个可以独立组合的维度：Split 控制空格、标点和英文缩写的切分方式，Num 控制数字串整体保留或逐码点拆分，Cn 控制中文不切、逐字切分或按词典分词。
中文词典模式使用 Trie + Viterbi Unigram 对连续汉字预切分，后续 BPE 只能在预切分片段内部合并，从而避免仅因局部共现频率产生跨中文词语的 piece。逐字模式则适合 CWS、NER 或以汉字为基本单位的模型。
PreTokenizer 配置会写入模型，训练和推理使用同一套边界规则。相比在 SentencePiece 外部维护额外的中文分词脚本，这种设计将规范化、预切分和子词编码组合成一个可复现的流程；关闭中文预切分后，仍可保留传统的无显式中文词边界方式。SentencePiece 本身能够处理中文，两者的区别在于 PieceTokenizer 原生提供了可选、显式的中文边界约束。
CN 模式（piece / sentencepiece）
BPE 只看共现频率，训练中文时易学出 ▁雨星朋友 这种跨词串。CN 模式在合并前先对连续汉字预切。训练和推理必须传同一个 --dict：
- 空 — 不启用，整段进 BPE。
no— 逐字模式：汉字全单字，并隐含强制cut=1 + split_digits=true（= Split=isolate + Num=split，由main.cc统一处理），于是中文/数字/标点各占一个 token，只有英文走 BPE。适合 char-level backbone / CWS / NER。- 词典路径 — 用 TSV
word\tfreq词典（Unigram）把汉字切成词，BPE 不跨词。
# 逐字模式（sentencepiece 带 character_coverage 保字，推荐）
./build/piece-tokenizer count --method sentencepiece --input cn.txt \
--vocab-size 16000 --model output/sp_char --dict no
echo "2024年8月,GPT-4 model release 苹果公司" | \
./build/piece-tokenizer tokenize --model output/sp_char.model --dict no
# → 2 0 2 4 年 8 月 , GPT - 4 ▁ model ▁ release ▁ 苹 果 公 司
# 词典模式
./build/piece-tokenizer count --method piece --input corpus.txt \
--model output/pc --dict dict.txt
echo "Tom 他是英国人Bat" | \
./build/piece-tokenizer tokenize --model output/pc.model --dict dict.txt
# → T om ▁ 他 是 英国 人 B at
配置持久化在 PreTokenizerSpec 里，推理自动按训练值走；老模型向后兼容。
追加特殊 token（无需重训）
给训练好的模型末尾追加 CONTROL token（自动去重、同步 vocab_size、<pad> 自动接上 pad_id）：
./build/piece-tokenizer insert-tokens --model in.model \
--extra-tokens "<pad>,<user>,<assistant>" --output out.model
训练时也可直接 count --extra-tokens "<pad>,<user>"，两条路径产出一致。
Python
import piece_tokenizer as pt
# 安装包内置模型
bertc = pt.Tokenizer("BERTc")
summer = pt.Tokenizer("Summer") # 自动加载配套中文词典
# 模型无关的预分词（三轴同 CLI）
pt.PreTokenizer(split='isolate', num='split', cn='no').tokenize("你好123 hi")
# → ['你', '好', '1', '2', '3', '▁', 'hi']
# 加载训练好的模型（dict 需与训练一致）
tok = pt.Tokenizer()
tok.load("output/bytepiece.model")
tok.encode("你好世界") # → [('你', 897), ('好', 411), ...]
tok.encode_as_ids("你好世界") # → [897, 411, ...]
tok.encode_bytes("😀") # → [(b'...', id), ...]，精确保留 byte fragment
tok.encode_as_piece_bytes("😀") # → [b'...', ...]
tok.decode([897, 411]) # → '你好'
tok.vocab_size(); tok.method
Byte-level BPE 的单个 piece 可能只是 UTF-8 字符的一部分，因此不保证能表示为 Python str。encode()、encode_as_pieces() 适用于 UTF-8 完整的 piece；处理任意文本或检查词表时，使用 encode_bytes()、encode_as_piece_bytes() 和 id_to_piece_bytes() 获取无损字节。
原理文档
完整的训练与编码原理统一收录在《底层实现：文本处理》：
License
MIT
1 maintainer
Metadata
1 maintainer
Release files for piece-tokenizer 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| piece_tokenizer-0.1.0.tar.gz | 4.2 MB | With provenance | Details for piece_tokenizer-0.1.0.tar.gz |
Built distributions (wheels)
Total release size: 103.6 MB
Release files / piece_tokenizer-0.1.0.tar.gz
| Download URL | piece_tokenizer-0.1.0.tar.gz |
|---|---|
| Size | 4.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0e07a1730e9a95e23cc9e98605a39db0ecdf20ac93ea435220123ac7ed5af0af
|
|
BLAKE2b-256 checksum How to use checksums |
265ee42fed7c1dc536ebc4c8b3dcd241b493d2904911c277f41f991753c3a945
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp314-cp314-win_amd64.whl
| Download URL | piece_tokenizer-0.1.0-cp314-cp314-win_amd64.whl |
|---|---|
| Size | 4.5 MB |
| Tags | CPython 3.14 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
6bd52d10583f37a10f9da43c9ac73fc07e5c08c9999b50823d9814a50bbadd1e
|
|
BLAKE2b-256 checksum How to use checksums |
e1206d2b6434ee150eabc3453b2066674662bfd49fad2b127cbb33c26f228d0b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
| Download URL | piece_tokenizer-0.1.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.14 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
05f45193900afc20397149588ac75b288410b033f478e9ddf2a54ff36349a538
|
|
BLAKE2b-256 checksum How to use checksums |
e7574ee8ab029d58975bd6dd682b75a93620b35c99a5fbfb26f2a71fd910aa35
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp314-cp314-macosx_14_0_x86_64.whl
| Download URL | piece_tokenizer-0.1.0-cp314-cp314-macosx_14_0_x86_64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.14 macOS 14.0+ x86-64 |
|
SHA-256 checksum How to use checksums |
345233ce2a33ddb430487947fd53eb13ea09b0ae1ff7f12eab65e3c91210d551
|
|
BLAKE2b-256 checksum How to use checksums |
55be8266469431c06efa9c5a51d4058b7a866b422a0a8cfff77aad363c13e653
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp314-cp314-macosx_14_0_arm64.whl
| Download URL | piece_tokenizer-0.1.0-cp314-cp314-macosx_14_0_arm64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.14 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
61d33f409bd681f43ce55b9f6a25e76501e04b1f076fc8a58a0d448dd144c05c
|
|
BLAKE2b-256 checksum How to use checksums |
dce1abb1a1635352600f98e88d306b1edaaff4e4da63f3621d0bf5a088eb846d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp313-cp313-win_amd64.whl
| Download URL | piece_tokenizer-0.1.0-cp313-cp313-win_amd64.whl |
|---|---|
| Size | 4.3 MB |
| Tags | CPython 3.13 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
3fd4739824a3733459ab2c4b68404e9fe14e831dc644551e9abc2a64ef61d279
|
|
BLAKE2b-256 checksum How to use checksums |
a163557c469cb6f2e6bf027c1b9a141e468fb3646a906b1a5c55e8742f2098dc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
| Download URL | piece_tokenizer-0.1.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.13 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
25eeb695d55b6220350a0df42b3c4347f16560f5c8807ebe9bb068cccd7eee16
|
|
BLAKE2b-256 checksum How to use checksums |
0c06bda2ffa178d6e13d87c282f3ea70ceaa7e0fd8ea7760c33100241f14f4a5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp313-cp313-macosx_14_0_x86_64.whl
| Download URL | piece_tokenizer-0.1.0-cp313-cp313-macosx_14_0_x86_64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.13 macOS 14.0+ x86-64 |
|
SHA-256 checksum How to use checksums |
933c8cfbf457d63cc914ad0e2496eaa1a0c14b337590f4bf97968d9e5edd142c
|
|
BLAKE2b-256 checksum How to use checksums |
4e58b9340bfda615eea49c1eb8f6a92013cb9939fa514d06ad82278c876172ef
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp313-cp313-macosx_14_0_arm64.whl
| Download URL | piece_tokenizer-0.1.0-cp313-cp313-macosx_14_0_arm64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.13 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
1eaccecddc560f925670de6e952b13319334d4302a4e3a30c414c51f0659a73a
|
|
BLAKE2b-256 checksum How to use checksums |
97bed187bab8bd1b303c0470f1f4b841e39224716e03b69c1f462b92ab0ce377
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp312-cp312-win_amd64.whl
| Download URL | piece_tokenizer-0.1.0-cp312-cp312-win_amd64.whl |
|---|---|
| Size | 4.3 MB |
| Tags | CPython 3.12 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
265c69074dc21991a7e94072bd1a5483ab17ab63602000f37dbc5786c3796fc7
|
|
BLAKE2b-256 checksum How to use checksums |
39ffaaab344b2fb4c184c9e1dd039f8cbd1d34e7f4e1fec26f87eb6e19080894
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
| Download URL | piece_tokenizer-0.1.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.12 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
d0c89d5df2d5cca607e32be83bd045115cece2097d6e165deeb7ab1f6b0fb3c5
|
|
BLAKE2b-256 checksum How to use checksums |
975dfd7f991addd8dee7d3e77f16b94e2010780f8176ee64e25f7e7dee12dd54
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp312-cp312-macosx_14_0_x86_64.whl
| Download URL | piece_tokenizer-0.1.0-cp312-cp312-macosx_14_0_x86_64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.12 macOS 14.0+ x86-64 |
|
SHA-256 checksum How to use checksums |
0ec7b08b4ae92f195da5a555fefba5453bfd6afd36b6344cb3823f64e44f3224
|
|
BLAKE2b-256 checksum How to use checksums |
14cf046c59c2d88fa34b1d9db10662edf46bac4604ab68523127dcf8d319bd25
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp312-cp312-macosx_14_0_arm64.whl
| Download URL | piece_tokenizer-0.1.0-cp312-cp312-macosx_14_0_arm64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.12 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
7f80545eecee3c194609409ed478b9009bf06c1f3daa855098c9fe23afdda2e7
|
|
BLAKE2b-256 checksum How to use checksums |
3b8e070e67414b2575adc626d1579b6fffbcf1e277dcbd256faa675fee7850d1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp311-cp311-win_amd64.whl
| Download URL | piece_tokenizer-0.1.0-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 4.3 MB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
1d367508660602f4cb46eb3144295e4f435e1d10cc33b11fd34a17760c7c180a
|
|
BLAKE2b-256 checksum How to use checksums |
052f37d73949eabe8a03713cb1f2619f4c5d892f39c25b418c83bd58d32560e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
| Download URL | piece_tokenizer-0.1.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.11 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
e9ba093ae58930e55d415f9b37871536d335f653d36be5eb8940d615808a8d87
|
|
BLAKE2b-256 checksum How to use checksums |
80019ddbc2bd5d3b9c860883d7adf641ae6482b5d3afe7de15af831ba0e8ee6f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp311-cp311-macosx_14_0_x86_64.whl
| Download URL | piece_tokenizer-0.1.0-cp311-cp311-macosx_14_0_x86_64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.11 macOS 14.0+ x86-64 |
|
SHA-256 checksum How to use checksums |
b58e11ea334efcc8e99c2fa06cece234cee72b8792a6fbf83c0938c12f422d7d
|
|
BLAKE2b-256 checksum How to use checksums |
c2c8c3028884a4e844c7c941802c43a944f78e3603d088c7bfcdbf16134332fb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp311-cp311-macosx_14_0_arm64.whl
| Download URL | piece_tokenizer-0.1.0-cp311-cp311-macosx_14_0_arm64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.11 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
6c17c90d329bece096c18fdac928447dede8039032e01d7ea7c7726aae04635e
|
|
BLAKE2b-256 checksum How to use checksums |
4f295d672c71ae8f05d9a4e4bcb63900d1384f29cbf186aba02fe981ab9ad0f7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp310-cp310-win_amd64.whl
| Download URL | piece_tokenizer-0.1.0-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 4.3 MB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
dac52166d83166068050d85fc94f8777f3a2390ae3ad493de649ceafe20523e6
|
|
BLAKE2b-256 checksum How to use checksums |
f5d31fb9f9b54ef5d68cc5d12fc4ef8682cb9d2f38a1da02bc520a2c8ec9c045
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
| Download URL | piece_tokenizer-0.1.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.10 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
0fa25f08ba2a5c2e6fa2195ec8ecff900b09f950c3e005935db1ad75cb09611a
|
|
BLAKE2b-256 checksum How to use checksums |
092cc76575224ba56cfc2c4d9668e7a682aa65d22bba1d99fe20b1e1bacc89ce
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp310-cp310-macosx_14_0_x86_64.whl
| Download URL | piece_tokenizer-0.1.0-cp310-cp310-macosx_14_0_x86_64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.10 macOS 14.0+ x86-64 |
|
SHA-256 checksum How to use checksums |
2e550e1b0bbd74ca277a66cbd28e7e1275264bef054ffb1fbe01fde5d75bcf14
|
|
BLAKE2b-256 checksum How to use checksums |
d6662e18066095ece6d7331bdd4a9e58f2632379976d936e4d24c7d719c8c13b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp310-cp310-macosx_14_0_arm64.whl
| Download URL | piece_tokenizer-0.1.0-cp310-cp310-macosx_14_0_arm64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.10 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
a7678adde01461ef61c335e89bd4cd75afdebc370f95177b1e6df67a963cd0a4
|
|
BLAKE2b-256 checksum How to use checksums |
2c43149501da7d1c7ece3251ebf3dd00cc79464d5aa2aebe31eaecd3e6ae1308
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp39-cp39-win_amd64.whl
| Download URL | piece_tokenizer-0.1.0-cp39-cp39-win_amd64.whl |
|---|---|
| Size | 4.3 MB |
| Tags | CPython 3.9 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
daa174b201e0857630a3b5e21013a0e0f486e0b4fe84dd2e1c2ec9190958dc16
|
|
BLAKE2b-256 checksum How to use checksums |
2521173f2f9de5f59c987a17e372f62b97f586f2962ea57a34fb65d4c3604660
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
| Download URL | piece_tokenizer-0.1.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.9 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
06f8fd4c559a2f9a6e5e4ff0b9a4f655aeb37e920d8ca64f6475e6c053774ca2
|
|
BLAKE2b-256 checksum How to use checksums |
40d8cac25b3b199f73e7d351d4a860a1d7864836764f5ff876e57e40f6904473
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp39-cp39-macosx_14_0_x86_64.whl
| Download URL | piece_tokenizer-0.1.0-cp39-cp39-macosx_14_0_x86_64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.9 macOS 14.0+ x86-64 |
|
SHA-256 checksum How to use checksums |
fe0e11d9c59e4fc4a401a6883773de77a6cb5397b0f1a92bf285cd9108216e9d
|
|
BLAKE2b-256 checksum How to use checksums |
5b07b361b3c82fee0fa10f29792d9c3bde7f169a1d989dcb486771809876d36c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logRelease files / piece_tokenizer-0.1.0-cp39-cp39-macosx_14_0_arm64.whl
| Download URL | piece_tokenizer-0.1.0-cp39-cp39-macosx_14_0_arm64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.9 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
155aaa2e3810a4d52c12ddef499b2596a2fa0a69846e6b100e51d497d77a837e
|
|
BLAKE2b-256 checksum How to use checksums |
1762ed2b5fc0a9fd0868514082df64abd05450c14ec1fd4c2926a77c224d7175
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logSecurity for piece-tokenizer 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 25 files in this release have been uploaded with provenance metadata.
Attestation for piece_tokenizer-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 29, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
