Skip to main content

push2tok

push2tok is a Python package for incremental byte-to-token-ID transduction.

How to use

# Download and convert Rinna unigram SentencePiece into a (non-deterministic) FST.
./tools/convert_rinna.sh

# Test run
echo "札幌から東京まで時速60キロで何時間かかる?" | \
  push2tok --fst dist/rinna-japanese-gpt2-medium.unigram.fst --mode beam --beam-width 5

From Python:

>>> import push2tok
>>> stream = push2tok.BeamFST.from_file("dist/rinna-japanese-gpt2-medium.unigram.fst").stream(5)

>>> stream.push("札幌から東京まで時速60キロで何時間かかる?".encode("utf-8"))
[3401, 28, 238, 109, 23490, 630, 4659, 19, 1059, 364, 9525, 0]

>>> stream.push("札".encode("utf-8"))
[]
>>> stream.push("幌から東".encode("utf-8"))
[3401]
>>> stream.push("京まで時".encode("utf-8"))
[28, 238, 109]
>>> stream.push("速60キ".encode("utf-8"))
[23490, 630]
>>> stream.push("ロで何時".encode("utf-8"))
[4659, 19, 1059]
>>> stream.push("間かかる?".encode("utf-8"))
[364, 9525, 0]

Metadata

Release files for push2tok 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for push2tok 0.1.0
File Size Uploaded Provenance Actions
push2tok-0.1.0.tar.gz 29.9 kB Without provenance Details for push2tok-0.1.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for push2tok 0.1.0
File Actions
push2tok-0.1.0-cp39-abi3-win_arm64.whl CPython 3.9 abi3 Windows ARM64 Details for push2tok-0.1.0-cp39-abi3-win_arm64.whl
push2tok-0.1.0-cp39-abi3-win_amd64.whl CPython 3.9 abi3 Windows x86-64 Details for push2tok-0.1.0-cp39-abi3-win_amd64.whl
push2tok-0.1.0-cp39-abi3-musllinux_1_2_x86_64.whl CPython 3.9 abi3 Linux musl 1.2+ x86-64 Details for push2tok-0.1.0-cp39-abi3-musllinux_1_2_x86_64.whl
push2tok-0.1.0-cp39-abi3-musllinux_1_2_aarch64.whl CPython 3.9 abi3 Linux musl 1.2+ ARM64 Details for push2tok-0.1.0-cp39-abi3-musllinux_1_2_aarch64.whl
push2tok-0.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.9 abi3 Linux glibc 2.17+ x86-64 Details for push2tok-0.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
push2tok-0.1.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl CPython 3.9 abi3 Linux glibc 2.17+ ARM64 Details for push2tok-0.1.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
push2tok-0.1.0-cp39-abi3-macosx_11_0_arm64.whl CPython 3.9 abi3 macOS 11.0+ ARM64 Details for push2tok-0.1.0-cp39-abi3-macosx_11_0_arm64.whl
push2tok-0.1.0-cp39-abi3-macosx_10_12_x86_64.whl CPython 3.9 abi3 macOS 10.12+ x86-64 Details for push2tok-0.1.0-cp39-abi3-macosx_10_12_x86_64.whl

Total release size: 3.5 MB

Release files / push2tok-0.1.0.tar.gz

Download URL push2tok-0.1.0.tar.gz
Size 29.9 kB
Tags Source
SHA-256 checksum
How to use checksums
879f2c4fe5cb596732fe6a939285c518ec6e512eafa8db929158f4c691e54473
BLAKE2b-256 checksum
How to use checksums
ff4a1cc860a1049fd0d03780511f6ae3f1bc09a34776e97065c6da410e9c0f93
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / push2tok-0.1.0-cp39-abi3-win_arm64.whl

Download URL push2tok-0.1.0-cp39-abi3-win_arm64.whl
Size 281.3 kB
Tags CPython 3.9 Windows ARM64 abi3
SHA-256 checksum
How to use checksums
3fb36316966b76fafeeb4b26da8518010162393900d71963a0dd19393d11c719
BLAKE2b-256 checksum
How to use checksums
d60803c908bd0052231a42bb609e45598ca28537254ae8aec807019e12b9bd1a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / push2tok-0.1.0-cp39-abi3-win_amd64.whl

Download URL push2tok-0.1.0-cp39-abi3-win_amd64.whl
Size 289.4 kB
Tags CPython 3.9 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
b11a90ee69efa1b1f396eb213e04be5a781994e86ed23067256d8bba31feb9f0
BLAKE2b-256 checksum
How to use checksums
a4bec23e5fb465eee90e4eebf0cee7fdf86c4f6aee30c7f585ffdad865404c14
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / push2tok-0.1.0-cp39-abi3-musllinux_1_2_x86_64.whl

Download URL push2tok-0.1.0-cp39-abi3-musllinux_1_2_x86_64.whl
Size 649.6 kB
Tags CPython 3.9 Linux musl 1.2+ x86-64 abi3
SHA-256 checksum
How to use checksums
f3f18330dbe40dbd399b2ff0b4225fd2091d69fc0a70b5a6ac03b5c4dd6e1594
BLAKE2b-256 checksum
How to use checksums
1ed91438e64f0a59dd0a5a97f6d560723fb1567a571b3457cbffa8c974d744af
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / push2tok-0.1.0-cp39-abi3-musllinux_1_2_aarch64.whl

Download URL push2tok-0.1.0-cp39-abi3-musllinux_1_2_aarch64.whl
Size 613.5 kB
Tags CPython 3.9 Linux musl 1.2+ ARM64 abi3
SHA-256 checksum
How to use checksums
e05d2999e884da820e763139c9d55827ce70c67c22bfd27f721dd8402120a68d
BLAKE2b-256 checksum
How to use checksums
c41579b34decb39f200267b013135c2253bcb78ea1bdbe71b164ca45199a55fe
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / push2tok-0.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL push2tok-0.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 443.1 kB
Tags CPython 3.9 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
a1d5159aec72de0b4c36b1838b0bd247bc7de758041fe4bdaaf8fe33f170086a
BLAKE2b-256 checksum
How to use checksums
cfeec0e830f7040715762faa943c81b3778e691aa5a653b62b81772f2fa33a00
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / push2tok-0.1.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL push2tok-0.1.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size 437.0 kB
Tags CPython 3.9 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
670a90e5bf3cd1d4048a3c553a6b9da860ca6b522adc9dffceab4541ff911724
BLAKE2b-256 checksum
How to use checksums
152ee7cbe4f478ba453af2d57e8a4a3b860dcb0791cb99186cd89a5e0a0b5269
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / push2tok-0.1.0-cp39-abi3-macosx_11_0_arm64.whl

Download URL push2tok-0.1.0-cp39-abi3-macosx_11_0_arm64.whl
Size 398.2 kB
Tags CPython 3.9 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
4fa709183d5e79a54cfc02a2227d8153f5fcab98acbd16655fa5038d91b01441
BLAKE2b-256 checksum
How to use checksums
03a89918b741b0feae377fcfe123edb1b8a9534d4796cb9d92fe96c640335ed4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / push2tok-0.1.0-cp39-abi3-macosx_10_12_x86_64.whl

Download URL push2tok-0.1.0-cp39-abi3-macosx_10_12_x86_64.whl
Size 404.7 kB
Tags CPython 3.9 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
3310eeb2a13075f705ec031af5f1c2668d3287005d07d620bdab1962ca913ec0
BLAKE2b-256 checksum
How to use checksums
46a7e3dc16da101c67fc09da1c394cf7edb6240359cc0169b36ecb4fc996bafb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for push2tok 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

9 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page