Skip to main content

BytePFT

CI Python License

Retrofitting language models, one byte at a time.

BytePFT is a Python library for adapting pretrained Qwen3 language models to UTF-8 bytes. It trains a byte-level encoder and decoder around the existing transformer, with patch-based processing and cached byte generation.

BytePFT is experimental. The reference implementation is tested on CPU and Apple MPS in fp32; full pretrained adaptation and retained language quality have not yet been evaluated. See project status for results and limitations.

Installation

Requires Python 3.11+.

pip install bytepft

Or install from source:

pip install git+https://github.com/skundu42/bytepft.git

Quickstart

Train a tiny model locally, without downloading pretrained weights:

from bytepft import BytePFT, BytePFTConfig

model = BytePFT.from_tiny(
    config=BytePFTConfig(hidden_size=32, encoder_layers=1, decoder_layers=1),
    device="cpu",
)
model.retrofit(
    ["hello byte world\n"] * 4,
    stage1_steps=100,
    learning_rate=0.01,
    validation_fraction=0,
    output_dir="outputs/tiny",
)

model = BytePFT.load("outputs/tiny")
print(model.generate("hello", max_new_bytes=32, temperature=0))

This example uses a randomly initialized tiny transformer. To adapt pretrained weights, use BytePFT.from_pretrained("Qwen/Qwen3-0.6B-Base") and follow the training guide. Adaptation must complete before generation.

Features

  • UTF-8 byte encoding with separate padding, beginning and end symbols.
  • Fixed, oracle and learned patch boundaries.
  • Frozen-backbone Stage 1 and optional end-to-end Stage 2 training.
  • Resumable checkpoints, validation and bits-per-byte evaluation.
  • Greedy and sampled generation with local state and global KV caching.

Defaults use CPU. Select device="mps" explicitly for Apple GPU execution; MPS validation currently covers fp32. CUDA remains unvalidated.

Command line

bytepft train-tiny --output outputs/tiny --device cpu
bytepft generate --model outputs/tiny --prompt "hello" --max-new-bytes 32
bytepft inspect Qwen/Qwen3-0.6B-Base

Run bytepft --help for all commands.

Documentation

Inspired by Retrofitting language models to operate over bytes, Nature (2026). BytePFT uses a simplified reference architecture; the research notes document its deviations.

License

Apache-2.0. See NOTICE for attribution.

Metadata

Release files for bytepft 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for bytepft 0.1.0
File Size Uploaded Provenance Actions
bytepft-0.1.0.tar.gz 282.3 kB With provenance Details for bytepft-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for bytepft 0.1.0
File Interpreter ABI Platform Actions
bytepft-0.1.0-py3-none-any.whl Python 3 none any Details for bytepft-0.1.0-py3-none-any.whl

Total release size: 335.4 kB

Release files / bytepft-0.1.0.tar.gz

Download URL bytepft-0.1.0.tar.gz
Size 282.3 kB
Tags Source
SHA-256 checksum
How to use checksums
627f40fcb00107fc581f0b697ef82070032d3463bb6688e199e5af3222949c6f
BLAKE2b-256 checksum
How to use checksums
cf6c6d7a5093ecf0f386fb28584a0edac42a68863b1f3546b437e9c2767cffb8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Release files / bytepft-0.1.0-py3-none-any.whl

Download URL bytepft-0.1.0-py3-none-any.whl
Size 53.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f322f143741f829e3280f8e912ff978286aafb7a0b03798dafca9708f171c554
BLAKE2b-256 checksum
How to use checksums
6975d03e17d8b52b728b2f701f18116a3deadbd327e290b69890d642f47e3b71
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Security for bytepft 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for bytepft-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page