riftco-transformer Python distribution

This package is the typed, runtime-dependency-free-by-default ctypes interface to libriftco_transformer_c, plus explicit data preparation, pretraining, post-training, experiment, artifact, generation, and local-serving modules. A platform wheel carries both the Python modules and its native C ABI library; users do not install the native framework separately.

Install

After a release has been published to PyPI:

python3 -m pip install riftco-transformer
python3 -c "from riftco_transformer import Context; print(Context().backend)"

riftco-transformer is the installable distribution name; Python code imports the stable riftco_transformer package.

The wheel stores libriftco_transformer_c.so, libriftco_transformer_c.dylib, or riftco_transformer_c.dll under riftco_transformer/.libs. It has no third-party runtime dependencies and needs no compiler or environment variable after installation. Initial binary wheels cover Linux x86_64 and aarch64 for both glibc (manylinux) and musl (musllinux), macOS x86_64 and arm64, and Windows AMD64. CPU is available on every supported platform; the macOS wheels also include Metal. Standard wheels recognize the stable cuda and tpu backend names but compile their unavailable stubs, so installing one does not add CUDA or libtpu runtime dependencies.

From a source checkout, install at the repository root with:

python3 -m pip install .

That source build compiles the C++20 implementation, so it needs a supported native compiler and platform SDK. RIFTCO_TRANSFORMER_LIBRARY remains an advanced development override for selecting a particular local native build; released wheels do not require it.

CUDA is an explicit source-build option. It needs CUDA Toolkit 12 or newer, a compatible NVIDIA driver, and an NVIDIA GPU:

CMAKE_ARGS="-DRIFTCO_TRANSFORMER_ENABLE_CUDA=ON" \
  python3 -m pip install .

The CUDA backend provides managed tensor storage plus native GPU matmul, materialized/Flash attention, attention gradients, and paged decode. Other framework capabilities remain functional through synchronous audited reference paths over that storage. Full fine-tuning, LoRA, held-out generalization evaluation, artifact creation, and serving can all use backend="cuda", but this is not a claim that every operation is GPU-accelerated.

TPU is a separate experimental source-build option for Linux x86-64 Cloud TPU VMs. It requires Google's external libtpu.so at runtime:

export RIFTCO_TRANSFORMER_TPU_LIBRARY=/absolute/path/to/libtpu.so
CMAKE_ARGS="-DRIFTCO_TRANSFORMER_ENABLE_TPU=ON" \
  python3 -m pip install .

The TPU backend compiles batched matmul, materialized attention and its gradients, and paged decode through the PJRT C API and StableHLO. Flash attention and the other capabilities use synchronous audited reference paths over host-mirrored storage, so full fine-tuning, LoRA, evaluation, and serving are functionally wired without implying an end-to-end TPU speedup. Real Cloud TPU validation is still pending. Default installs never load libtpu.

The Python package follows the framework release version (0.3.0 here), while the native C ABI has its own compatibility version (2.2). The client accepts the same ABI major and an equal or newer additive minor, and rejects older or breaking ABIs before use.

Backend names are "cpu", "metal", "cuda", and "tpu". High-level configurations also accept "auto", which prefers TPU when available, then CUDA, Metal, and CPU. An explicitly requested unavailable backend raises an error instead of silently changing the experiment backend.

Release automation

.github/workflows/release.yml builds and verifies the source distribution and self-contained platform wheels. workflow_dispatch is verification-only. Pushing a v<version> tag creates a GitHub Release after the artifacts pass. It also publishes to PyPI only when the repository variable PUBLISH_TO_PYPI is true.

PyPI publication uses Trusted Publishing rather than a stored API token. The publisher configuration is project riftco-transformer, owner quangng2000, repository riftco-transformer, workflow release.yml, and environment pypi. The project is licensed under Apache-2.0. Do not enable publication until the Trusted Publisher is configured.

Selectable tokenizers

Tokenizer offers interchangeable byte and byte-pair-encoding strategies:

from riftco_transformer import Tokenizer

with Tokenizer(
    "Hello, café 🙂 Hello again.",
    method="bpe",
    vocabulary_size=272,
    minimum_pair_frequency=2,
) as tokenizer:
    token_ids = tokenizer.encode("café 🙂")
    assert tokenizer.decode(token_ids) == "café 🙂"
    print(tokenizer.method, tokenizer.vocab_size)
    print(tokenizer.vocabulary)

The BPE vocabulary begins with all 256 single-byte tokens and appends learned pair pieces, so unseen bytes remain encodable. vocabulary_size is a maximum: learning can stop earlier when no pair reaches minimum_pair_frequency. Repeated pair counts are resolved deterministically.

For backward compatibility, Tokenizer(corpus) selects the corpus-derived byte method. It assigns IDs by sorted unsigned byte value:

with Tokenizer(b"cab\ncab") as tokenizer:
    assert tokenizer.method == "byte"
    assert tokenizer.vocabulary_bytes == b"\nabc"

Tokenizer accepts a str corpus, encoded as UTF-8, or a bytes/bytearray/memoryview corpus. encode() and decode() are strict UTF-8 conveniences. Use encode_bytes() and decode_bytes() for arbitrary binary data, including embedded NUL bytes. vocabulary returns a tuple of byte pieces for either method; vocabulary_bytes is the byte-only compatibility property.

End-to-end training

An end-to-end BPE training step uses only public Python objects:

from riftco_transformer import (
    Adam,
    DecoderOnlyTransformer,
    Tokenizer,
    TransformerConfig,
    cross_entropy,
)

corpus = "hello hello hello"
with Tokenizer(
    corpus,
    method="bpe",
    vocabulary_size=272,
) as tokenizer:
    encoded = tokenizer.encode(corpus)
    tokens = [encoded[:-1]]
    targets = [encoded[1:]]

    config = TransformerConfig(
        vocabulary_size=tokenizer.vocab_size,
        maximum_context=len(tokens[0]),
        model_width=16,
        head_count=4,
        block_count=1,
        feed_forward_width=32,
    )
    with DecoderOnlyTransformer(
        config,
        attention="flash",
        activation_checkpointing="block",
    ).to("cpu") as model:
        with model.parameters() as parameters:
            with Adam(parameters) as optimizer:
                with cross_entropy(model(tokens), targets) as loss:
                    loss_value = loss.item()
                    loss.backward()
                    stats = optimizer.step()
                print(loss_value, stats.gradient_norm)

Change .to("cpu") to .to("metal") on systems with the Metal backend, .to("cuda") in a CUDA-enabled source build, or .to("tpu") in a TPU-enabled Cloud TPU build. Computation graphs are single-use: build a fresh forward/loss graph for each training step. attention="flash" selects the dependency-free exact memory-linear full-sequence forward/backward implementation; omit it to keep the "materialized" default. The Flash path saves [batch, heads, time] row maxima and exponential sums and reconstructs probabilities during backward, rather than saving [batch, heads, time, time] probabilities. The explicit probability-returning diagnostic remains materialized. This selector does not change incremental serving, and no speedup is assumed without measuring the target workload.

activation_checkpointing="block" retains only transformer-block boundaries and replays each block during backward. Omit it for the "disabled" default. This reduces retained activation graph state at the cost of another block forward calculation during backward. It composes with FlashAttention and LoRA, but does not affect model artifacts or incremental decode.

On Metal, Flash working storage is proportional to the per-head width and must fit the device's threadgroup-memory limit. The native runtime preflights the complete forward/backward path before starting the forward pass. If the device rejects a very wide head, use more heads or select attention="materialized".

Hugging Face data and rank experiments

The riftco_transformer.data package is also dependency-free. Its default transport uses urllib to read bounded pages from the official Hugging Face Dataset Viewer API, while adapters convert TinyStories, Dolly 15K, and HH-RLHF into stage-specific files. Preparation removes exact duplicates, assigns records to deterministic content-hash splits, and writes an atomic directory with a provenance manifest and SHA-256 file digests.

From the framework directory:

PYTHONPATH="$PWD/python" \
python3 examples/python/prepare_huggingface_data.py \
  --preset dolly \
  --output data/external/huggingface/dolly-lora-v1 \
  --limit 2000 \
  --seed lora-v1

HF_TOKEN is an optional environment variable, never a CLI argument. Prepared downloads under data/external/ are ignored by Git. The Dolly adapter maps instruction plus optional context into prompt, preserves response, and retains category. TinyStories becomes plain text. HH-RLHF remains chosen/rejected preference data and is not accepted by the current SFT pipeline.

The riftco_transformer.experiments package compares LoRA ranks from the same immutable base:

from riftco_transformer.artifacts import ModelBundle
from riftco_transformer.experiments import (
    LoraRankExperimentConfig,
    compare_lora_ranks,
    load_prepared_instruction_splits,
)

base = ModelBundle.load("results/stages/tinystories_pretrained.rift")
splits = load_prepared_instruction_splits(
    "data/external/huggingface/dolly-lora-v1"
)
comparison = compare_lora_ranks(
    base,
    splits,
    LoraRankExperimentConfig(
        ranks=(1, 2, 4, 8),
        alpha_over_rank=2.0,
        steps=20,
        backend="cpu",
    ),
)
print(comparison.best_rank, comparison.selected_test.loss)

Every rank shares data fingerprints, seeds, sampler, optimizer controls, LoRA targets, and alpha / rank. Validation selects the winner; held-out test evaluation begins only after selection. The objective is still full-sequence causal SFT, not response-only loss. All adapters are merged before persistence, so ranks have the same serving topology and inference timings are only smoke measurements. The CLI atomically publishes a new, complete output directory and embeds the verified prepared-data manifest plus its SHA-256 in comparison.json.

compare_fine_tuning() generalizes the same held-out protocol to fixed full fine-tuning recipes and LoRA rank groups. It exhaustively evaluates train and validation to calculate a comparable generalization gap, then evaluates test only for the fixed full recipe and validation-selected LoRA rank. The example CLI is:

python3 examples/python/compare_fine_tuning.py \
  --base results/stages/tinystories_pretrained.rift \
  --data data/external/huggingface/dolly-lora-v1 \
  --output results/experiments/full-vs-lora \
  --full-learning-rate 0.001 \
  --lora-learning-rate 0.005 \
  --backend cpu

The resulting test split is consumed for a final method comparison and must be retired before further tuning.

See docs/DATASETS_AND_LORA_EXPERIMENTS.md in the framework repository for license links, exact TinyStories train/validation commands, sample-size guidance, provenance details, and the CLI rank workflow.

Staged pipeline

The high-level modules make the stage boundaries explicit:

from riftco_transformer.artifacts import ModelBundle
from riftco_transformer import LoraConfig
from riftco_transformer.post_training import (
    PostTrainingConfig,
    post_train_jsonl,
)
from riftco_transformer.pretraining import PretrainingConfig, pretrain_file
from riftco_transformer.serving import ServingConfig, serve_model

base = pretrain_file(
    "data/pretraining/tiny_corpus.txt",
    PretrainingConfig(
        steps=20,
        backend="cpu",
        attention="flash",
        activation_checkpointing="block",
    ),
)
base.bundle.save("results/stages/tiny_pretrained.rift")

restored = ModelBundle.load("results/stages/tiny_pretrained.rift")
assistant = post_train_jsonl(
    restored,
    "data/post_training/tiny_instructions.jsonl",
    PostTrainingConfig(
        steps=10,
        backend="cpu",
        attention="flash",
        activation_checkpointing="block",
        fine_tuning_method="lora",
        lora=LoraConfig(rank=4, alpha=8.0),
    ),
)
assistant.bundle.save("results/stages/tiny_post_trained.rift")

serve_model(
    "results/stages/tiny_post_trained.rift",
    host="127.0.0.1",
    port=8000,
    config=ServingConfig(
        backend="cpu",
        kv_cache="paged",
        kv_cache_block_size=16,
    ),
)

The HTTP adapter serves a dependency-free browser chat at / and keeps POST /v1/generate as the stable JSON generation endpoint. Each chat message is formatted with PlainChatFormatter as one independent single-turn SFT prompt; the visual transcript is not added to the model context. Custom formatter templates are not persisted in the current artifact format. GET /health reports the selected backend, context and vocabulary sizes, and active KV-cache strategy.

ModelBundle persists the exact byte/BPE tokenizer definition, model configuration, named parameter shapes, float32 weights, checksums, stage metadata, and parent artifact ID. It is an immutable inference or warm-start artifact, not a resumable training checkpoint: Adam moments, optimizer step, data position, and random-generator state are deliberately not included yet. LoRA post-training optimizes only adapter factors, then merges them before capturing this ordinary serving-ready bundle. Full-parameter post-training remains the default; adapter-only persistence is not part of ModelBundle.

The first post-training objective is explicitly full_sequence_causal_sft: it applies causal cross-entropy to the complete formatted prompt/response sequence. Response-only loss masking is a future extension.

Incremental generation

Native models use the current ABI 2.2 DecodeSession surface instead of rerunning the full-sequence training forward for every generated token. TextGenerator creates a request-local session, prefills the prompt one token at a time, and then performs one-token decode:

from riftco_transformer.artifacts import ModelBundle
from riftco_transformer.serving import TextGenerator

bundle = ModelBundle.load("results/stages/tiny_post_trained.rift")
with bundle.instantiate("cpu") as runtime:
    result = TextGenerator(
        runtime.model,
        runtime.tokenizer,
        kv_cache="paged",
        kv_cache_block_size=16,
    ).generate("Tensor:", max_new_tokens=32)
    print(result.text)

Paged caching is the default; use kv_cache="contiguous" for the reference strategy. CPU, Metal, CUDA, and TPU have backend-owned paged-decode implementations; TPU stages the request through PJRT from host-mirrored storage. When the learned absolute-position context fills, TextGenerator resets the cache and replays the retained suffix from position zero. A raw session exposes the lower-level step/reset contract:

with bundle.instantiate("cpu") as runtime:
    with runtime.model.decode_session(
        cache="paged",
        block_size=16,
    ) as session:
        for token in runtime.tokenizer.encode("Tensor:"):
            logits = session.step(token)

The raw session does not tokenize, sample, or implement rollover. Paged storage also does not yet imply continuous batching, a request scheduler, or prefix sharing. Full-sequence attention="flash" does not alter this path: serving prefill is still token-at-a-time and decode remains paged.

See docs/PIPELINE.md, docs/SERVING.md, docs/TOKENIZATION.md, and docs/BACKENDS_AND_PYTHON.md in the framework repository for the complete workflow, lifecycle, backend, and error-handling contracts.

Package layout

The physical package mirrors the runtime boundaries:

riftco_transformer/
├── native/          # stable C ABI bindings
├── artifacts/       # ModelBundle persistence
├── data/            # external dataset adapters and preparation
├── experiments/     # controlled full/LoRA and rank comparisons
├── training/        # shared batches, metrics, and trainer
├── pretraining/     # next-token pretraining stage
├── post_training/   # supervised continuation stage
└── serving/         # generation, model service, and HTTP

The package root re-exports the public low-level API. The breaking rename installs only riftco_transformer; no legacy package-name alias is provided.

License

Copyright 2026 Quang T Nguyen. Licensed under the Apache License 2.0. The full license text is included in the source distribution and every wheel.