Skip to main content

custom-transformer

A PyTorch-based transformer library for language modeling and automatic speech recognition (ASR). Includes both a numpy-based multi-head attention implementation (mytorch) and a full PyTorch transformer toolkit (transformerlib) with training, decoding, and data loading utilities.

Installation

pip install custom-transformer

Features

mytorch — NumPy Attention Primitives

Pure numpy implementations for educational and prototyping purposes:

  • Linear — fully-connected layer with forward and backward passes
  • Softmax — numerically stable softmax activation
  • ScaledDotProductAttention — attention mechanism with optional masking
  • MultiHeadAttention — multi-head attention with split/concat head logic
from mytorch.nn import MultiHeadAttention
mha = MultiHeadAttention(d_model=512, num_heads=8)

transformerlib — PyTorch Transformer Toolkit

Full-featured transformer models and training infrastructure:

Models

  • DecoderOnlyTransformer — GPT-style causal language model
  • EncoderDecoderTransformer — encoder-decoder for sequence-to-sequence tasks (e.g., ASR)
  • Pre-LN architecture with sinusoidal positional encoding
  • Weight tying, layer dropout, and mixed precision support
  • from_pretrained_decoder for initializing encoder-decoder models from pretrained decoder weights

Data

  • LMDataset — text dataset with tokenization, SOS/EOS framing, and collation
  • ASRDataset — speech dataset with filterbank features, global MVN normalization, and SpecAugment
  • H4Tokenizer — BPE tokenizer wrapper (char, 1k, 5k, 10k vocab sizes included)

Training

  • LMTrainer — language model training with gradient accumulation, mixed precision, and WandB logging
  • ASRTrainer — ASR training with CTC + cross-entropy joint loss
  • ProgressiveTrainer — curriculum learning with gradual layer unfreezing and data subsetting

Decoding

  • SequenceGenerator — greedy search, beam search, and nucleus sampling
  • Language model shallow fusion for ASR recognition

Quick Start

from transformerlib.model import DecoderOnlyTransformer

model = DecoderOnlyTransformer(
    num_layers=6,
    d_model=512,
    num_heads=8,
    d_ff=2048,
    dropout=0.1,
    max_len=512,
    num_classes=10000
)
from transformerlib.model import EncoderDecoderTransformer

model = EncoderDecoderTransformer(
    num_encoder_layers=6,
    num_decoder_layers=6,
    d_model=512,
    num_heads=8,
    d_ff=2048,
    dropout=0.1,
    max_len=512,
    num_classes=1000,
    feat_dim=80
)

Architecture

mytorch/
  nn/
    linear.py
    activation.py
    scaled_dot_product_attention.py
    multi_head_attention.py

transformerlib/
  model/        — masks, positional encoding, sublayers, encoder/decoder layers, transformers
  data/         — tokenizer, LM dataset, ASR dataset
  trainers/     — base trainer, LM trainer, ASR trainer, progressive trainer
  decoding/     — sequence generator (greedy, beam, sampling)
  utils/        — optimizer and LR scheduler factories

Requirements

  • Python >= 3.9
  • PyTorch >= 2.0
  • torchaudio
  • numpy, tqdm, wandb, torchmetrics, tokenizers, pandas, matplotlib

License

MIT

Metadata

Release files for custom-transformer 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for custom-transformer 0.1.0
File
custom_transformer-0.1.0-cp313-cp313-win_amd64.whl CPython 3.13 CPython 3.13 Windows x86-64 Details
custom_transformer-0.1.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.13 CPython 3.13 Linux glibc 2.17+ x86-64 Details
custom_transformer-0.1.0-cp313-cp313-macosx_10_13_universal2.whl CPython 3.13 CPython 3.13 macOS 10.13+ universal2 (ARM64, x86-64) Details
custom_transformer-0.1.0-cp312-cp312-win_amd64.whl CPython 3.12 CPython 3.12 Windows x86-64 Details
custom_transformer-0.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.12 CPython 3.12 Linux glibc 2.17+ x86-64 Details
custom_transformer-0.1.0-cp312-cp312-macosx_10_13_universal2.whl CPython 3.12 CPython 3.12 macOS 10.13+ universal2 (ARM64, x86-64) Details
custom_transformer-0.1.0-cp311-cp311-win_amd64.whl CPython 3.11 CPython 3.11 Windows x86-64 Details
custom_transformer-0.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.17+ x86-64 Details
custom_transformer-0.1.0-cp311-cp311-macosx_10_9_universal2.whl CPython 3.11 CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64) Details
custom_transformer-0.1.0-cp310-cp310-win_amd64.whl CPython 3.10 CPython 3.10 Windows x86-64 Details
custom_transformer-0.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.17+ x86-64 Details
custom_transformer-0.1.0-cp310-cp310-macosx_10_9_universal2.whl CPython 3.10 CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64) Details

Total release size: 38.1 MB

Release files / custom_transformer-0.1.0-cp313-cp313-win_amd64.whl

Download URL custom_transformer-0.1.0-cp313-cp313-win_amd64.whl
Size 1.0 MB
Tags CPython 3.13 Windows x86-64
SHA-256 checksum
How to use checksums
e7be9e7b93604f7e07bc1ab60c5c29854bf6c3c54c6d5210b1a19bad0cea22d4
BLAKE2b-256 checksum
How to use checksums
55b72a9dfa7df64db5f40b2453a77eb30f73a8acbc74347b197688053db6ecb2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.

Transparency log

Release files / custom_transformer-0.1.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL custom_transformer-0.1.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 6.4 MB
Tags CPython 3.13 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
ac2c498ede2db9e6c55be60e6017a60b094f060508bf9b0b96cae62abde6e51d
BLAKE2b-256 checksum
How to use checksums
4f1e829fc134260a87f4c2c3b69da67991b65baed83f5ca0bf6c09fa41441755
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.

Transparency log

Release files / custom_transformer-0.1.0-cp313-cp313-macosx_10_13_universal2.whl

Download URL custom_transformer-0.1.0-cp313-cp313-macosx_10_13_universal2.whl
Size 2.1 MB
Tags CPython 3.13 macOS 10.13+ universal2 (ARM64, x86-64)
SHA-256 checksum
How to use checksums
e78e8ebc41d253b8f996a0d0fabae6e4069478866a25ede6aa24df58058f860c
BLAKE2b-256 checksum
How to use checksums
692054d2f006759a15041b7b8a31be88d92b2ddec5f439fbda1d365184f1d557
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.

Transparency log

Release files / custom_transformer-0.1.0-cp312-cp312-win_amd64.whl

Download URL custom_transformer-0.1.0-cp312-cp312-win_amd64.whl
Size 1.0 MB
Tags CPython 3.12 Windows x86-64
SHA-256 checksum
How to use checksums
65c6b0f33c039ea53cc1070e874b6e67a5772f631acd7a295c1979837722f09b
BLAKE2b-256 checksum
How to use checksums
2e546ffa6eb8589ce15cbe8279d196149044cf563b179afe44faf77b92c1aace
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.

Transparency log

Release files / custom_transformer-0.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL custom_transformer-0.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 6.6 MB
Tags CPython 3.12 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
d456f7ac73cec1584ccceb746a93d6d75aad77dcc7b6c0e4e682640c8bb7242b
BLAKE2b-256 checksum
How to use checksums
c5e43e1d22164783e7f4dee4d25e85a139d672293bee68605835eef806accf5f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.

Transparency log

Release files / custom_transformer-0.1.0-cp312-cp312-macosx_10_13_universal2.whl

Download URL custom_transformer-0.1.0-cp312-cp312-macosx_10_13_universal2.whl
Size 2.1 MB
Tags CPython 3.12 macOS 10.13+ universal2 (ARM64, x86-64)
SHA-256 checksum
How to use checksums
5a94ed782428d43e65ba1624042ff955eeb34b4a13c684b7d85d43baf179110c
BLAKE2b-256 checksum
How to use checksums
8c045c366669128b8cd5ea47527cc3f785f592f336150a1852ca1b5c4c0321f8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.

Transparency log

Release files / custom_transformer-0.1.0-cp311-cp311-win_amd64.whl

Download URL custom_transformer-0.1.0-cp311-cp311-win_amd64.whl
Size 1.1 MB
Tags CPython 3.11 Windows x86-64
SHA-256 checksum
How to use checksums
028ebbf76e09c349252185523e93e46b6c1d64e323b948bcc1321ad91d332c7c
BLAKE2b-256 checksum
How to use checksums
208ed16619b522b47f31e4582bad1fcb2121fc0edecf891c0768a056973903e6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.

Transparency log

Release files / custom_transformer-0.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL custom_transformer-0.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 6.4 MB
Tags CPython 3.11 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
023c465c9847e010e85a306fe6a2eb96c1bcc3dbc07aaee75e8b0786c1c1e98f
BLAKE2b-256 checksum
How to use checksums
da46b446b7a300fb13c88b2bd20f954b8fb150d3ca7f9e45328171fb227afe39
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.

Transparency log

Release files / custom_transformer-0.1.0-cp311-cp311-macosx_10_9_universal2.whl

Download URL custom_transformer-0.1.0-cp311-cp311-macosx_10_9_universal2.whl
Size 2.1 MB
Tags CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64)
SHA-256 checksum
How to use checksums
dcec103ce64af1ec90cf2a5eeb0eac9f4de9ae8edf6ea2995f662286d7f28332
BLAKE2b-256 checksum
How to use checksums
bc234576257bce46a04d8415c4026950897a4fbd48fb06b9af4c7e3e6ae29128
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.

Transparency log

Release files / custom_transformer-0.1.0-cp310-cp310-win_amd64.whl

Download URL custom_transformer-0.1.0-cp310-cp310-win_amd64.whl
Size 1.1 MB
Tags CPython 3.10 Windows x86-64
SHA-256 checksum
How to use checksums
7429d7d1b051e84f7bef807b600534359e3991319b0e0f998400af149af60cf9
BLAKE2b-256 checksum
How to use checksums
66e8d37e48022222fda59a29933c9849dcd37d32b4a2c9587c9392a3843259bd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.

Transparency log

Release files / custom_transformer-0.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL custom_transformer-0.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 6.0 MB
Tags CPython 3.10 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
a2721f066ff65e25d23b6b15c608be500ba3242ff47dfc7de2cbb4c0361d73e7
BLAKE2b-256 checksum
How to use checksums
18595ecf580c4c8b351d6954a2b621a7bbed24962786d90ef8f2c5388eb23743
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.

Transparency log

Release files / custom_transformer-0.1.0-cp310-cp310-macosx_10_9_universal2.whl

Download URL custom_transformer-0.1.0-cp310-cp310-macosx_10_9_universal2.whl
Size 2.1 MB
Tags CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64)
SHA-256 checksum
How to use checksums
bbfd620e1d098e905715cbca765983eba485d59e03938db89f7f8d80d31ac59a
BLAKE2b-256 checksum
How to use checksums
a30e220b11c357c302733f81486ea545528e0be34f37f3ad4bf74ef063e86cc2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.

Transparency log

Release history Release notifications | RSS feed

This release

0.1.0 This release

12 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page