custom-transformer
A PyTorch-based transformer library for language modeling and automatic speech recognition (ASR). Includes both a numpy-based multi-head attention implementation (mytorch) and a full PyTorch transformer toolkit (transformerlib) with training, decoding, and data loading utilities.
Installation
pip install custom-transformer
Features
mytorch — NumPy Attention Primitives
Pure numpy implementations for educational and prototyping purposes:
Linear— fully-connected layer with forward and backward passesSoftmax— numerically stable softmax activationScaledDotProductAttention— attention mechanism with optional maskingMultiHeadAttention— multi-head attention with split/concat head logic
from mytorch.nn import MultiHeadAttention
mha = MultiHeadAttention(d_model=512, num_heads=8)
transformerlib — PyTorch Transformer Toolkit
Full-featured transformer models and training infrastructure:
Models
DecoderOnlyTransformer— GPT-style causal language modelEncoderDecoderTransformer— encoder-decoder for sequence-to-sequence tasks (e.g., ASR)- Pre-LN architecture with sinusoidal positional encoding
- Weight tying, layer dropout, and mixed precision support
from_pretrained_decoderfor initializing encoder-decoder models from pretrained decoder weights
Data
LMDataset— text dataset with tokenization, SOS/EOS framing, and collationASRDataset— speech dataset with filterbank features, global MVN normalization, and SpecAugmentH4Tokenizer— BPE tokenizer wrapper (char, 1k, 5k, 10k vocab sizes included)
Training
LMTrainer— language model training with gradient accumulation, mixed precision, and WandB loggingASRTrainer— ASR training with CTC + cross-entropy joint lossProgressiveTrainer— curriculum learning with gradual layer unfreezing and data subsetting
Decoding
SequenceGenerator— greedy search, beam search, and nucleus sampling- Language model shallow fusion for ASR recognition
Quick Start
from transformerlib.model import DecoderOnlyTransformer
model = DecoderOnlyTransformer(
num_layers=6,
d_model=512,
num_heads=8,
d_ff=2048,
dropout=0.1,
max_len=512,
num_classes=10000
)
from transformerlib.model import EncoderDecoderTransformer
model = EncoderDecoderTransformer(
num_encoder_layers=6,
num_decoder_layers=6,
d_model=512,
num_heads=8,
d_ff=2048,
dropout=0.1,
max_len=512,
num_classes=1000,
feat_dim=80
)
Architecture
mytorch/
nn/
linear.py
activation.py
scaled_dot_product_attention.py
multi_head_attention.py
transformerlib/
model/ — masks, positional encoding, sublayers, encoder/decoder layers, transformers
data/ — tokenizer, LM dataset, ASR dataset
trainers/ — base trainer, LM trainer, ASR trainer, progressive trainer
decoding/ — sequence generator (greedy, beam, sampling)
utils/ — optimizer and LR scheduler factories
Requirements
- Python >= 3.9
- PyTorch >= 2.0
- torchaudio
- numpy, tqdm, wandb, torchmetrics, tokenizers, pandas, matplotlib
License
MIT
Metadata
Release files for custom-transformer 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
Total release size: 38.1 MB
Release files / custom_transformer-0.1.0-cp313-cp313-win_amd64.whl
| Download URL | custom_transformer-0.1.0-cp313-cp313-win_amd64.whl |
|---|---|
| Size | 1.0 MB |
| Tags | CPython 3.13 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
e7be9e7b93604f7e07bc1ab60c5c29854bf6c3c54c6d5210b1a19bad0cea22d4
|
|
BLAKE2b-256 checksum How to use checksums |
55b72a9dfa7df64db5f40b2453a77eb30f73a8acbc74347b197688053db6ecb2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.
Transparency logRelease files / custom_transformer-0.1.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | custom_transformer-0.1.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 6.4 MB |
| Tags | CPython 3.13 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
ac2c498ede2db9e6c55be60e6017a60b094f060508bf9b0b96cae62abde6e51d
|
|
BLAKE2b-256 checksum How to use checksums |
4f1e829fc134260a87f4c2c3b69da67991b65baed83f5ca0bf6c09fa41441755
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.
Transparency logRelease files / custom_transformer-0.1.0-cp313-cp313-macosx_10_13_universal2.whl
| Download URL | custom_transformer-0.1.0-cp313-cp313-macosx_10_13_universal2.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.13 macOS 10.13+ universal2 (ARM64, x86-64) |
|
SHA-256 checksum How to use checksums |
e78e8ebc41d253b8f996a0d0fabae6e4069478866a25ede6aa24df58058f860c
|
|
BLAKE2b-256 checksum How to use checksums |
692054d2f006759a15041b7b8a31be88d92b2ddec5f439fbda1d365184f1d557
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.
Transparency logRelease files / custom_transformer-0.1.0-cp312-cp312-win_amd64.whl
| Download URL | custom_transformer-0.1.0-cp312-cp312-win_amd64.whl |
|---|---|
| Size | 1.0 MB |
| Tags | CPython 3.12 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
65c6b0f33c039ea53cc1070e874b6e67a5772f631acd7a295c1979837722f09b
|
|
BLAKE2b-256 checksum How to use checksums |
2e546ffa6eb8589ce15cbe8279d196149044cf563b179afe44faf77b92c1aace
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.
Transparency logRelease files / custom_transformer-0.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | custom_transformer-0.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 6.6 MB |
| Tags | CPython 3.12 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
d456f7ac73cec1584ccceb746a93d6d75aad77dcc7b6c0e4e682640c8bb7242b
|
|
BLAKE2b-256 checksum How to use checksums |
c5e43e1d22164783e7f4dee4d25e85a139d672293bee68605835eef806accf5f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.
Transparency logRelease files / custom_transformer-0.1.0-cp312-cp312-macosx_10_13_universal2.whl
| Download URL | custom_transformer-0.1.0-cp312-cp312-macosx_10_13_universal2.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.12 macOS 10.13+ universal2 (ARM64, x86-64) |
|
SHA-256 checksum How to use checksums |
5a94ed782428d43e65ba1624042ff955eeb34b4a13c684b7d85d43baf179110c
|
|
BLAKE2b-256 checksum How to use checksums |
8c045c366669128b8cd5ea47527cc3f785f592f336150a1852ca1b5c4c0321f8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.
Transparency logRelease files / custom_transformer-0.1.0-cp311-cp311-win_amd64.whl
| Download URL | custom_transformer-0.1.0-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 1.1 MB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
028ebbf76e09c349252185523e93e46b6c1d64e323b948bcc1321ad91d332c7c
|
|
BLAKE2b-256 checksum How to use checksums |
208ed16619b522b47f31e4582bad1fcb2121fc0edecf891c0768a056973903e6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.
Transparency logRelease files / custom_transformer-0.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | custom_transformer-0.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 6.4 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
023c465c9847e010e85a306fe6a2eb96c1bcc3dbc07aaee75e8b0786c1c1e98f
|
|
BLAKE2b-256 checksum How to use checksums |
da46b446b7a300fb13c88b2bd20f954b8fb150d3ca7f9e45328171fb227afe39
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.
Transparency logRelease files / custom_transformer-0.1.0-cp311-cp311-macosx_10_9_universal2.whl
| Download URL | custom_transformer-0.1.0-cp311-cp311-macosx_10_9_universal2.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64) |
|
SHA-256 checksum How to use checksums |
dcec103ce64af1ec90cf2a5eeb0eac9f4de9ae8edf6ea2995f662286d7f28332
|
|
BLAKE2b-256 checksum How to use checksums |
bc234576257bce46a04d8415c4026950897a4fbd48fb06b9af4c7e3e6ae29128
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.
Transparency logRelease files / custom_transformer-0.1.0-cp310-cp310-win_amd64.whl
| Download URL | custom_transformer-0.1.0-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 1.1 MB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
7429d7d1b051e84f7bef807b600534359e3991319b0e0f998400af149af60cf9
|
|
BLAKE2b-256 checksum How to use checksums |
66e8d37e48022222fda59a29933c9849dcd37d32b4a2c9587c9392a3843259bd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.
Transparency logRelease files / custom_transformer-0.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | custom_transformer-0.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 6.0 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
a2721f066ff65e25d23b6b15c608be500ba3242ff47dfc7de2cbb4c0361d73e7
|
|
BLAKE2b-256 checksum How to use checksums |
18595ecf580c4c8b351d6954a2b621a7bbed24962786d90ef8f2c5388eb23743
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.
Transparency logRelease files / custom_transformer-0.1.0-cp310-cp310-macosx_10_9_universal2.whl
| Download URL | custom_transformer-0.1.0-cp310-cp310-macosx_10_9_universal2.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64) |
|
SHA-256 checksum How to use checksums |
bbfd620e1d098e905715cbca765983eba485d59e03938db89f7f8d80d31ac59a
|
|
BLAKE2b-256 checksum How to use checksums |
a30e220b11c357c302733f81486ea545528e0be34f37f3ad4bf74ef063e86cc2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 10, 2026.
Transparency log