OSC-Transformers
🚀 Configuration-driven Modular Transformer Model Building Framework
Flexible, efficient, and extensible Transformer model building tools
中文文档: README-zh.md
✨ Features
- 🔧 Configuration Driven: Build Transformer models through simple configuration files
- 🧩 Modular Design: Support custom registration of various components
- ⚡ High Performance: Support CUDA Graph and Paged Attention
🛠️ Supported Components
| Component Type | Built-in Implementation |
|---|---|
| Attention Mechanism | PagedAttention |
| Feedforward Network | SwiGLU |
| Normalization | RMSNorm |
| Embedding Layer | VocabEmbedding |
| Output Head | LMHead |
📦 Installation
- Install latest version PyTorch
- Install flash-attn: It is recommended to download the official pre-built whl package to avoid compilation issues
- Install osc-transformers
pip install osc-transformers
🚀 Quick Start
Create model.cfg(Qwen3-0.6B):
[model]
@architecture = "TransformerDecoder"
num_layers = 28
prenorm = "True"
[model.attention]
@attention = "PagedAttention"
in_dim = 1024
num_heads = 16
head_dim = 128
num_query_groups = 8
rope_base = 1000000
q_bias = "False"
k_bias = "False"
v_bias = "False"
o_bias = "False"
[model.attention.k_norm]
@normalization = "RMSNorm"
in_dim = 128
eps = 0.000001
[model.attention.q_norm]
@normalization = "RMSNorm"
in_dim = 128
eps = 0.000001
[model.embedding]
@embedding = "VocabEmbedding"
num_embeddings = 151936
embedding_dim = 1024
[model.feedforward]
@feedforward = "SwiGLU"
in_dim = 1024
hidden_dim = 3072
up_bias = "False"
gate_bias = "False"
down_bias = "False"
[model.head]
@head = "LMHead"
in_dim = 1024
out_dim = 151936
bias = "False"
[model.norm]
@normalization = "RMSNorm"
in_dim = 1024
eps = 0.000001
Code example:
from osc_transformers import TransformerDecoder, Sequence, SamplingParams
# Build model
model = TransformerDecoder.from_config("model.cfg")
model.setup(gpu_memory_utilization=0.9, max_model_len=40960, device="cuda:0")
# Batch inference
seqs = [Sequence(token_ids=[1,2,3,4,5,6,7,8,9,10], sampling_params=SamplingParams(temperature=0.5, max_generate_tokens=1024))]
seqs = model.batch(seqs)
# Streaming inference
seq = Sequence(token_ids=[1,2,3,4,5,6,7,8,9,10], sampling_params=SamplingParams(temperature=0.5, max_generate_tokens=1024))
for token in model.stream(seq):
pass
📚 Inference Performance
osc-transformers bench examples/configs/qwen3-0_6B.cfg --num_seqs 64 --max_input_len 1024 --max_output_len 1024 --gpu_memory_utilization 0.9
| Architecture | Model | Device | Throughput(tokens/s) |
|---|---|---|---|
| TransformerDecoder | Qwen3-0.6B | 4090 | 5400 |
| TransformerDecoder | Qwen3-0.6B | 3090 | 4000 |
📚 Acknowledgments
The core code of this project mainly comes from the following projects:
🤝 Contributing
Welcome to submit Issue and Pull Request!
📄 License
MIT License
Metadata
Release files for osc-transformers 0.3.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| osc_transformers-0.3.1.tar.gz | 25.8 kB | Without provenance | Details for osc_transformers-0.3.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| osc_transformers-0.3.1-py3-none-any.whl | Python 3 | none | any | Details for osc_transformers-0.3.1-py3-none-any.whl |
Total release size: 56.4 kB
Release files / osc_transformers-0.3.1.tar.gz
| Download URL | osc_transformers-0.3.1.tar.gz |
|---|---|
| Size | 25.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
487ecda3353d3d89f621794182bf313b72594f475defd96acf377c7adc9526e4
|
|
BLAKE2b-256 checksum How to use checksums |
2759d5bfbfadc70022d0654d39bad6e17e95bed371a509660777813b543f79f6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.18 {"installer":{"name":"uv","version":"0.9.18","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / osc_transformers-0.3.1-py3-none-any.whl
| Download URL | osc_transformers-0.3.1-py3-none-any.whl |
|---|---|
| Size | 30.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0abc454d7a92014e2fb3e4c469edb87a9027ea6e2ad4585ba531bcded443446d
|
|
BLAKE2b-256 checksum How to use checksums |
1c54e7279ebe6b65f0f8de964b7b50b1f0f145ff077b9e6ed3b850c4fa58c876
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.18 {"installer":{"name":"uv","version":"0.9.18","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for osc-transformers 0.3.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.