OSC-LLM
A lightweight LLM inference toolkit focused on minimizing inference latency.
Features
- CUDA Graph: Compilation optimizations that reduce inference latency
- PagedAttention: Efficient KV-cache management enabling long-sequence inference
- Continuous batching: Supports dynamic batch inference optimization
Installation
- Install the latest PyTorch
- Install flash-attn: recommended to use the official prebuilt wheel to avoid build issues
- Install osc-llm
pip install osc-llm --upgrade
Quick Start
Basic Usage
from osc_llm import LLM, SamplingParams
# Initialize the model
llm = LLM("checkpoints/Qwen/Qwen3-0.6B", gpu_memory_utilization=0.5, device="cuda:0")
# Chat
messages = [
{"role": "user", "content": "Hello! What's your name?"}
]
sampling_params = SamplingParams(temperature=0.5, top_p=0.95, top_k=40)
result = llm.chat(messages=messages, sampling_params=sampling_params, enable_thinking=True, stream=False)
print(result)
# Streaming generation
for token in llm.chat(messages=messages, sampling_params=sampling_params, enable_thinking=True, stream=True):
print(token, end="", flush=True)
Supported Models
- Qwen3ForCausalLM
- Qwen2ForCausalLM
1 maintainer
Metadata
1 maintainer
Release files for osc-llm 0.2.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| osc_llm-0.2.5.tar.gz | 12.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| osc_llm-0.2.5-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 29.6 kB
Release files / osc_llm-0.2.5.tar.gz
| Download URL | osc_llm-0.2.5.tar.gz |
|---|---|
| Size | 12.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c64e092d9196a6ff486da3a2f64c04a4b92b6fff1d50922ec0a9c679c03dba36
|
|
BLAKE2b-256 checksum How to use checksums |
458560860cca97c352ae269b3ae2f1a9b691900f9b0a86ab7854e22502815873
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Release files / osc_llm-0.2.5-py3-none-any.whl
| Download URL | osc_llm-0.2.5-py3-none-any.whl |
|---|---|
| Size | 17.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9b78b0513b267cf45caeacaaed432466ab6dc3363b497cfb781da0f3a92698fd
|
|
BLAKE2b-256 checksum How to use checksums |
55d1e0b37188bc4ee71bb456d6f754588ee02dc09f79de491032c5346fd5e709
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
