Skip to main content

Kernel Library for SGLang

Project description

SGL Kernel

Kernel Library for SGLang

PyPI

Installation

For CUDA 11.8:

pip3 install sgl-kernel -i https://docs.sglang.ai/whl/cu118

For CUDA 12.1 or CUDA 12.4:

pip3 install sgl-kernel

Developer Guide

Development Environment Setup

Use Docker to set up the development environment. See Docker setup guide.

Create and enter development container:

docker run -itd --shm-size 32g --gpus all -v $HOME/.cache:/root/.cache --ipc=host --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh
docker exec -it sglang_zhyncs /bin/zsh

Project Structure

Dependencies

Third-party libraries:

Kernel Development

Steps to add a new kernel:

  1. Implement the kernel in csrc
  2. Expose the interface in include/sgl_kernel_ops.h
  3. Create torch extension in csrc/torch_extension.cc
  4. Update setup.py to include new CUDA source
  5. Expose Python interface in python

Build & Install

Development build:

make build

Note:

The sgl-kernel is rapidly evolving. If you experience a compilation failure, try using make rebuild.

Testing & Benchmarking

  1. Add pytest tests in tests/
  2. Add benchmarks using triton benchmark in benchmark/
  3. Run test suite

Release new version

Update version in pyproject.toml and version.py

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

File details

Details for the file sgl_kernel-0.0.5.post1-cp39-abi3-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for sgl_kernel-0.0.5.post1-cp39-abi3-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 f7236bfd66dbe0e3fa291685afb0a2dc9fa58e17359bb66ec89473824dd51a84
MD5 153efdc8f639dff67d8f32b9ac85aa0b
BLAKE2b-256 383ca2db42482496a49952962a97bf526dac2b585eeddbfc2e51a5894f7d79ea

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page