No project description provided
Project description
| Documentation | Blog | User Forum | Developer Slack (#sig-tpu) |
Upcoming Events 🔥
- Join us at the PyTorch Conference, October 22-23 in San Francisco!
- Join us at Ray Summit, November 3-5 in San Francisco!
- Join us at JAX DevLab on November 18th in Sunnyvale!
Latest News 🔥
Previous News 🔥
About
vLLM TPU is now powered by tpu-inference, an expressive and powerful new hardware plugin unifying JAX and PyTorch under a single lowering path within the vLLM project. The new backend now provides a framework for developers to:
- Push the limits of TPU hardware performance in open source.
- Provide more flexibility to JAX and PyTorch users by running PyTorch model definitions performantly on TPU without any additional code changes, while also extending native support to JAX.
- Retain vLLM standardization: keep the same user experience, telemetry, and interface.
Recommended models and features
Although vLLM TPU’s new unified backend makes out-of-the-box high performance serving possible with any model supported in vLLM, the reality is that we're still in the process of implementing a few core components.
For this reason, we’ve provided a Recommended Models and Features page detailing the models and features that are validated through unit, integration, and performance testing.
Get started
Get started with vLLM on TPUs by following the quickstart guide.
Visit our documentation to learn more.
Compatible TPU Generations
- Recommended: v5e, v6e
- Experimental: v3, v4, v5p
Check out a few v6e recipes here!
Contribute
We're always looking for ways to partner with the community to accelerate vLLM TPU development. If you're interested in contributing to this effort, check out the Contributing guide and Issues to start. We recommend filtering Issues on the good first issue tag if it's your first time contributing.
Contact us
- For technical questions and feature requests, open a GitHub Issue
- For feature requests, please open one on Github here
- For discussing with fellow users, use the TPU support topic in the vLLM Forum
- For coordinating contributions and development, use the Developer Slack
- For collaborations and partnerships, contact us at vllm-tpu@google.com
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file tpu_inference-0.0.1rc1.tar.gz.
File metadata
- Download URL: tpu_inference-0.0.1rc1.tar.gz
- Upload date:
- Size: 362.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e68c945eca55a46c62b7fef727c34403486b67b3fdcc3b439671d1c2ba3bdc9d
|
|
| MD5 |
59ca886104bff52688618ccf1bd6c8f1
|
|
| BLAKE2b-256 |
e17fdca7f11affb5d7846c1a83877730a23009dc6f324fff520b0ed38ef6cd06
|
File details
Details for the file tpu_inference-0.0.1rc1-py3-none-any.whl.
File metadata
- Download URL: tpu_inference-0.0.1rc1-py3-none-any.whl
- Upload date:
- Size: 431.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c5ffebf9b3650da3acf22345bde3c083dda8860e6fee39305534ab034cdb4649
|
|
| MD5 |
c2bf8d87793b41dc0b3dfe9c6a40b3fe
|
|
| BLAKE2b-256 |
97aa6464adebe0bfb8175b09fc9bec8bb7c03a5ea26ca49ce9a8f421e4f6d065
|
