SHARK Shortfin Machine Learning Deployment Tools
Project description
SHARK Shortfin Serving Infrastructure
WARNING: This is an early preview that is in progress. It is not ready for general use.
This sub-project contains components and infrastructure for serving various forms of sharktank compiled models. Instead of coming with models, it defines ABIs that compiled models should adhere to in order to be served. It then allows them to be delivered as web endpoints via popular APIs.
As emulation can be the sincerest form of flattery, this project derives substantial inspiration from vllm and the OpenAI APIs, emulating and interopping with them where possible. It is intended to be the lightest weight possible reference implementation for serving models with an opinionated compiled form, built elsewhere in the project.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for shortfin-0.1.dev3-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3f0eee7d3e86aace5df8652199cda07509794303548d5a1e14ee9067bdc393ca |
|
| MD5 | e77c00385ac771e7cfd8cd30d0a51069 |
|
| BLAKE2b-256 | 022fe52e82c93e02cafa1c2477384c70494047f20646f64edf60a0c7409c94b2 |
