A seamless bridge from model development to model delivery
Project description
Truss
The simplest way to serve AI/ML models in production
Why Truss?
- Write once, run anywhere: Package and test model code, weights, and dependencies with a model server that behaves the same in development and production.
- Fast developer loop: Implement your model with fast feedback from a live reload server, and skip Docker and Kubernetes configuration with a batteries-included model serving environment.
- Support for all Python frameworks: From
transformers
anddiffusers
toPyTorch
andTensorFlow
toTensorRT
andTriton
, Truss supports models created and served with any framework.
See Trusses for popular models including:
- 🦙 Llama 2 7B (13B) (70B)
- 🎨 Stable Diffusion XL
- 🗣 Whisper
and dozens more examples.
Installation
Install Truss with:
pip install --upgrade truss
Quickstart
As a quick example, we'll package a text classification pipeline from the open-source transformers
package.
Create a Truss
To get started, create a Truss with the following terminal command:
truss init text-classification
When prompted, give your Truss a name like Text classification
.
Then, navigate to the newly created directory:
cd text-classification
Implement the model
One of the two essential files in a Truss is model/model.py
. In this file, you write a Model
class: an interface between the ML model that you're packaging and the model server that you're running it on.
There are two member functions that you must implement in the Model
class:
load()
loads the model onto the model server. It runs exactly once when the model server is spun up or patched.predict()
handles model inference. It runs every time the model server is called.
Here's the complete model/model.py
for the text classification model:
from transformers import pipeline
class Model:
def __init__(self, **kwargs):
self._model = None
def load(self):
self._model = pipeline("text-classification")
def predict(self, model_input):
return self._model(model_input)
Add model dependencies
The other essential file in a Truss is config.yaml
, which configures the model serving environment. For a complete list of the config options, see the config reference.
The pipeline model relies on Transformers and PyTorch. These dependencies must be specified in the Truss config.
In config.yaml
, find the line requirements
. Replace the empty list with:
requirements:
- torch==2.0.1
- transformers==4.30.0
No other configuration is needed.
Deployment
Truss is maintained by Baseten, which provides infrastructure for running ML models in production. We'll use Baseten as the remote host for your model.
Other remotes are coming soon, starting with AWS SageMaker.
Get an API key
To set up the Baseten remote, you'll need a Baseten API key. If you don't have a Baseten account, no worries, just sign up for an account and you'll be issued plenty of free credits to get you started.
Run truss push
With your Baseten API key ready to paste when prompted, you can deploy your model:
truss push
You can monitor your model deployment from your model dashboard on Baseten.
Invoke the model
After the model has finished deploying, you can invoke it from the terminal.
Invocation
truss predict -d '"Truss is awesome!"'
Response
[
{
"label": "POSITIVE",
"score": 0.999873161315918
}
]
Truss contributors
Truss is backed by Baseten and built in collaboration with ML engineers worldwide. Special thanks to Stephan Auerhahn @ stability.ai and Daniel Sarfati @ Salad Technologies for their contributions.
We enthusiastically welcome contributions in accordance with our contributors' guide and code of conduct.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for truss-0.9.27rc3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1d23cef161959c6d19b450b9534730770a3993989abd19ca7cf5bcbd6ab54944 |
|
MD5 | 511178c755216eca3840fd83befd4832 |
|
BLAKE2b-256 | 7c1f92d81a7d78b1b62e7c274bc9c02878a1f2e02ef6cbb939527992b850f4c4 |