truss 0.1.13rc1

A seamless bridge from model development to model delivery

Truss

Serve any model without boilerplate code

Meet Truss, a seamless bridge from model development to model delivery. Truss presents an open-source standard for packaging models built in any framework for sharing and deployment in any environment, local or production.

Get started with the end-to-end tutorial.

What can I do with Truss?

If you've ever tried to get a model out of a Jupyter notebook, Truss is for you.

Truss exposes just the right amount of complexity around things like Docker and APIs without you really having to think about them. Here are some of the things Truss does:

• 🏎 Turns your Python model into a microservice with a production-ready API endpoint, no need for Flask or Django.
• 🎚 For most popular frameworks, includes automatic model serialization and deserialization.
• 🛍 Freezes dependencies via Docker to make your training environment portable.
• 🕰 Enables rapid iteration with local development that matches your production environment.
• 🗃 Encourages shipping parsing and even business logic alongside your model with integrated pre- and post-processing functions.
• 🤖 Supports running predictions on GPUs. (Currently limited to certain hardware, more coming soon)
• 🙉 Bundles secret management to securely give your model access to API keys.

Installation

Truss requires Python >=3.7, <3.11

To install from PyPi, run:

pip install truss

To download the source code directly (for development), clone this repository and follow the setup commands in our contributors' guide.

Truss is actively developed, and we recommend using the latest version. To update your Truss installation, run:

pip install --upgrade truss

Though Truss is in beta, we do care about backward compatibility. Review the release notes before upgrading, and note that we follow semantic versioning, so any breaking changes require the release of a new major version.

How to use Truss

Generate and serve predictions from a Truss with this Jupyter notebook.

Quickstart: making a Truss

!pip install --upgrade scikit-learn truss

import truss
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris

# Load the iris data set
iris = load_iris()
data_x = iris['data']
data_y = iris['target']

# Train the model
rfc = RandomForestClassifier()
rfc.fit(data_x, data_y)

# Create the Truss (serializing & packaging model)
tr = truss.mk_truss(rfc, target_directory="iris_rfc_truss")

# Serve a prediction from the model
tr.server_predict({"inputs": [[0, 0, 0, 0]]})

Package your model

The truss.mk_truss() command can be used with any supported framework:

• Hugging Face
• LightGBM
• PyTorch
• scikit-learn
• Tensorflow
• XGBoost

But in more complex cases, you can build a Truss manually for any model. Start with truss init my_truss and follow this guide.

Serve your model locally

Serving your model with Truss, on Docker, lets you interface with your model via HTTP requests. Start your model server with:

truss run-image iris_rfc_truss

Then, as long as the container is running, you can invoke the model as an API as follows:

curl -X POST http://127.0.0.1:8080/v1/models/model:predict -d '{"inputs": [[0, 0, 0, 0]]}'

Configure your model for deployment

Truss is configurable to its core. Every Truss must include a file config.yaml in its root directory, which is automatically generated when the Truss is created. However, configuration is optional. Every configurable value has a sensible default, and a completely empty config file is valid.

The Truss we generated above in the quickstart sample has a good example of a typical Truss config:

model_framework: sklearn
model_metadata:
  model_binary_dir: model
  supports_predict_proba: true
python_version: py39
requirements:
- scikit-learn==1.0.2
- threadpoolctl==3.0.0
- joblib==1.1.0
- numpy==1.20.3
- scipy==1.7.3

Follow the configuration guide and use the complete reference of configurable properties to make your Truss perform exactly as you wish.

Deploy your model

You can deploy a Truss anywhere that can run a Docker image, as well as purpose-built platforms like Baseten.

Follow step-by-step deployment guides for the following platforms:

• AWS ECS
• Baseten
• GCP Cloud Run

Contributing

We hope this vision excites you, and we gratefully welcome contributions in accordance with our contributors' guide and code of conduct.

Truss was first developed at Baseten by maintainers Phil Howes, Pankaj Gupta, and Alex Gillmor.

GitHub Codespace

If your organization allows to access to GitHub Codespaces, you can launch a Codespace for truss development. If you are a GPU Codespace, make sure to use the .devcontainer/gpu/devcontainer.json configuration to have access to a GPU and be able to use it in Docker with truss.