A seamless bridge from model development to model delivery
Serve any model without boilerplate code
Meet Truss, a seamless bridge from model development to model delivery. Truss presents an open-source standard for packaging models built in any framework for sharing and deployment in any environment, local or production.
Get started with the end-to-end tutorial.
What can I do with Truss?
If you've ever tried to get a model out of a Jupyter notebook, Truss is for you.
Truss exposes just the right amount of complexity around things like Docker and APIs without you really having to think about them. Here are some of the things Truss does:
- 🏎 Turns your Python model into a microservice with a production-ready API endpoint, no need for Flask or Django.
- 🎚 For most popular frameworks, includes automatic model serialization and deserialization.
- 🛍 Freezes dependencies via Docker to make your training environment portable.
- 🕰 Enables rapid iteration with local development that matches your production environment.
- 🗃 Encourages shipping parsing and even business logic alongside your model with integrated pre- and post-processing functions.
- 🤖 Supports running predictions on GPUs. (Currently limited to certain hardware, more coming soon)
- 🙉 Bundles secret management to securely give your model access to API keys.
Truss requires Python >=3.7, <3.11
To install from PyPi, run:
pip install truss
To download the source code directly (for development), clone this repository and follow the setup commands in our contributors' guide.
Truss is actively developed, and we recommend using the latest version. To update your Truss installation, run:
pip install --upgrade truss
Though Truss is in beta, we do care about backward compatibility. Review the release notes before upgrading, and note that we follow semantic versioning, so any breaking changes require the release of a new major version.
How to use Truss
Generate and serve predictions from a Truss with this Jupyter notebook.
Quickstart: making a Truss
!pip install --upgrade scikit-learn truss import truss from sklearn.ensemble import RandomForestClassifier from sklearn.datasets import load_iris # Load the iris data set iris = load_iris() data_x = iris['data'] data_y = iris['target'] # Train the model rfc = RandomForestClassifier() rfc.fit(data_x, data_y) # Create the Truss (serializing & packaging model) tr = truss.create(rfc, target_directory="iris_rfc_truss") # Serve a prediction from the model tr.predict([[0, 0, 0, 0]])
Package your model
truss.create() command can be used with any supported framework:
But in more complex cases, you can build a Truss manually for any model. Start with
truss init my_truss and follow this guide.
Serve your model locally
Serving your model with Truss, on Docker, lets you interface with your model via HTTP requests. Start your model server with:
truss run-image iris_rfc_truss
Then, as long as the container is running, you can invoke the model as an API as follows:
curl -X POST http://127.0.0.1:8080/v1/models/model:predict -d '[[0, 0, 0, 0]]'
Configure your model for deployment
Truss is configurable to its core. Every Truss must include a file
config.yaml in its root directory, which is automatically generated when the Truss is created. However, configuration is optional. Every configurable value has a sensible default, and a completely empty config file is valid.
The Truss we generated above in the quickstart sample has a good example of a typical Truss config:
model_framework: sklearn model_metadata: model_binary_dir: model supports_predict_proba: true python_version: py39 requirements: - scikit-learn==1.0.2 - threadpoolctl==3.0.0 - joblib==1.1.0 - numpy==1.20.3 - scipy==1.7.3
Follow the configuration guide and use the complete reference of configurable properties to make your Truss perform exactly as you wish.
Deploy your model
You can deploy a Truss anywhere that can run a Docker image, as well as purpose-built platforms like Baseten.
Follow step-by-step deployment guides for the following platforms:
We hope this vision excites you, and we gratefully welcome contributions in accordance with our contributors' guide and code of conduct.
Truss was first developed at Baseten by maintainers Phil Howes, Pankaj Gupta, and Alex Gillmor.
If your organization allows to access to GitHub Codespaces, you can launch a Codespace for truss development. If you are a GPU Codespace, make sure to use the
.devcontainer/gpu/devcontainer.json configuration to have access to a GPU and be able to use it in Docker with truss.
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.