Skip to main content

A seamless bridge from model development to model delivery

Project description

Truss

The simplest way to serve AI/ML models in production

PyPI version ci_status

Why Truss?

  • Write once, run anywhere: Package and test model code, weights, and dependencies with a model server that behaves the same in development and production.
  • Fast developer loop: Implement your model with fast feedback from a live reload server, and skip Docker and Kubernetes configuration with a batteries-included model serving environment.
  • Support for all Python frameworks: From transformers and diffusers to PyTorch and TensorFlow to TensorRT and Triton, Truss supports models created and served with any framework.

See Trusses for popular models including:

and dozens more examples.

Installation

Install Truss with:

pip install --upgrade truss

Quickstart

As a quick example, we'll package a text classification pipeline from the open-source transformers package.

Create a Truss

To get started, create a Truss with the following terminal command:

truss init text-classification

When prompted, give your Truss a name like Text classification.

Then, navigate to the newly created directory:

cd text-classification

Implement the model

One of the two essential files in a Truss is model/model.py. In this file, you write a Model class: an interface between the ML model that you're packaging and the model server that you're running it on.

There are two member functions that you must implement in the Model class:

  • load() loads the model onto the model server. It runs exactly once when the model server is spun up or patched.
  • predict() handles model inference. It runs every time the model server is called.

Here's the complete model/model.py for the text classification model:

from transformers import pipeline


class Model:
    def __init__(self, **kwargs):
        self._model = None

    def load(self):
        self._model = pipeline("text-classification")

    def predict(self, model_input):
        return self._model(model_input)

Add model dependencies

The other essential file in a Truss is config.yaml, which configures the model serving environment. For a complete list of the config options, see the config reference.

The pipeline model relies on Transformers and PyTorch. These dependencies must be specified in the Truss config.

In config.yaml, find the line requirements. Replace the empty list with:

requirements:
  - torch==2.0.1
  - transformers==4.30.0

No other configuration is needed.

Deployment

Truss is maintained by Baseten, which provides infrastructure for running ML models in production. We'll use Baseten as the remote host for your model.

Other remotes are coming soon, starting with AWS SageMaker.

Get an API key

To set up the Baseten remote, you'll need a Baseten API key. If you don't have a Baseten account, no worries, just sign up for an account and you'll be issued plenty of free credits to get you started.

Run truss push

With your Baseten API key ready to paste when prompted, you can deploy your model:

truss push

You can monitor your model deployment from your model dashboard on Baseten.

Invoke the model

After the model has finished deploying, you can invoke it from the terminal.

Invocation

truss predict -d '"Truss is awesome!"'

Response

[
  {
    "label": "POSITIVE",
    "score": 0.999873161315918
  }
]

Truss contributors

Truss is backed by Baseten and built in collaboration with ML engineers worldwide. Special thanks to Stephan Auerhahn @ stability.ai and Daniel Sarfati @ Salad Technologies for their contributions.

We enthusiastically welcome contributions in accordance with our contributors' guide and code of conduct.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

truss-0.9.111.tar.gz (338.9 kB view details)

Uploaded Source

Built Distribution

truss-0.9.111-py3-none-any.whl (453.3 kB view details)

Uploaded Python 3

File details

Details for the file truss-0.9.111.tar.gz.

File metadata

  • Download URL: truss-0.9.111.tar.gz
  • Upload date:
  • Size: 338.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for truss-0.9.111.tar.gz
Algorithm Hash digest
SHA256 53f39360b25dd8fa3cc2dc1c27dc248c59aa48a17e2c64710e6271573cce40d6
MD5 a4c7189af795380cf2038fe0d496a6f1
BLAKE2b-256 0cc9b125524894f983f3c1689da25655e441e206f46aa12d00fdf01c854995a6

See more details on using hashes here.

Provenance

The following attestation bundles were made for truss-0.9.111.tar.gz:

Publisher: release.yml on basetenlabs/truss

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file truss-0.9.111-py3-none-any.whl.

File metadata

  • Download URL: truss-0.9.111-py3-none-any.whl
  • Upload date:
  • Size: 453.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for truss-0.9.111-py3-none-any.whl
Algorithm Hash digest
SHA256 e260ad43a3fa202162b74b13869e835233bcd00f4141b3b5a7681324be32fae0
MD5 d2b86ebf3c8bcd069d763afeaff04658
BLAKE2b-256 9c891735f28f82c8c44e6c015b2129f51dc94c14a34ed13847045a49a317881e

See more details on using hashes here.

Provenance

The following attestation bundles were made for truss-0.9.111-py3-none-any.whl:

Publisher: release.yml on basetenlabs/truss

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page