This release is a pre-release and may not be stable for production use.
Truss
The simplest way to serve AI/ML models in production
Truss is the CLI for deploying and serving ML models on Baseten. Package your model's serving logic in Python, launch training jobs, and deploy to production—Truss handles containerization, dependency management, and GPU configuration.
Truss lets you serve models with the Baseten Inference Stack as well as deploy models from any open-source framework: vLLM, SGLang, TensorRT-LLM, transformers, diffusers, PyTorch, TensorFlow, and more.
Get started | 100+ examples | Documentation
Why Truss?
- Write once, run anywhere: Package model code, weights, and dependencies with a model server that behaves the same in development and production.
- Fast developer loop: Iterate with live reload, skip Docker and Kubernetes configuration, and use a batteries-included serving environment.
- Support for all Python frameworks: From
transformersanddiffusersto PyTorch and TensorFlow to vLLM, SGLang, and TensorRT-LLM, Truss supports models created and served with any framework. - Production-ready: Built-in support for GPUs, secrets, caching, and autoscaling when deployed to Baseten or your own infrastructure.
Installation
Install Truss with:
pip install --upgrade truss
Quickstart
Deploying a model to Baseten via Truss turns a Hugging Face model into a production-ready API endpoint. You write a config.yaml that specifies the model, the hardware, and the engine, then uvx truss push builds a TensorRT-optimized container and deploys it. No Python code, no Dockerfile, no container management.
This guide walks through deploying Qwen 2.5 3B Instruct, a small but capable LLM, from a config file to a production API. You'll set up Truss, write a config, deploy to Baseten, and call the model's OpenAI-compatible endpoint.
Set up your environment
Before you begin:
- Sign up or sign in to Baseten.
- Install uv, a fast Python package manager. This guide uses
uvxto run Truss commands without a separate install step.
Authenticate with Baseten
Log in:
uvx truss login
The CLI asks how you want to authenticate. Generate an API key from Settings > API keys if you pick the paste path:
💻 Let's add a Baseten remote!
How would you like to authenticate?
Paste an API key
Log in via browser (OAuth)
🤫 Quietly paste your API_KEY:
Skip the picker with --browser or --api-key:
uvx truss login --browser
uvx truss login --api-key "paste-your-api-key-here"
BASETEN_API_KEY is what the OpenAI client uses later to call the model. It does not skip truss login.
Create a Truss project
Scaffold a new project:
uvx truss init qwen-2.5-3b && cd qwen-2.5-3b
When prompted, name the model Qwen 2.5 3B.
? 📦 Name this model: Qwen 2.5 3B
Truss Qwen 2.5 3B was created in ~/qwen-2.5-3b
This creates a directory with a config.yaml, a model/ directory, and supporting files. For engine-based deployments like this one, you only need config.yaml. The model/ directory is for custom Python code when you need custom preprocessing, postprocessing, or unsupported model architectures.
Write the config
Replace the contents of config.yaml with:
model_metadata:
tags:
- openai-compatible
model_name: Qwen-2.5-3B
resources:
accelerator: L4
use_gpu: true
trt_llm:
build:
base_model: decoder
checkpoint_repository:
source: HF
repo: "Qwen/Qwen2.5-3B-Instruct"
max_seq_len: 8192
quantization_type: fp8
tensor_parallel_count: 1
num_builder_gpus: 2
That's the entire deployment specification.
model_nameidentifies the model in your Baseten dashboard.resourcesselects an L4 GPU (24 GB VRAM), which is plenty for a 3B parameter model.trt_llmtells Baseten to use Engine-Builder-LLM, which compiles the model with TensorRT-LLM for optimized inference.checkpoint_repositorypoints to the model weights on Hugging Face. Qwen 2.5 3B Instruct is ungated, so no access token is needed.quantization_type: fp8compresses weights to 8-bit floating point, cutting memory usage roughly in half with negligible quality loss.max_seq_len: 8192sets the maximum context length for requests.num_builder_gpus: 2uses two GPUs during the build phase. FP8 quantization needs more GPU memory at compile time than at inference. The first-model guide says a single L4 can run out of memory during compilation without this.
Deploy
Push the model to Baseten. A default push is a published deployment. --watch (development mode with live reload) is not supported for TRT-LLM. For custom Python models, see Customize a model.
uvx truss push
You should see:
Deploying as a published deployment. Use --watch for a development deployment.
✨ Model Qwen 2.5 3B was successfully pushed ✨
Model ID: abc1d2ef
Deployment ID: xyz123
Endpoint: https://model-abc1d2ef.api.baseten.co
Logs: https://app.baseten.co/models/abc1d2ef/logs/xyz123
You'll need the model ID to call the model's API. You can also find it in your Baseten dashboard.
Baseten now downloads the model weights from Hugging Face, compiles them with TensorRT-LLM, and deploys the resulting container to an L4 GPU. You can watch progress in the logs linked above.
Call the model
Engine-based deployments serve an OpenAI-compatible API. Once the deployment shows "Active" in the dashboard, call it using the OpenAI SDK or cURL. Replace {model_id} with your model ID from the deployment output.
Install the OpenAI SDK if you don't have it, and export the API key the client will send:
uv pip install openai
export BASETEN_API_KEY="paste-your-api-key-here"
Create a chat completion:
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["BASETEN_API_KEY"],
base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)
response = client.chat.completions.create(
model="Qwen-2.5-3B",
messages=[
{"role": "user", "content": "What is machine learning?"}
],
)
print(response.choices[0].message.content)
You should see a response like:
Machine learning is a branch of artificial intelligence where systems learn
patterns from data to make predictions or decisions without being explicitly
programmed for each task...
Any code that works with the OpenAI SDK works with your deployment. Just point the base_url at your model's endpoint.
Your model ID is the string after /models/ in the logs URL from uvx truss push. You can also find it in your Baseten dashboard.
IDE support
Truss ships a JSON schema for config.yaml. Projects created with truss init include a schema reference automatically, giving you autocompletion, hover docs, and validation in any editor that supports the YAML language server (VS Code, JetBrains, Neovim, and others).
To add schema support to an existing config.yaml, add this comment as the first line:
# yaml-language-server: $schema=https://raw.githubusercontent.com/basetenlabs/truss/main/truss/config.schema.json
7 maintainers
Metadata
7 maintainers
Release files for truss 0.18.36rc1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| truss-0.18.36rc1.tar.gz | 673.9 kB | With provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| truss-0.18.36rc1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.5 MB
Release files / truss-0.18.36rc1.tar.gz
| Download URL | truss-0.18.36rc1.tar.gz |
|---|---|
| Size | 673.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d07c3bbb910e03aa00d9f65fe15dfdac225a003d3ec8f1e8d7e3e813f33524ce
|
|
BLAKE2b-256 checksum How to use checksums |
e1b22e6a29a86edd0567718680ef31ee0e0bd36e09f65543387a4679efd679c6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / truss-0.18.36rc1-py3-none-any.whl
| Download URL | truss-0.18.36rc1-py3-none-any.whl |
|---|---|
| Size | 799.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
71db3ce6feefd0d4ee48e47e48a4f907faccc081496354c896ebb92b88d26a87
|
|
BLAKE2b-256 checksum How to use checksums |
7a743c4b2134167e8c837ee00048ca76132ab7525f8f8039d7df1aee019d1bf2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logSecurity for truss 0.18.36rc1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for truss-0.18.36rc1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
