Unified Model Serving Framework
🍱 Build model inference APIs and multi-model serving systems with any open-source or custom AI models. 👉 Join our Slack community!
What is BentoML?
BentoML is a Python library for building online serving systems optimized for AI apps and model inference.
- 🍱 Easily build APIs for Any AI/ML Model. Turn any model inference script into a REST API server with just a few lines of code and standard Python type hints.
- 🐳 Docker Containers made simple. No more dependency hell! Manage your environments, dependencies and model versions with a simple config file. BentoML automatically generates Docker images, ensures reproducibility, and simplifies how you deploy to different environments.
- 🧭 Maximize CPU/GPU utilization. Build high performance inference APIs leveraging built-in serving optimization features like dynamic batching, model parallelism, multi-stage pipeline and multi-model inference-graph orchestration.
- 👩💻 Fully customizable. Easily implement your own APIs or task queues, with custom business logic, model inference and multi-model composition. Supports any ML framework, modality, and inference runtime.
- 🚀 Ready for Production. Develop, run and debug locally. Seamlessly deploy to production with Docker containers or BentoCloud.
Getting started
Install BentoML:
# Requires Python≥3.9
pip install -U bentoml
Define APIs in a service.py file.
from __future__ import annotations
import bentoml
@bentoml.service(
resources={"cpu": "4"}
)
class Summarization:
def __init__(self) -> None:
import torch
from transformers import pipeline
device = "cuda" if torch.cuda.is_available() else "cpu"
self.pipeline = pipeline('summarization', device=device)
@bentoml.api(batchable=True)
def summarize(self, texts: list[str]) -> list[str]:
results = self.pipeline(texts)
return [item['summary_text'] for item in results]
Run the service code locally (serving at http://localhost:3000 by default):
pip install torch transformers # additional dependencies for local run
bentoml serve
Now you can run inference from your browser at http://localhost:3000 or with a Python script:
import bentoml
with bentoml.SyncHTTPClient('http://localhost:3000') as client:
summarized_text: str = client.summarize([bentoml.__doc__])[0]
print(f"Result: {summarized_text}")
Deploying your first Bento
To deploy your BentoML Service code, configure the runtime environment in service.py.
...
my_image = bentoml.images.PythonImage(python_version="3.11") \
.requirements_file("requirements.txt")
@bentoml.service(
image=my_image,
resources={"cpu": "4"},
traffic={"timeout": 30},
)
...
Then, choose one of the following ways for deployment:
🐳 Docker Container
Run bentoml build to package necessary code, models, dependency configs into a Bento - the standardized deployable artifact in BentoML:
bentoml build
Ensure Docker is running. Generate a Docker container image for deployment:
bentoml containerize summarization:latest
Run the generated image:
docker run --rm -p 3000:3000 summarization:latest
☁️ BentoCloud
BentoCloud provides compute infrastructure for rapid and reliable GenAI adoption. It helps speed up your BentoML development process leveraging cloud compute resources, and simplify how you deploy, scale and operate BentoML in production.
Sign up for BentoCloud for personal access; for enterprise use cases, contact our team.
# After signup, run the following command to create an API token:
bentoml cloud login
# Deploy from current directory:
bentoml deploy
For detailed explanations, read the Hello World example.
Examples
- LLMs: Llama 3.2, Mixtral, Solar, and Mistral.
- Image Generation: Stable Diffusion 3 Medium, Stable Video Diffusion, Stable Diffusion XL Turbo, ControlNet, and LCM LoRAs.
- Embeddings: SentenceTransformers and ColPali
- Audio: ChatTTS, XTTS, WhisperX, Bark
- Computer Vision: YOLO and ResNet
- Advanced examples: Function calling, LangGraph, CrewAI
Check out the full list for more sample code and usage.
Advanced topics
- Model composition
- Workers and model parallelization
- Adaptive batching
- GPU inference
- Distributed serving systems
- Concurrency and autoscaling
- Model loading and Model Store
- Observability
- BentoCloud deployment
See Documentation for more tutorials and guides.
Community
Get involved and join our Community Slack 💬, where thousands of AI/ML engineers help each other, contribute to the project, and talk about building AI products.
To report a bug or suggest a feature request, use GitHub Issues.
Contributing
There are many ways to contribute to the project:
- Report bugs and "Thumbs up" on issues that are relevant to you.
- Investigate issues and review other developers' pull requests.
- Contribute code or documentation to the project by submitting a GitHub pull request.
- Check out the Contributing Guide and Development Guide to learn more.
- Share your feedback and discuss roadmap plans in the
#bentoml-contributorschannel here.
Thanks to all of our amazing contributors!
Usage tracking and feedback
The BentoML framework collects anonymous usage data that helps our community improve the product. Only BentoML's internal API calls are being reported. This excludes any sensitive information, such as user code, model data, model names, or stack traces. Here's the code used for usage tracking. You can opt-out of usage tracking by the --do-not-track CLI option:
bentoml [command] --do-not-track
Or by setting the environment variable:
export BENTOML_DO_NOT_TRACK=True
License
Release files for bentoml 1.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| bentoml-1.4.0.tar.gz | 968.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| bentoml-1.4.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 2.1 MB
Release files / bentoml-1.4.0.tar.gz
| Download URL | bentoml-1.4.0.tar.gz |
|---|---|
| Size | 968.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2e6b9d37c7af19112f1772c2383659412a324237703c64fb806ff8738ad60d86
|
|
BLAKE2b-256 checksum How to use checksums |
ab3dcb8f05176431e2711ea546439618a74ab95cc8ae34ade617277cf9274da8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 20, 2025.
Transparency logRelease files / bentoml-1.4.0-py3-none-any.whl
| Download URL | bentoml-1.4.0-py3-none-any.whl |
|---|---|
| Size | 1.1 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0bb7e3d676dac499929dd0317c6c0612b45264e407108917dfbc74042f837194
|
|
BLAKE2b-256 checksum How to use checksums |
5fe1cc8c93e017188009ed08122c0f63fc6f9bcce218e7be312fb9f674c227b8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 20, 2025.
Transparency log
