Skip to main content

BentoML: Build Production-Grade AI Applications

Project description

bentoml

BentoML: The Unified AI Application Framework

pypi_status CI Twitter Community

BentoML is a framework for building reliable, scalable, and cost-efficient AI applications. It comes with everything you need for model serving, application packaging, and production deployment.

👉 Join our Slack community!

Highlights

🍱 Bento is the container for AI apps

  • Open standard and SDK for AI apps, pack your code, inference pipelines, model files, dependencies, and runtime configurations in a Bento.
  • Auto-generate API servers, supporting REST API, gRPC, and long-running inference jobs.
  • Auto-generate Docker container images.

🏄 Freedom to build with any AI models

🤖️ Inference optimization for AI applications

  • Integrate with high-performance runtimes such as ONNX-runtime and TorchScript to boost response time and throughput.
  • Support parallel processing of model inferences for improved speed and efficiency.
  • Implement adaptive batching to optimize processing.
  • Built-in optimization for specific model architectures (like OpenLLM for LLMs).

🍭 Simplify modern AI application architecture

  • Python-first! Effortlessly scale complex AI workloads.
  • Enable GPU inference without the headache.
  • Compose multiple models to run concurrently or sequentially, over multiple GPUs or on a Kubernetes Cluster.
  • Natively integrates with MLFlow, LangChain, Kubeflow, Triton, Spark, Ray, and many more to complete your production AI stack.

🚀 Deploy anywhere

  • One-click deployment to ☁️ BentoCloud, the Serverless platform made for hosting and operating AI apps.
  • Scalable BentoML deployment with 🦄️ Yatai on Kubernetes.
  • Deploy auto-generated container images anywhere Docker runs.

Documentation

🛠️ What you can build with BentoML

Getting started

This example demonstrates how to serve and deploy a simple text summarization application.

Serving a model locally

Install dependencies:

pip install torch transformers "bentoml>=1.2.0a0"

Define the serving logic of your model in a service.py file.

from __future__ import annotations
import bentoml
from transformers import pipeline


@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 10},
)
class Summarization:
    def __init__(self) -> None:
        # Load model into pipeline
        self.pipeline = pipeline('summarization')

    @bentoml.api
    def summarize(self, text: str) -> str:
        result = self.pipeline(text)
        return result[0]['summary_text']

Run this BentoML Service locally, which is accessible at http://localhost:3000.

bentoml serve service:Summarization

Send a request to summarize a short news paragraph:

curl -X 'POST' \
  'http://localhost:3000/summarize' \
  -H 'accept: text/plain' \
  -H 'Content-Type: application/json' \
  -d '{
  "text": "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson'\''s cat, Whiskers, performed what witnesses are calling a '\''miraculous and gravity-defying leap.'\'' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson'\''s backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as '\''The Leap of the Century."
}'

Deployment

After your Service is ready, you can deploy it to BentoCloud or as a Docker image.

First, create a bentofile.yaml file for building a Bento.

service: "service:Summarization"
labels:
  owner: bentoml-team
  project: gallery
include:
  - "*.py"
python:
  packages:
  - torch
  - transformers

Then, choose one of the following ways for deployment:

BentoCloud

Make sure you have logged in to BentoCloud and then run the following command:

bentoml deploy .
Docker

Build a Bento to package necessary dependencies and components into a standard distribution format.

bentoml build

Containerize the Bento.

bentoml containerize summarization:latest

Run this image with Docker.

docker run --rm -p 3000:3000 summarization:latest

For detailed explanations, read Quickstart.


Community

BentoML supports billions of model runs per day and is used by thousands of organizations around the globe.

Join our Community Slack 💬, where thousands of AI application developers contribute to the project and help each other.

To report a bug or suggest a feature request, use GitHub Issues.

Contributing

There are many ways to contribute to the project:

  • Report bugs and "Thumbs up" on issues that are relevant to you.
  • Investigate issues and review other developers' pull requests.
  • Contribute code or documentation to the project by submitting a GitHub pull request.
  • Check out the Contributing Guide and Development Guide to learn more
  • Share your feedback and discuss roadmap plans in the #bentoml-contributors channel here.

Thanks to all of our amazing contributors!


Usage Reporting

BentoML collects usage data that helps our team to improve the product. Only BentoML's internal API calls are being reported. We strip out as much potentially sensitive information as possible, and we will never collect user code, model data, model names, or stack traces. Here's the code for usage tracking. You can opt-out of usage tracking by the --do-not-track CLI option:

bentoml [command] --do-not-track

Or by setting environment variable BENTOML_DO_NOT_TRACK=True:

export BENTOML_DO_NOT_TRACK=True

License

Apache License 2.0

FOSSA Status

Citation

If you use BentoML in your research, please cite using the following citation:

@software{Yang_BentoML_The_framework,
author = {Yang, Chaoyu and Sheng, Sean and Pham, Aaron and  Zhao, Shenyang and Lee, Sauyon and Jiang, Bo and Dong, Fog and Guan, Xipeng and Ming, Frost},
license = {Apache-2.0},
title = {{BentoML: The framework for building reliable, scalable and cost-efficient AI application}},
url = {https://github.com/bentoml/bentoml}
}

Project details


Release history Release notifications | RSS feed

This version

1.2.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bentoml-1.2.0.tar.gz (901.2 kB view details)

Uploaded Source

Built Distribution

bentoml-1.2.0-py3-none-any.whl (1.1 MB view details)

Uploaded Python 3

File details

Details for the file bentoml-1.2.0.tar.gz.

File metadata

  • Download URL: bentoml-1.2.0.tar.gz
  • Upload date:
  • Size: 901.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/4.0.2 CPython/3.11.7

File hashes

Hashes for bentoml-1.2.0.tar.gz
Algorithm Hash digest
SHA256 ba8ca221e7253d14b1a35ac1f4b38075e3034a66c1ca2e2692b4425cdf8e2bda
MD5 77baad0e9da91c509876d97ba8330372
BLAKE2b-256 eef8b893b5dbe758fd8a6caa556ca5c9b7df708c2c3212de3ae7455671d40349

See more details on using hashes here.

File details

Details for the file bentoml-1.2.0-py3-none-any.whl.

File metadata

  • Download URL: bentoml-1.2.0-py3-none-any.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/4.0.2 CPython/3.11.7

File hashes

Hashes for bentoml-1.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 30779d7020c3f866a5c11c57c5dc0ca6d652b54135552bf56c2839c0138ac77d
MD5 68eb4751142ff06badf0cc1cc2cf2468
BLAKE2b-256 f38c375f1f1e3efa20a06b6e79cf647489345759a819dd19534e8f45c34957ab

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page