Model Serving made Efficient in the Cloud
Reason this release was yanked:
potential deadlock when too many disconnects
Project description
Model Serving made Efficient in the Cloud.
Introduction
Mosec is a high-performance and flexible model serving framework for building ML model-enabled backend and microservices. It bridges the gap between any machine learning models you just trained and the efficient online service API.
- Highly performant: web layer and task coordination built with Rust 🦀, which offers blazing speed in addition to efficient CPU utilization powered by async I/O
- Ease of use: user interface purely in Python 🐍, by which users can serve their models in an ML framework-agnostic manner using the same code as they do for offline testing
- Dynamic batching: aggregate requests from different users for batched inference and distribute results back
- Pipelined stages: spawn multiple processes for pipelined stages to handle CPU/GPU/IO mixed workloads
- Cloud friendly: designed to run in the cloud, with the model warmup, graceful shutdown, and Prometheus monitoring metrics, easily managed by Kubernetes or any container orchestration systems
- Do one thing well: focus on the online serving part, users can pay attention to the model optimization and business logic
Installation
Mosec requires Python 3.7 or above. Install the latest PyPI package with:
pip install -U mosec
Usage
Write the server
Import the libraries and set up a basic logger to better observe what happens.
from io import BytesIO
from typing import List
import torch # type: ignore
from diffusers import StableDiffusionPipeline # type: ignore
from mosec import Server, Worker, get_logger
from mosec.mixin import MsgpackMixin
logger = get_logger()
Then, we build an API to generate the images for a given prompt. To achieve that, we simply inherit the MsgpackMixin
and Worker
class, then override the forward
method. Note that the input data
is by default a JSON-decoded object, but MsgpackMixin
will override it to use msgpack for the request and response data, e.g., wishfully it receives data like [b'a cut cat playing with a red ball']
. Noted that the returned objects will also be encoded by the MsgpackMixin
.
class StableDiffusion(MsgpackMixin, Worker):
def __init__(self):
"""Init the model for inference."""
self.pipe = StableDiffusionPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16
)
self.pipe = self.pipe.to("cuda")
def forward(self, data: List[str]) -> List[memoryview]:
"""Override the forward process."""
logger.debug("generate images for %s", data)
res = self.pipe(data)
logger.debug("NSFW: %s", res[1])
images = []
for img in res[0]:
dummy_file = BytesIO()
img.save(dummy_file, format="JPEG")
images.append(dummy_file.getbuffer())
# need to return the same number of images in the same request order
# `len(data) == len(images)`
return images
Finally, we append the worker to the server to construct a single-stage workflow, and specify the number of processes we want it to run in parallel. Then run the server.
if __name__ == "__main__":
server = Server()
# by configuring the `max_batch_size` with the value >= 1, the input data in your `forward` function will be a batch
# otherwise, it's a single item
server.append_worker(StableDiffusion, num=1, max_batch_size=16)
server.run()
Run the server
After merging the snippets above into a file named server.py
, we can first have a look at the command line arguments:
python examples/stable_diffusion/server.py --help
Then let's start the server with debug logs:
python examples/stable_diffusion/server.py --debug
And in another terminal, test it:
python examples/stable_diffusion/client.py --prompt "a cut cat playing with a red ball" --output cat.jpg --port 8000
You will get an image named "cat.jpg" in the current directory.
You can check the metrics:
curl http://127.0.0.1:8000/metrics
That's it! You have just hosted your stable-diffusion model as a server! 😉
Examples
More ready-to-use examples can be found in the Example section. It includes:
- Multi-stage workflow demo: a simple CPU demo.
- Shared memory IPC
- Customized GPU allocation: deploy multiple replicas, each using different GPUs
- Customized metrics
- Jax jitted inference
- PyTorch deep learning models:
- sentiment analysis: a NLP demo.
- image recognition: a CV demo.
- stable diffusion: with msgpack serialization.
Configuration
- Dynamic batching
max_batch_size
is configured when youappend_worker
(make sure inference with the max value won't cause the out-of-memory in GPU)--wait (default=10ms)
is configured through CLI arguments (this usually should <= one batch inference duration)- If enabled, it will collect a batch either when it reaches the
max_batch_size
or thewait
time.
- Check the arguments doc.
Deployment
- This may require some shared memory, remember to set the
--shm-size
flag if you are using docker. - This service doesn't require Gunicorn or NGINX, but you can certainly use the ingress controller. BTW, it should be the PID 1 process in the container since it controls multiple processes.
- Remember to collect the metrics.
mosec_service_batch_size_bucket
shows the batch size distribution.mosec_service_process_duration_second_bucket
shows the duration for each stage (excluding the IPC time).mosec_service_remaining_task
shows the number of currently processing tasksmosec_service_throughput
shows the service throughput
- Stop the service with
SIGINT
orSIGTERM
since it has the graceful shutdown logic.
Contributing
We welcome any kind of contribution. Please give us feedback by raising issues or discussing on Discord. You could also directly contribute your code and pull request!
To start develop, you can use envd to create an isolated and clean Python & Rust environment. Check the envd-docs or build.envd for more information.
Qualitative Comparison*
Batcher | Pipeline | Parallel | I/O Format(1) | Framework(2) | Backend | Activity | |
---|---|---|---|---|---|---|---|
TF Serving | ✅ | ✅ | ✅ | Limited(a) | Heavily TF | C++ | |
Triton | ✅ | ✅ | ✅ | Limited | Multiple | C++ | |
MMS | ✅ | ❌ | ✅ | Limited | Heavily MX | Java | |
BentoML | ✅ | ❌ | ❌ | Limited(b) | Multiple | Python | |
Streamer | ✅ | ❌ | ✅ | Customizable | Agnostic | Python | |
Flask(3) | ❌ | ❌ | ❌ | Customizable | Agnostic | Python | |
Mosec | ✅ | ✅ | ✅ | Customizable | Agnostic | Rust |
*As accessed on 08 Oct 2021. By no means is this comparison showing that other frameworks are inferior, but rather it is used to illustrate the trade-off. The information is not guaranteed to be absolutely accurate. Please let us know if you find anything that may be incorrect.
(1): Data format of the service's request and response. "Limited" in the sense that the framework has pre-defined requirements on the format. (2): Supported machine learning frameworks. "Heavily" means the serving framework is designed towards a specific ML framework. Thus it is hard, if not impossible, to adapt to others. "Multiple" means the serving framework provides adaptation to several existing ML frameworks. "Agnostic" means the serving framework does not necessarily care about the ML framework. Hence it supports all ML frameworks (in Python). (3): Flask is a representative of general purpose web frameworks to host ML models.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for mosec-0.5.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1d2073d3b7a0b9f341397c631bb84f97a14dd97a1657e5892c576f438e496aea |
|
MD5 | d2c58c498ee551716dcfae99ce0e299c |
|
BLAKE2b-256 | fef8edd63e19e8f07ff451fde4fcbf5cc12ff56a71aa39971308409ac4de35f6 |
Hashes for mosec-0.5.1-pp39-pypy39_pp73-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2314683e6cdf2ebba07bad3cf648495269899b8d5b6551e7d2acf072d73de2c |
|
MD5 | bc5a234a742e02a4105802d00646c590 |
|
BLAKE2b-256 | 0ac6f51ccfcb9ad4767fc0f9a8f1a729b4f630cc397bc2884e3d2e38fbc73a60 |
Hashes for mosec-0.5.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d374fc0ef3ce652336bc300ec9753b074f57c2dbc087137cffea4f6c49752957 |
|
MD5 | 9f1b4b33ae0dd04ca5ee31e1947ebd2d |
|
BLAKE2b-256 | 557159f6599c43d96a293cdec44be0ea6c46ca0d8292b59c79cf68c118aec148 |
Hashes for mosec-0.5.1-pp38-pypy38_pp73-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1f22a8b4a5e241db06c336b749ee04e71c622fa0c1c6b59f5e8d3bfd8fb24379 |
|
MD5 | 0591a6044ff5be1c4a37363edf4471df |
|
BLAKE2b-256 | 731073e79e5665137111340014e4e0349939cc4a332f3fa3320134dbef3f95e4 |
Hashes for mosec-0.5.1-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e52a8b20179d2e137cf00b7db16a7685bbf48908825f73614d1cf21665abe91f |
|
MD5 | c34b5b507fa6061c1e2b32ae36e4ca75 |
|
BLAKE2b-256 | ba99227e9c887cc6065b073f9ba859db22c9d3960427efc2da72a67399b84d82 |
Hashes for mosec-0.5.1-pp37-pypy37_pp73-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 141d61c83583abd23c89b41769b92b9cbed441f5cfd513fb1d5ae115dc45e5cf |
|
MD5 | cf238842d2eabd33b454b421b905a4be |
|
BLAKE2b-256 | 69a29d381f008f9ff059727b84130398b4ac696452b2697a47248c0715a64c4d |
Hashes for mosec-0.5.1-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cae50a9f5ad5467e2d77a593a81897daa51892c3bd15cfce53b13b055d258caf |
|
MD5 | 6e96d8f595475fd2ec74b8d770ac3022 |
|
BLAKE2b-256 | 94d996f052a3453b06342edf1d1265b9393879c81954ea169975d7d7f9c327b8 |
Hashes for mosec-0.5.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a19d64be6e793ed370fc9f607a813536fea021cffae20601a0ade73ecee462c1 |
|
MD5 | bda5e9775cffc9a9b54a4c3764a3ab9f |
|
BLAKE2b-256 | 4e18672a751faf02825d8cf07d112743d1a6d6246f1ff084542e46d81f66005e |
Hashes for mosec-0.5.1-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec6fc889566e775adf29d2e3a39d695fd8fc1f69dc3b8e5ed03f939f6883fc1a |
|
MD5 | e7c81a61591a1562c9f3af97298c8556 |
|
BLAKE2b-256 | ce1e6cd5adb4e21de63d6e5f8487c6d2c5d180276dbf13eaf9887720c654f590 |
Hashes for mosec-0.5.1-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 82914475ad324e671b4b9b179187914662cd580f53d0cd06f1d517b66e4b9582 |
|
MD5 | 37dfdbe91cea15d1c643cbb9a071951f |
|
BLAKE2b-256 | eb0ba731791c85f828fd263acc3a9a9dbf1a17dcb327c9ff93ca9069f8f264d5 |
Hashes for mosec-0.5.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f9f24428fc1f5730f81e4157edc4532ceec17f9e65f26f9b88ac2b93b0041db |
|
MD5 | 3cb4294e034db62a5c8344eb51606eed |
|
BLAKE2b-256 | 9f4a6106318665b3c9fdd8d928076f87b7bc8567ee6637da5792242b48fe74d8 |
Hashes for mosec-0.5.1-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 774cbc9e7ead631c7a5bfaca96af5284cfdeb070fc85da7125711b536f72392e |
|
MD5 | 009144bbc07916ea1fe3ba607da10bfb |
|
BLAKE2b-256 | 600845a780bcfedd4fabe063df1ec4ebcb4f60f4c60e486531366b081b8c6cc7 |
Hashes for mosec-0.5.1-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 252faf074246410574e0198bf3ad6e7e6f3966d469d569eb7b7348c9bf5bb626 |
|
MD5 | 2400064075ed6c0360f7cca8edf12480 |
|
BLAKE2b-256 | d4fac3e52d0818bc36e0d37b87c54641dc61d8595128ebd628be4b847f645dd1 |
Hashes for mosec-0.5.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d14c5cb3d39c12946e033b5d69f82f9d54c5c87be2a655d75770164417685422 |
|
MD5 | 25483f580461d0823c5d3b509330ad92 |
|
BLAKE2b-256 | dffce997ee11c3bc5a5772c79be0ef97b911effcce4c329ac37981454f043cd6 |
Hashes for mosec-0.5.1-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 16c595bbb6e443cfc6e920a3eb6f0330993e2fe54c044446ec20b9f80692c934 |
|
MD5 | 2caf61021d9ec8289518ae6aa38ec0b1 |
|
BLAKE2b-256 | 6f56d3f56d537406414107c285860559301325b34396d6ade345a9ffab7f265f |
Hashes for mosec-0.5.1-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 59a9c216d497fccb4b0241cfc06fdb509e61b003515d6b40cd2ac1a0a984b4a5 |
|
MD5 | 8124ee384fb688a015f620f7e13bd2a5 |
|
BLAKE2b-256 | e55fc314468da94c3a914dc4facde4928414eefd3a6d20c76100343a510e26fe |
Hashes for mosec-0.5.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 65b3d55a10e9513beb1c6e8a14bb4a914e7b86ef2084751b569d225b38ae099c |
|
MD5 | 0f76241a80c2144931e10ac4eafdd552 |
|
BLAKE2b-256 | 17f2cefa945393b002978d250ddc432570f750df35d1fff699be0d72bb1872ba |
Hashes for mosec-0.5.1-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b9435d2840a469149c8e588d9aad0920b2554a9829f50f82d87d8f0328f95352 |
|
MD5 | 523c6ff0120ff5791507bcf69058c28f |
|
BLAKE2b-256 | 80f44b7ccb2754105b0c58d972440f4e39c6803ae580b92c584f8c35fcff0dff |
Hashes for mosec-0.5.1-cp37-cp37m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 90b61ebca3f0f416e0c6e4858e18d0573cef1032c99231b0b9eca93e73cdeb96 |
|
MD5 | 40296a37a439711230eede41feb8c687 |
|
BLAKE2b-256 | 1cabf1ace1a8f9adc95f0d77e9e69f22ebe6241867a19dd38045c871514b316b |
Hashes for mosec-0.5.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 47442597e4d04616c9d7fffe631afec3e8b32b7e55f11e1a2fc1ae5f5280ce0d |
|
MD5 | b61a6007e3c1351dacf914311551ffbc |
|
BLAKE2b-256 | 74cd901f54963205ca3a0bf95086a5a2920031e9939d940a3d219fafb92bd4ad |
Hashes for mosec-0.5.1-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9d63a72cd472a2a54e31719207cfe1c85e6cc35945022bbc520cfbcbc673d476 |
|
MD5 | fed8b60c297f91f64d77a6254649a7b3 |
|
BLAKE2b-256 | 7c07428fa3b05e3fd6bff5ab3c6067a92f164d3d1eb50e7969b29f374ee41b95 |