Model Serving made Efficient in the Cloud.
Introduction
Mosec is a high-performance and flexible model serving framework for building ML model-enabled backends and microservices. It bridges the gap between any machine learning models you just trained and the efficient online service API.
- Highly performant: web layer and task coordination built with Rust 🦀, which offers blazing speed in addition to efficient CPU utilization powered by async I/O
- Ease of use: user interface purely in Python 🐍, by which users can serve their models in an ML framework-agnostic manner using the same code as they do for offline testing
- Dynamic batching: aggregate requests from different users for batched inference and distribute results back
- Pipelined stages: spawn multiple processes for pipelined stages to handle CPU/GPU/IO mixed workloads
Installation
Mosec requires Python 3.6 or above. Install the latest PyPI package with:
pip install -U mosec
Usage
Write the server
Import the libraries and setup a basic logger to better observe what happens:
import logging
from pydantic import BaseModel # we need this to define our input/output schemas
from mosec import Server, Worker
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter(
"%(asctime)s - %(process)d - %(levelname)s - %(filename)s:%(lineno)s - %(message)s"
)
sh = logging.StreamHandler()
sh.setFormatter(formatter)
logger.addHandler(sh)
Define our service schemas for both input and output. These schemas will help us for data validation:
class Request(BaseModel):
x: float
class Response(BaseModel):
y: float
Now, we are going to build an API to calculate the exponential with base e for a given number. To achieve that, we simply inherit the Worker class and override the forward method:
import math
class CalculateExp(Worker):
def forward(self, req: Request):
y = math.exp(req.x) # f(x) = e ^ x
logger.debug(f"e ^ {req.x} = {y}")
return Response(y=y)
Finally, we run the server when the file is executed:
if __name__ == "__main__":
server = Server(Request, Response)
server.append_worker(
CalculateExp, num=2
) # we spawn two processes for our calculator
server.run()
Run the server
After merging the snippets above into a file named server.py, we can first have a look at the supported arguments:
python server.py --help
Then let's start the server...
python server.py
and test it:
curl -X POST http://127.0.0.1:8000/inference -d '{"x": 2}'
That's it! You have just hosted your exponential-computing model as a server! 😉
Example
More ready-to-use examples can be found in the Example section.
Contributing
We welcome any kind of contributions. Please give us feedback by raising issues or directly contribute your code and pull request!
Release files for mosec 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| mosec-0.1.0.tar.gz | 17.5 kB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| mosec-0.1.0-cp39-cp39-manylinux1_x86_64.whl | CPython 3.9 | CPython 3.9 | Linux glibc 2.5+ x86-64 | Details |
| mosec-0.1.0-cp39-cp39-macosx_10_9_x86_64.whl | CPython 3.9 | CPython 3.9 | macOS 10.9+ x86-64 | Details |
| mosec-0.1.0-cp38-cp38-manylinux1_x86_64.whl | CPython 3.8 | CPython 3.8 | Linux glibc 2.5+ x86-64 | Details |
| mosec-0.1.0-cp38-cp38-macosx_10_9_x86_64.whl | CPython 3.8 | CPython 3.8 | macOS 10.9+ x86-64 | Details |
| mosec-0.1.0-cp37-cp37m-manylinux1_x86_64.whl | CPython 3.7 | CPython 3.7 pymalloc | Linux glibc 2.5+ x86-64 | Details |
| mosec-0.1.0-cp37-cp37m-macosx_10_9_x86_64.whl | CPython 3.7 | CPython 3.7 pymalloc | macOS 10.9+ x86-64 | Details |
| mosec-0.1.0-cp36-cp36m-manylinux1_x86_64.whl | CPython 3.6 | CPython 3.6 pymalloc | Linux glibc 2.5+ x86-64 | Details |
| mosec-0.1.0-cp36-cp36m-macosx_10_9_x86_64.whl | CPython 3.6 | CPython 3.6 pymalloc | macOS 10.9+ x86-64 | Details |
Total release size: 22.5 MB
Release files / mosec-0.1.0.tar.gz
| Download URL | mosec-0.1.0.tar.gz |
|---|---|
| Size | 17.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
023d1883e45e77557f169c6cacd17aea3f771ce6e83a7d7dee7fac79d4f9f22e
|
|
BLAKE2b-256 checksum How to use checksums |
8b1601910673e527ad0eea836d5ec3c433356592ccdb86bdbf09ef181f6e113e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7
|
Release files / mosec-0.1.0-cp39-cp39-manylinux1_x86_64.whl
| Download URL | mosec-0.1.0-cp39-cp39-manylinux1_x86_64.whl |
|---|---|
| Size | 3.3 MB |
| Tags | CPython 3.9 Linux glibc 2.5+ x86-64 |
|
SHA-256 checksum How to use checksums |
96f2da24c2306d2e82f14fc635065547e94c27f8c8714f2691968907b2a8a0aa
|
|
BLAKE2b-256 checksum How to use checksums |
33374ac0abad73c337cb67349b4df5c98c987f0fe6f8294644efbb7f4dd76cb8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7
|
Release files / mosec-0.1.0-cp39-cp39-macosx_10_9_x86_64.whl
| Download URL | mosec-0.1.0-cp39-cp39-macosx_10_9_x86_64.whl |
|---|---|
| Size | 2.3 MB |
| Tags | CPython 3.9 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
24fd91e3d45a28caf1682bd83465cd64a67acd2ff6062fbb6b6415bc3d815294
|
|
BLAKE2b-256 checksum How to use checksums |
b05a4ba4b36943fe584b0de27d128b784bf3008a31ce7dbc9d8d2686e61ee747
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7
|
Release files / mosec-0.1.0-cp38-cp38-manylinux1_x86_64.whl
| Download URL | mosec-0.1.0-cp38-cp38-manylinux1_x86_64.whl |
|---|---|
| Size | 3.3 MB |
| Tags | CPython 3.8 Linux glibc 2.5+ x86-64 |
|
SHA-256 checksum How to use checksums |
1daac1e326d88992b997b57f1cf103a7aefc825ddb9fd0bd633e8c13dd57c2c2
|
|
BLAKE2b-256 checksum How to use checksums |
b3f88612c2e9b4414fd4a1246981dc3e874142a753f318f048ab2bc20b0640cf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7
|
Release files / mosec-0.1.0-cp38-cp38-macosx_10_9_x86_64.whl
| Download URL | mosec-0.1.0-cp38-cp38-macosx_10_9_x86_64.whl |
|---|---|
| Size | 2.3 MB |
| Tags | CPython 3.8 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
a3c18706b5e49a540f8cb9bc2252fada6c63762503e0fd42daf56a0026e94954
|
|
BLAKE2b-256 checksum How to use checksums |
4520c5a78d7b38afdeeea460080aebf4d12184bc0c8fcaab2653a85560dc01e4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7
|
Release files / mosec-0.1.0-cp37-cp37m-manylinux1_x86_64.whl
| Download URL | mosec-0.1.0-cp37-cp37m-manylinux1_x86_64.whl |
|---|---|
| Size | 3.3 MB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.5+ x86-64 |
|
SHA-256 checksum How to use checksums |
fdcb70d729e0083d3d93256458d0e310d3639822bc48a1ed498327d68b4e7931
|
|
BLAKE2b-256 checksum How to use checksums |
27f0c0e4d51638f82b89999134e56bacbf0afc870388db3ead668523b8fac09d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7
|
Release files / mosec-0.1.0-cp37-cp37m-macosx_10_9_x86_64.whl
| Download URL | mosec-0.1.0-cp37-cp37m-macosx_10_9_x86_64.whl |
|---|---|
| Size | 2.3 MB |
| Tags | CPython 3.7 CPython 3.7 pymalloc macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
fda82afdc6aba8c4c0efa04e70128bdfcc724f254388efa6c37c21d14978abe3
|
|
BLAKE2b-256 checksum How to use checksums |
8077b2cd0e37ff28adda696dcb28c8867bd6672e1d5f61a705e9f36e2023b4a8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7
|
Release files / mosec-0.1.0-cp36-cp36m-manylinux1_x86_64.whl
| Download URL | mosec-0.1.0-cp36-cp36m-manylinux1_x86_64.whl |
|---|---|
| Size | 3.3 MB |
| Tags | CPython 3.6 CPython 3.6 pymalloc Linux glibc 2.5+ x86-64 |
|
SHA-256 checksum How to use checksums |
452cb9ae2615cef3ccfbcf346a635496ac45ff8747a5be3c661a6d95576fb5eb
|
|
BLAKE2b-256 checksum How to use checksums |
3c2ef2a07c7559f13b230d56608c2fb52413d24c2ff251a900123ac4738ef4cf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7
|
Release files / mosec-0.1.0-cp36-cp36m-macosx_10_9_x86_64.whl
| Download URL | mosec-0.1.0-cp36-cp36m-macosx_10_9_x86_64.whl |
|---|---|
| Size | 2.3 MB |
| Tags | CPython 3.6 CPython 3.6 pymalloc macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
24df708b7b27ff77b09e0afd33a616095bbe6d3e6a5bd1ad9e1f2e624bee47d8
|
|
BLAKE2b-256 checksum How to use checksums |
8f50f84ba283a7dea89dc15d41cf77529cb0a706b156f762d22c3bafc47bf193
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7
|
