fastDeploy
easy and performant micro-services for Python Deep Learning inference pipelines
- Deploy any python inference pipeline with minimal extra code
- Auto batching of concurrent inputs is enabled out of the box
- no changes to inference code (unlike tf-serving etc), entire pipeline is run as is
- Promethues metrics (open metrics) are exposed for monitoring
- Auto generates clean dockerfiles and kubernetes health check, scaling friendly APIs
- sequentially chained inference pipelines are supported out of the box
- can be queried from any language via easy to use rest apis
- easy to understand (simple consumer producer arch) and simple code base
Installation:
pip install --upgrade fastdeploy fdclient
# fdclient is optional, only needed if you want to use python client
CLI explained
Start fastDeploy server on a recipe:
# Invoke fastdeploy
python -m fastdeploy --help
# or
fastdeploy --help
# Start prediction "loop" for recipe "echo"
fastdeploy --loop --recipe recipes/echo
# Start rest apis for recipe "echo"
fastdeploy --rest --recipe recipes/echo
Send a request and get predictions:
auto generate dockerfile and build docker image:
# Write the dockerfile for recipe "echo"
# and builds the docker image if docker is installed
# base defaults to python:3.8-slim
fastdeploy --build --recipe recipes/echo
# Run docker image
docker run -it -p8080:8080 fastdeploy_echo
Serving your model (recipe):
Where to use fastDeploy?
- to deploy any non ultra light weight models i.e: most DL models, >50ms inference time per example
- if the model/pipeline benefits from batch inference, fastDeploy is perfect for your use-case
- if you are going to have individual inputs (example, user's search input which needs to be vectorized or image to be classified)
- in the case of individual inputs, requests coming in at close intervals will be batched together and sent to the model as a batch
- perfect for creating internal micro services separating your model, pre and post processing from business logic
- since prediction loop and inference endpoints are separated and are connected via sqlite backed queue, can be scaled independently
Where not to use fastDeploy?
- non cpu/gpu heavy models that are better of running parallely rather than in batch
- if your predictor calls some external API or uploads to s3 etc in a blocking way
- io heavy non batching use cases (eg: query ES or db for each input)
- for these cases better to directly do from rest api code (instead of consumer producer mechanism) so that high concurrency can be achieved
1 maintainer
Release files for fastdeploy 3.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| fastdeploy-3.1.1.tar.gz | 16.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| fastdeploy-3.1.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 33.6 kB
Release files / fastdeploy-3.1.1.tar.gz
| Download URL | fastdeploy-3.1.1.tar.gz |
|---|---|
| Size | 16.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d080338d0806b8176df0a8779f6c0fbb4f1fa6379cb527eca2457d5608e7f329
|
|
BLAKE2b-256 checksum How to use checksums |
f08735d833b9c938f29ab873714e4a744c83545a6a27725bfb7e2b63e13c8a9e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/5.1.1 CPython/3.12.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2024.
Transparency logRelease files / fastdeploy-3.1.1-py3-none-any.whl
| Download URL | fastdeploy-3.1.1-py3-none-any.whl |
|---|---|
| Size | 16.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4c209874f6884c74d3d33d3b9f76b54ef8641b1bb8f0384eab782c3f6b36180b
|
|
BLAKE2b-256 checksum How to use checksums |
84f1a8bf61b9dd0b58a8d0a73e42e73c4fe7e77206757dca7035018003f8bf99
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/5.1.1 CPython/3.12.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2024.
Transparency log
