Model Server implementation for Merlin PyFunc model
Project description
PyFunc Server
PyFunc Server is Kserve extension for deploying user-defined python function model. It leverages mlflow.pyfunc model for model loading.
Usage
HTTP Server
Run following command to load sample echo-model
model and start HTTP server:
PROMETHEUS_MULTIPROC_DIR=prometheus \
python -m pyfuncserver --model_dir echo-model/model
This will start http server at port 8080 which you can test using curl command
curl localhost:8080/v1/models/model-1:predict -H "Content-Type: application/json" -d '{}'
UPI V1 Server
Run following command to load sample echo-model
model and start UPI v1 server:
PROMETHEUS_MULTIPROC_DIR=prometheus \
CARAML_PROTOCOL=UPI_V1 \
WORKERS=2 python -m pyfuncserver --model_dir echo-model/model
Since UPI v1 interface is gRPC then you can use grpcurl to send request
grpcurl -plaintext -d '{}' localhost:9000 caraml.upi.v1.UniversalPredictionService/PredictValues
Development
Requirements:
- pipenv
To install locally (including test dependency):
pipenv shell
make setup
To run all test
make test
To run benchmark
make benchmark
Configuration
Pyfunc server can be configured via following environment variables
Environment Variable | Description |
---|---|
CARAML_PROTOCOL | Protocol to be used, the valid values are HTTP_JSON and UPI_V1 |
CARAML_HTTP_PORT | Pyfunc server will start http server listening to this port when CARAML_PROTOCOL = HTTP_JSON |
CARAML_GRPC_PORT | Pyfunc server will start grpc server listening to this port when CARAML_PROTOCOL = UPI_V1 |
CARAML_MODEL_NAME | Model name |
CARAML_MODEL_VERSION | Model version |
CARAML_MODEL_FULL_NAME | Model full name in the format of ${CARAML_MODEL_NAME}-${CARAML_MODEL_FULL_NAME} |
WORKERS | Number of Python processes that will be created to allow multi processing (default = 1) |
LOG_LEVEL | Log level, valid values are INFO , ERROR , DEBUG , WARN , CRITICAL (default='INFO') |
GRPC_OPTIONS | GRPC options to configure UPI server as json string. The possible options can be found in grpc_types.h. Example: '{"grpc.max_concurrent_streams":100}' |
GRPC_CONCURRENCY | Size of grpc handler threadpool per worker (default = 10) |
PUSHGATEWAY_ENABLED | Enable pushing metrics to prometheus push gateway, only available when CARAML_PROTOCOL is set to UPI_V1 (default = false) |
PUSHGATEWAY_URL | Url of the prometheus push gateway (default = localhost:9091) |
PUSHGATEWAY_PUSH_INTERVAL_SEC | Interval in seconds for pushing metrics to prometheus push gateway (default = 30) |
Directory Structure
├── benchmark <- Benchmarking artifacts
├── docker <- Dockerfiles and environment files
├── Dockerfile <- Dockerfile that will be used by kaniko to build user image in the cluster
├── base.Dockerfile <- Base docker image that will be used by `Dockerfile`
├── examples <- Examples of PyFunc models implementation
├── test <- Test package
├── pyfuncserver <- Source code of this workflow
│ ├── __main__.py <- Entry point of pyfuncserver
│ ├── config.py <- Pyfuncserver configurations
│ ├── server.py <- Main server that orchestrate the initialization of all server within pyfuncserver
│ ├── metrics <- Module related to pubishing prometheus metrics
│ ├── utils <- Utility package
│ ├── model <- Model package
│ └── protocol <- Protocol implementations
│ └── rest <- Server implementation for HTTP_JSON protocol
│ └── upi <- Server implementation for UPI_V1 protocol
├── .gitignore
├── Makefile <- Makefile
├── README.md <- The top-level README for developers using this project.
├── requirements.txt <- pyfuncserver dependencies
├── setup.py <- setup.py
├── run.sh <- Script to activate `merlin-model` environment and run pyfuncserver when `docker run` is invoked
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file merlin_pyfunc_server-0.45.3.tar.gz
.
File metadata
- Download URL: merlin_pyfunc_server-0.45.3.tar.gz
- Upload date:
- Size: 50.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.8.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | db69a4b70d3abb7d20e64ebc43da7fe41d8dc70f6206d374da88dd094db527b4 |
|
MD5 | 1fea467785f15fb2270b7163c62cdc43 |
|
BLAKE2b-256 | 0aa2e178afe3a15d071c5e88990d487816deeffe30e563af68dda8291caaf53e |
File details
Details for the file merlin_pyfunc_server-0.45.3-py3-none-any.whl
.
File metadata
- Download URL: merlin_pyfunc_server-0.45.3-py3-none-any.whl
- Upload date:
- Size: 26.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.8.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c665a7c0e873085de8c53634fd89b10b83d0262f10940dc9423dae45503094cb |
|
MD5 | 8d8a401d65b605f0b0e359622187983d |
|
BLAKE2b-256 | 0efbb8f8aec17748ebda867349ed76dbcc939301afa1affbf279aea122b45085 |