Skip to main content

Model Server implementation for Merlin PyFunc model

Project description

PyFunc Server

PyFunc Server is Kserve extension for deploying user-defined python function model. It leverages mlflow.pyfunc model for model loading.

Usage

HTTP Server

Run following command to load sample echo-model model and start HTTP server:

PROMETHEUS_MULTIPROC_DIR=prometheus \
python -m pyfuncserver --model_dir echo-model/model

This will start http server at port 8080 which you can test using curl command

curl localhost:8080/v1/models/model-1:predict -H "Content-Type: application/json" -d '{}'

UPI V1 Server

Run following command to load sample echo-model model and start UPI v1 server:

PROMETHEUS_MULTIPROC_DIR=prometheus \
CARAML_PROTOCOL=UPI_V1 \
WORKERS=2 python -m pyfuncserver --model_dir echo-model/model

Since UPI v1 interface is gRPC then you can use grpcurl to send request

grpcurl -plaintext -d '{}'  localhost:9000 caraml.upi.v1.UniversalPredictionService/PredictValues

Development

Requirements:

  • pipenv

To install locally (including test dependency):

pipenv shell
make setup

To run all test

make test

To run benchmark

make benchmark

Configuration

Pyfunc server can be configured via following environment variables

Environment Variable Description
CARAML_PROTOCOL Protocol to be used, the valid values are HTTP_JSON and UPI_V1
CARAML_HTTP_PORT Pyfunc server will start http server listening to this port when CARAML_PROTOCOL = HTTP_JSON
CARAML_GRPC_PORT Pyfunc server will start grpc server listening to this port when CARAML_PROTOCOL = UPI_V1
CARAML_MODEL_NAME Model name
CARAML_MODEL_VERSION Model version
CARAML_MODEL_FULL_NAME Model full name in the format of ${CARAML_MODEL_NAME}-${CARAML_MODEL_FULL_NAME}
WORKERS Number of Python processes that will be created to allow multi processing (default = 1)
LOG_LEVEL Log level, valid values are INFO, ERROR, DEBUG, WARN, CRITICAL (default='INFO')
GRPC_OPTIONS GRPC options to configure UPI server as json string. The possible options can be found in grpc_types.h. Example: '{"grpc.max_concurrent_streams":100}'
GRPC_CONCURRENCY Size of grpc handler threadpool per worker (default = 10)
PUSHGATEWAY_ENABLED Enable pushing metrics to prometheus push gateway, only available when CARAML_PROTOCOL is set to UPI_V1 (default = false)
PUSHGATEWAY_URL Url of the prometheus push gateway (default = localhost:9091)
PUSHGATEWAY_PUSH_INTERVAL_SEC Interval in seconds for pushing metrics to prometheus push gateway (default = 30)

Directory Structure

├── benchmark              <- Benchmarking artifacts
├── docker                 <- Dockerfiles and environment files
    ├── Dockerfile         <- Dockerfile that will be used by kaniko to build user image in the cluster
    ├── base.Dockerfile    <- Base docker image that will be used by `Dockerfile`
├── examples               <- Examples of PyFunc models implementation
├── test                   <- Test package
├── pyfuncserver           <- Source code of this workflow
│   ├── __main__.py        <- Entry point of pyfuncserver
│   ├── config.py          <- Pyfuncserver configurations
│   ├── server.py          <- Main server that orchestrate the initialization of all server within pyfuncserver
│   ├── metrics            <- Module related to pubishing prometheus metrics
│   ├── utils              <- Utility package
│   ├── model              <- Model package
│   └── protocol           <- Protocol implementations
│       └── rest           <- Server implementation for HTTP_JSON protocol
│       └── upi            <- Server implementation for UPI_V1 protocol
├── .gitignore
├── Makefile               <- Makefile
├── README.md              <- The top-level README for developers using this project.
├── requirements.txt       <- pyfuncserver dependencies
├── setup.py               <- setup.py
├── run.sh                 <- Script to activate `merlin-model` environment and run pyfuncserver when `docker run` is invoked

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

merlin_pyfunc_server-0.45.3.tar.gz (50.1 kB view details)

Uploaded Source

Built Distribution

merlin_pyfunc_server-0.45.3-py3-none-any.whl (26.6 kB view details)

Uploaded Python 3

File details

Details for the file merlin_pyfunc_server-0.45.3.tar.gz.

File metadata

  • Download URL: merlin_pyfunc_server-0.45.3.tar.gz
  • Upload date:
  • Size: 50.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.8.18

File hashes

Hashes for merlin_pyfunc_server-0.45.3.tar.gz
Algorithm Hash digest
SHA256 db69a4b70d3abb7d20e64ebc43da7fe41d8dc70f6206d374da88dd094db527b4
MD5 1fea467785f15fb2270b7163c62cdc43
BLAKE2b-256 0aa2e178afe3a15d071c5e88990d487816deeffe30e563af68dda8291caaf53e

See more details on using hashes here.

File details

Details for the file merlin_pyfunc_server-0.45.3-py3-none-any.whl.

File metadata

File hashes

Hashes for merlin_pyfunc_server-0.45.3-py3-none-any.whl
Algorithm Hash digest
SHA256 c665a7c0e873085de8c53634fd89b10b83d0262f10940dc9423dae45503094cb
MD5 8d8a401d65b605f0b0e359622187983d
BLAKE2b-256 0efbb8f8aec17748ebda867349ed76dbcc939301afa1affbf279aea122b45085

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page