KServe Python SDK
Python SDK for KServe Server and Client.
Installation
KServe Python SDK can be installed by pip or uv.
pip install
pip install kserve
To install Kserve with storage support
pip install kserve[storage]
UV
Install via uv.
make dev_install
To install Kserve with storage support
uv install --extra storage
KServe Python Server
KServe's python server libraries implement a standardized library that is extended by model serving frameworks such as Scikit Learn, XGBoost and PyTorch. It encapsulates data plane API definitions and storage retrieval for models.
It provides many functionalities, including among others:
- Registering a model and starting the server
- Prediction Handler
- Pre/Post Processing Handler
- Liveness Handler
- Readiness Handlers
It supports the following storage providers:
- Google Cloud Storage with a prefix: "gs://"
- By default, it uses
GOOGLE_APPLICATION_CREDENTIALSenvironment variable for user authentication. - If
GOOGLE_APPLICATION_CREDENTIALSis not provided, anonymous client will be used to download the artifacts.
- By default, it uses
- S3 Compatible Object Storage with a prefix "s3://"
- By default, it uses
S3_ENDPOINT,AWS_ACCESS_KEY_ID, andAWS_SECRET_ACCESS_KEYenvironment variables for user authentication.
- By default, it uses
- Azure Blob Storage with the format: "https://{$STORAGE_ACCOUNT_NAME}.blob.core.windows.net/{$CONTAINER}/{$PATH}"
- By default, it uses anonymous client to download the artifacts.
- For e.g. https://kfserving.blob.core.windows.net/triton/simple_string/
- Local filesystem either without any prefix or with a prefix "file://". For example:
- Absolute path:
/absolute/pathorfile:///absolute/path - Relative path:
relative/pathorfile://relative/path - For local filesystem, we recommended to use relative path without any prefix.
- Absolute path:
- Persistent Volume Claim (PVC) with the format "pvc://{$pvcname}/[path]".
- The
pvcnameis the name of the PVC that contains the model. - The
[path]is the relative path to the model on the PVC. - For e.g.
pvc://mypvcname/model/path/on/pvc
- The
- Generic URI, over either
HTTP, prefixed withhttp://orHTTPS, prefixed withhttps://. For example:https://<some_url>.com/model.joblibhttp://<some_url>.com/model.joblib
Metrics
For latency metrics, send a request to /metrics. Prometheus latency histograms are emitted for each of the steps (pre/postprocessing, explain, predict).
Additionally, the latencies of each step are logged per request.
| Metric Name | Description | Type |
|---|---|---|
| request_preprocess_seconds | pre-processing request latency | Histogram |
| request_explain_seconds | explain request latency | Histogram |
| request_predict_seconds | prediction request latency | Histogram |
| request_postprocess_seconds | pre-processing request latency | Histogram |
KServe Client
Getting Started
KServe's python client interacts with KServe control plane APIs for executing operations on a remote KServe cluster, such as creating, patching and deleting of a InferenceService instance. See the Sample for Python SDK Client to get started.
Documentation for Client API
Please review KServe Client API docs.
Documentation For Models
- KnativeAddressable
- KnativeCondition
- KnativeURL
- KnativeVolatileTime
- NetUrlUserinfo
- V1alpha1InferenceGraph
- V1alpha1InferenceGraphList
- V1alpha1InferenceGraphSpec
- V1alpha1InferenceGraphStatus
- V1alpha1InferenceRouter
- V1alpha1InferenceStep
- V1alpha1InferenceTarget
- V1beta1AlibiExplainerSpec
- V1beta1Batcher
- V1beta1ComponentExtensionSpec
- V1beta1ComponentStatusSpec
- V1beta1CustomExplainer
- V1beta1CustomPredictor
- V1beta1CustomTransformer
- V1beta1ExplainerConfig
- V1beta1ExplainerSpec
- V1beta1ExplainersConfig
- V1beta1InferenceService
- V1beta1InferenceServiceList
- V1beta1InferenceServiceSpec
- V1beta1InferenceServiceStatus
- V1beta1InferenceServicesConfig
- V1beta1IngressConfig
- V1beta1LoggerSpec
- V1beta1ModelSpec
- V1beta1ONNXRuntimeSpec
- V1beta1PodSpec
- V1beta1PredictorConfig
- V1beta1PredictorExtensionSpec
- V1beta1PredictorSpec
- V1beta1PredictorsConfig
- V1beta1SKLearnSpec
- V1beta1TFServingSpec
- V1beta1TorchServeSpec
- V1beta1TrainedModel
- V1beta1TrainedModelList
- V1beta1TrainedModelSpec
- V1beta1TrainedModelStatus
- V1beta1TransformerConfig
- V1beta1TransformerSpec
- V1beta1TransformersConfig
- V1beta1TritonSpec
- V1beta1XGBoostSpec
2 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file kserve-0.20.0.tar.gz.
File metadata
- Download URL: kserve-0.20.0.tar.gz
- Upload date:
- Size: 407.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.7.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2a23f7ef06a540ce132d7a52114d4995fa597189c9817869264906b704153389
|
|
| MD5 |
c439594a5f07a884274cbabb34ed5e31
|
|
| BLAKE2b-256 |
edb8d4508f9d5b8ab7fbd15a20d5e15e27c5033e2e6d95ec1fbb8f4c10847f7d
|
File details
Details for the file kserve-0.20.0-py3-none-any.whl.
File metadata
- Download URL: kserve-0.20.0-py3-none-any.whl
- Upload date:
- Size: 559.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.7.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
6762eae3ae633f69178e08f531052722ea34013ab611050e0263679c7c9ca8c7
|
|
| MD5 |
43da8ade6515373f6a0a1660ed28b8cc
|
|
| BLAKE2b-256 |
bebb8cb0b5d14eff6f4bb5967e73819e9dd400530893d7003681d2c8c033bcb5
|
