Use fastapi for model inference over http
Project description
model-inference-fastapi
Running inference on ML model over http using fastapi
Installation
pip install model-inference-fastapi
Usage
-
Create a model service file
model_service.py
implementing theModelServiceBase
interface like so.from typing import Dict, List, Optional from model_inference_fastapi.model_service_base import ModelServiceBase class ModelService(ModelServiceBase): __model = None @staticmethod def validate(data: Optional[List[Dict]] = None, image_files: Optional[List[bytes]] = None) -> Optional[str]: return @staticmethod def predict(data: Optional[List[Dict]] = None, image_files: Optional[List[bytes]] = None) -> List[Dict]: return [{"status": "Ready to Go!!!"}]
For instance if you are trying to deploy an image classification model ( e.g. ResNet50) using torch. You will need the following
model_service.py
example in pytorch.import json from typing import Dict, List, Optional import torch from torchvision.models import resnet50 from torchvision.transforms import Resize, Compose, CenterCrop, ToTensor, Normalize from model_inference_fastapi.model_service_base import ModelServiceBase from model_inference_fastapi.utils import convert_file_to_pil_image # derived from https://keras.io/api/applications/ class ModelService(ModelServiceBase): __model = resnet50(weights="IMAGENET1K_V2") __transform = Compose([ Resize(size=256), CenterCrop(size=224), ToTensor(), Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) ]) with open("imagenet1000_clsidx_to_labels.txt") as f: __class_names = eval(f.read()) @staticmethod def validate(*, data: Optional[List[Dict]] = None, image_files: Optional[List[bytes]] = None): if len(image_files) != 1: return "There must be at least one image file in image_files" @staticmethod def predict(*, data: Optional[List[Dict]] = None, image_files: Optional[List[bytes]] = None) -> List[Dict]: test_image_tensor = ModelService.__transform(convert_file_to_pil_image(image_files[0])) test_image_tensor = test_image_tensor.view(1, 3, 224, 224) with torch.no_grad(): ModelService.__model.eval() # Model outputs log probabilities out = ModelService.__model(test_image_tensor) ps = torch.exp(out) top_k, top_class = ps.topk(1, dim=1) print("Output class : ", ModelService.__class_names[top_class.cpu().numpy()[0][0]]) return [{"class": ModelService.__class_names[top_class.cpu().numpy()[0][0]]}]
-
Run Server
uvicorn model_inference_fastapi.main:app
-
Visit the API testing page powered by FastAPI here http://127.0.0.1:8000/docs/.
- Navigate to the
/predict/
AKA Prediction Route dropdown - Click the Try it out button
- For the
image_files
array click Add string item button - Choose a file to upload
- Click execute
- Scroll down to see the server response
- Navigate to the
Testing model_service.py
Here is a sample test file for image classification model_service.py
# replace this import with import for you own `model_service.py`
from model_inference_fastapi.model_service import ModelService
def test_predict():
with open('/path/to/image.jpg', 'rb') as file:
prediction = ModelService.predict(image_files=[file.read()])
assert prediction[0]["class"] == "dog"
Using the docker image
See example dir for more info.
- create a
Dockerfile
like so
FROM robotstech/model-inference-fastapi
EXPOSE 8000
WORKDIR /app
COPY requirements.txt ./requirements.txt
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
# move model_service.py
COPY ./model_service.py ./model_inference_fastapi/
# move data or other resources to root dir of model-inference
COPY ./imagenet1000_clsidx_to_labels.txt .
The key info here is to move the model_service.py
and data/resource to the right directory
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for model_inference_fastapi-0.0.1a0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14ae01ecb8e384d1090ff2d50531f75da88d6ed195178ed9ac22e9b2d7b458ea |
|
MD5 | 15e59fe6c3096624933ec3023e7e4633 |
|
BLAKE2b-256 | b40572307b1e6a1a20a083006376ab48252128e3ef04ec546832011dc5d51e68 |
Hashes for model_inference_fastapi-0.0.1a0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 72481c2beb7166c17290e6dfdca26c09b67237e9cc22228d2ce48dd0ab8a59eb |
|
MD5 | 1a081543421e9e085af7afdaf8ad3c0c |
|
BLAKE2b-256 | c46baedb446468f3939b8d06ccd18f68db0c6061b2bbc1c1a9512b5a30727960 |