MLOps framework
Project description
MLOps Framework
This is a framework for MLOps. Currently it deploys models as CDF Functions.
Getting Started:
Follow these steps:
- Install package:
pip install akerbp.mlops
- Define
ENV
andCOGNITE_*
environmental variables as described in https://akerbp.atlassian.net/wiki/spaces/SIMDev/pages/1181384729/MLOps - Become familiar with the model template
- Copy pipeline file
bitbucket-pipelines.yml
and config filemlops_settings.py
to your repo - Fill in user settings.
- Model artifacts should not be committed to the repo.
- Follow the file and folder structure (described later)
- It's possible to have several models per repo: they need to be registered in the settings, and then they need to have their own model and test files.
- Follow the import guidelines (described later)
- Make sure the prediction service gets access to model artifacts (described later)
- Add the new files to your repository, commit and push
- Follow or request the Bitbucket setup (described later)
A this point every git push in master branch will trigger a deployment in the test environment. More information about the deployments pipelines is provided later.
User Guide
MLOps Files and Folders
These are the files and folders from the MLOps framework:
mlops_settings.py
contains the user settings- Folder
model_code
is a model template included to show the model interface. It is not needed by the framework, but it is recommended to become familiar with it. model_artifact
stores the artifacts for the model shown inmodel_code
. This is to help to test the model and learn the framework.mlops
contains deployment codebitbucket-pipelines.yml
describes the deployment pipeline in Bitbucket
Import Guidelines
The repo's root folder is the base folder when importing. For example, assume
you have these files in the model code folder: model_code/model.py
,
model_code/helper.py
and model_code/data.csv
. If model.py
needs to import
helper.py
, use: import model_code.helper
. If model.py
needs to read
data.csv
, the right path is os.path.join('model_code', 'data.csv')
.
Mlops library can of course be imported as well, e.g. its logger:
from akerbp.mlops.utils import logger
logging=logger.get_logger()
logging.debug("This is a debug log")
Files and Folders Structure
All the model code and files should be under a single folder. Required files:
model.py
: implements our standard model interfacetest_model.py
: tests to verify that the model code is correct and to verify correct deploymentrequirements.model
: libraries needed (with specific version numbers), can't be calledrequirements.txt
.
Projects with multiple models: models can be in different folders, but if there are common files this structure should be followed (when deploying a model the top folder of the path is chosen as parent model code folder):
models/model1/
models/model2/
models/common_code/
Services
We consider two types of services: prediction and training. Prediction services
are deployed with model artifacts so that they are available at prediction time
(downloading would require waiting time, and files written during run time
consume ram memory). Services output has a status field ('ok' or 'error'). If
they are 'ok', they have also a 'prediction'/'training' field. The former is
determined by the predict
method of the model, while the latter combines
artifact metadata and model metadata produced by the train
function.
Prediction service has also a 'model_id' field to keep track of which model was
used to predict.
Model Artifacts for the Prediction Service
Deployment of a prediction service requires a model artifact folder. Model artifacts are segregated by environment (e.g. only production models can be deployed to production). Model artifacts are versioned and stored in CDF Files together with user-defined metadata. Uploading a new model increases the version count by 1 for that model and environment. It's important not to delete model files manually, since that would mess with the model manager. When deploying a model service, the latest model version is chosen (however, we can discuss the possibility of deploying specific versions or filtering by metadata).
The general rule is that model artifacts have to be uploaded manually before deployment. If there are multiple models, you need to do this one at at time. Code example:
from akerbp.mlops.cdf.helpers import set_up_cdf_client
from akerbp.mlops.cdf.helpers import upload_new_model_version
set_up_cdf_client()
metadata = train(model_dir, secrets) # or define it directly
folder_info = upload_new_model_version(
model_name,
env,
folder_path,
metadata
)
Note that model_name
corresponds to one of the elements in model names
defined in mlops_settings.py
, env
is the target environment (where the model
should be available), folder_path
is the local model artifact folder and
metadata
is a dictionary with artifact metadata, e.g. performance, git commit,
etc. Each model update adds a new version (environment dependent) and note that
updating a model doesn't modify the models used in existing prediction services.
Recommended process to update a model:
- New model features implemented in a feature branch
- New artifact generated and uploaded to test environment
- Feature branch merged with master
- Test deployment is triggered automatically: prediction service is deployed to test with the latest artifacts
- Prediction service in test is verified, and if things go well
- New artifact uploaded to prod environment
- Production deployment is triggered manually: prediction service is deployed to prod with the latest artifacts
However, in projects with a training service, you can rely on it to upload a first version of the model. The first prediction service deployment will fail, but you can deploy again after the training service has produced a model.
Another exception is that, when you deploy from the development environment
(covered later in this document), the model artifacts in the settings file can
point to existing local folders. These will then be used for the deployment.
Version is then fixed to model_name/dev/1
. Note that these artifacts are not
uploaded to CDF Files.
Local Testing and Deployment
It's possible to tests the functions locally, which can help you debug errors quickly. This is recommended before a deployment. From your repo's root folder:
python -m pytest model_code
(replacemodel_code
by your model code folder name)bash deploy_prediction_service.sh
bash deploy_training_service.sh
(if there's a training service)
The first one will run your model tests. The last two run model tests but also the service tests implemented in the framework and simulate deployment.
If you really want to deploy from your development environment, you can run
this: LOCAL_DEPLOYMENT=True bash deploy_prediction_service.sh
Automated Deployments from Bitbucket
Deployments to the test environment are triggered by commits (you need to push them). Deployments to the production environment are enabled manually from the Bitbucket pipeline dashboard. Branches that match 'deploy/*' behave as master.
It is assumed that most projects won't include a training service. A branch that matches 'trainpred/*' deploys both prediction and training services. If a project includes both services, the pipeline file could instead be edited so that master deployed both services.
It is possible to schedule the training service in CDF, and then it can make sense to schedule the deployment pipeline of the model service (as often as new models are trained)
Bitbucket Setup
The following environments need to be defined in repository settings > deployments:
- test deployments: test-prediction and test-training, each with ENV=test
- production deployments: production-prediction and production-training, each with ENV=prod
The following need to be defined in respository settings > repository variables: COGNITE_API_KEY_DATA, COGNITE_API_KEY_FUNCTIONS, COGNITE_API_KEY_FILES
The pipeline needs to be enabled.
Developer Guide
Build and Upload Package
Edit setup.py
file and note the following:
- Register dependencies
- Bash scripts will be installed in a
bin
folder in the PATH. Create an account in pypi, then create a token and a$HOME/.pypirc
file. It's possible to build and upload the library from the development environment:
bash build.sh
However, there's no need to do that since the pipeline is setup to run that script before the service steps.
The library can be installed locally in developer mode (installed package links to the source code, so that it can be modified without the need to reinstall). From the package folder:
pip install -e .
TODO
Decisions/Findings:
- can we install custom libraries in CDF? -> yes, if we upload them to pypi
- Generate function-specific tests -> data scientist
- Can we define environmental variables in CDF functions? -> not currently, but we can use a settings file
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file akerbp.mlops-0.20210129105542.tar.gz
.
File metadata
- Download URL: akerbp.mlops-0.20210129105542.tar.gz
- Upload date:
- Size: 20.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/52.0.0 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | db5570ddd44a5897b37e64cd460c463836fc26eca9db9f32a3a07c5b22be329f |
|
MD5 | 51b5d946544140bad709750b92021d08 |
|
BLAKE2b-256 | fd4f976692a4ce3a88880cbe00a3af363ff176571e84325e0f0a89168fdae882 |
File details
Details for the file akerbp.mlops-0.20210129105542-py3-none-any.whl
.
File metadata
- Download URL: akerbp.mlops-0.20210129105542-py3-none-any.whl
- Upload date:
- Size: 23.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/52.0.0 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c2c88a44e929414e104a38ba8d46718d93f9fbf59493a0c7f80ccf6a3446515 |
|
MD5 | 4e8adb7835f1a3edf5b1d5d5165dd5a4 |
|
BLAKE2b-256 | afb11b58c806463aa7201b74c1788d5d750f9f7dc6225d111c24a10a8d7759b5 |