Utilities of MLOps for INRIA
Project description
common-mlops
This is a set of utilities for end-to-end lifecycle of Machine Learning applications in Python for Inria-Chile. Could be used like a framework and tooling for rapid development of data science and MLOps functions.
Here’s the folder structure for the repository:
common-mlops/
|
├── data/
│ ├── prepared/
│ └── raw/
│ └── metrics/
├── model/
├── mlops/
| └── *.py
├── test/
| └── test_*.py
├── venv/
├── check_credentials.sh
├── install.sh
├── LICENSE
├── README.md
├── requirements.txt
├── s3*.py
└── setup.py
There are the folloging folders in our repository:
- data/ is all versions of the dataset.
- data/raw/ is the data obtained from an external source.
- data/prepared/ is for data modified internally.
- data/metrics/ is for tracking the performance metrics of our models.
- model/ is for machine learning models.
- mlops/ is the source code. Python classes used by the Python scripts.
- test/ is the pyunit of source code.
- venv/ is the virtual environment.
- check_credentials.sh bash for check the AWS credentials.
- install.sh bash for check the installation.
- LICENSE
- requirements.txt the dependencies.
- s3*.py Python scripts for bash executions.
- setup.py the configuration of the package.
Installation Linux
Perform the following steps in order to install this program. Replace 'access_key_id' and 'secret_access_key' by the account AWS Credentials associated to poc-inriacl.
git clone https://gitlab.com/Inria-Chile/common-mlops.git
./common-mlops/install.sh access_key_id secret_access_key
pip3 uninstall common-mlops
# Version of test
pip3 install -i https://test.pypi.org/simple/ common-mlops
# Version of production
pip3 install common-mlops
Updates in test.pypi.org
Perform the following steps in order to publish new versions to test.pypi.org:
sudo rm -rf build common_mlops.egg-info dist
bumpversion --current-version 1.0.0 minor setup.py mlops/__init__.py
python3 setup.py sdist bdist_wheel
# Version of test
twine check dist/*
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
# Version of production
twine upload dist/*
Tests
Run the unit tests:
python3 -m unittest
Usage
See the example of usage in:
- https://gitlab.com/Inria-Chile/oceania/data-read
- http://twiki.inriadev.cl/twiki/bin/view/Sandbox/OceaniaClassDiagramPOCS3DVC
Docker
Build the image:
docker build -t common-mlops .
docker pull common-mlops
Run the image:
docker run -v$HOME/.aws:/root/.aws:ro -e S3_ACCESS_KEY_ID=access_key_id -e S3_SECRET_ACCESS_KEY=secret_access_key -e AWS_ACCESS_KEY_ID=access_key_id -e AWS_SECRET_ACCESS_KEY=secret_access_key common-mlops .
References
- https://test.pypi.org/project/common-mlops
- https://pypi.org/project/common-mlops
- https://anno-ai.medium.com/mlops-and-data-managing-large-ml-datasets-with-dvc-and-s3-part-1-d5b8f2fb8280
- https://martinfowler.com/articles/cd4ml.html
- https://christophergs.com/machine%20learning/2020/03/14/how-to-monitor-machine-learning-models/
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for common_mlops-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af9d3028a776f22bfdb8723a515d9c32fe318ccd1fbfce67ae00de034c83a13d |
|
MD5 | 800e67f375c76d08b65721197d8805a0 |
|
BLAKE2b-256 | b908e7cf1f765c4cd5a40109715809d850d76612f339ba5e2a61ac3b8cfeff5e |