Utilities of MLOps for INRIA
Project description
common-mlops
This is a set of utilities for end-to-end lifecycle of Machine Learning applications in Python for Inria-Chile. Could be used like a framework and tooling for rapid development of data science and MLOps functions.
Here’s the folder structure for the repository:
common-mlops/
|
├── data/
│ ├── prepared/
│ └── raw/
│ └── metrics/
├── model/
├── mlops/
| └── *.py
├── test/
| └── test_*.py
├── venv/
├── check_credentials.sh
├── install.sh
├── LICENSE
├── README.md
├── requirements.txt
├── s3*.py
└── setup.py
There are the folloging folders in our repository:
- data/ is all versions of the dataset.
- data/raw/ is the data obtained from an external source.
- data/prepared/ is for data modified internally.
- data/metrics/ is for tracking the performance metrics of our models.
- model/ is for machine learning models.
- mlops/ is the source code. Python classes used by the Python scripts.
- test/ is the pyunit of source code.
- venv/ is the virtual environment.
- check_credentials.sh bash for check the AWS credentials.
- install.sh bash for check the installation.
- LICENSE
- requirements.txt the dependencies.
- s3*.py Python scripts for bash executions.
- setup.py the configuration of the package.
Installation Linux
Perform the following steps in order to install this program. Replace 'access_key_id' and 'secret_access_key' by the account AWS Credentials associated to poc-inriacl.
git clone https://gitlab.com/Inria-Chile/common-mlops.git
./common-mlops/install.sh access_key_id secret_access_key
pip3 uninstall common-mlops
# Version of test
pip3 install -i https://test.pypi.org/simple/ common-mlops
# Version of production
pip3 install common-mlops
Updates in test.pypi.org
Perform the following steps in order to publish new versions to test.pypi.org:
sudo rm -rf build common_mlops.egg-info dist
bumpversion --current-version 1.0.0 minor setup.py mlops/__init__.py
python3 setup.py sdist bdist_wheel
# Version of test
twine check dist/*
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
# Version of production
twine upload dist/*
Tests
Run the unit tests:
python3 -m unittest
Usage
See the example of usage in:
- https://gitlab.com/Inria-Chile/oceania/data-read
- http://twiki.inriadev.cl/twiki/bin/view/Sandbox/OceaniaClassDiagramPOCS3DVC
Docker
Build the image:
docker build -t common-mlops .
docker pull common-mlops
Run the image:
docker run -v$HOME/.aws:/root/.aws:ro -e S3_ACCESS_KEY_ID=access_key_id -e S3_SECRET_ACCESS_KEY=secret_access_key -e AWS_ACCESS_KEY_ID=access_key_id -e AWS_SECRET_ACCESS_KEY=secret_access_key common-mlops .
References
- https://test.pypi.org/project/common-mlops
- https://pypi.org/project/common-mlops
- https://anno-ai.medium.com/mlops-and-data-managing-large-ml-datasets-with-dvc-and-s3-part-1-d5b8f2fb8280
- https://martinfowler.com/articles/cd4ml.html
- https://christophergs.com/machine%20learning/2020/03/14/how-to-monitor-machine-learning-models/
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file common-mlops-0.1.1.tar.gz
.
File metadata
- Download URL: common-mlops-0.1.1.tar.gz
- Upload date:
- Size: 14.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/53.0.0 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 33c6a5a66933fcf2af55531c3c96e6c63c3a0f17313ea29f095ac2df9ae2c6fe |
|
MD5 | 94680e2cb2a26e53d7e9401ad1b7d44a |
|
BLAKE2b-256 | 115f2e095da2fb6acf320d2a496f17da904eb23fd45f24ea9a8c750f8b5f71b9 |
File details
Details for the file common_mlops-0.1.1-py3-none-any.whl
.
File metadata
- Download URL: common_mlops-0.1.1-py3-none-any.whl
- Upload date:
- Size: 16.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/53.0.0 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | af9d3028a776f22bfdb8723a515d9c32fe318ccd1fbfce67ae00de034c83a13d |
|
MD5 | 800e67f375c76d08b65721197d8805a0 |
|
BLAKE2b-256 | b908e7cf1f765c4cd5a40109715809d850d76612f339ba5e2a61ac3b8cfeff5e |