Skip to main content

Utilities of MLOps for INRIA

Project description

common-mlops

This is a set of utilities for end-to-end lifecycle of Machine Learning applications in Python for Inria-Chile. Could be used like a framework and tooling for rapid development of data science and MLOps functions.

Here’s the folder structure for the repository:

common-mlops/
|
├── data/
│   ├── prepared/
│   └── raw/
│   └── metrics/
├── model/
├── mlops/
|   └── *.py
├── test/
|   └── test_*.py
├── venv/
├── check_credentials.sh
├── install.sh
├── LICENSE
├── README.md
├── requirements.txt
├── s3*.py
└── setup.py

There are the folloging folders in our repository:

  • data/ is all versions of the dataset.
  • data/raw/ is the data obtained from an external source.
  • data/prepared/ is for data modified internally.
  • data/metrics/ is for tracking the performance metrics of our models.
  • model/ is for machine learning models.
  • mlops/ is the source code. Python classes used by the Python scripts.
  • test/ is the pyunit of source code.
  • venv/ is the virtual environment.
  • check_credentials.sh bash for check the AWS credentials.
  • install.sh bash for check the installation.
  • LICENSE
  • requirements.txt the dependencies.
  • s3*.py Python scripts for bash executions.
  • setup.py the configuration of the package.

Installation Linux

Perform the following steps in order to install this program. Replace 'access_key_id' and 'secret_access_key' by the account AWS Credentials associated to poc-inriacl.

git clone https://gitlab.com/Inria-Chile/common-mlops.git
./common-mlops/install.sh access_key_id secret_access_key
pip3 uninstall common-mlops
# Version of test
pip3 install -i https://test.pypi.org/simple/ common-mlops
# Version of production
pip3 install common-mlops

Updates in test.pypi.org

Perform the following steps in order to publish new versions to test.pypi.org:

sudo rm -rf build common_mlops.egg-info dist
bumpversion --current-version 1.0.0 minor setup.py mlops/__init__.py
python3 setup.py sdist bdist_wheel
# Version of test
twine check dist/*
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
# Version of production
twine upload dist/*

Tests

Run the unit tests:

python3 -m unittest

Usage

See the example of usage in:

Docker

Build the image:

docker build -t common-mlops .
docker pull common-mlops

Run the image:

docker run -v$HOME/.aws:/root/.aws:ro -e S3_ACCESS_KEY_ID=access_key_id -e S3_SECRET_ACCESS_KEY=secret_access_key -e AWS_ACCESS_KEY_ID=access_key_id -e AWS_SECRET_ACCESS_KEY=secret_access_key common-mlops .

References

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

common-mlops-0.1.1.tar.gz (14.6 kB view hashes)

Uploaded source

Built Distribution

common_mlops-0.1.1-py3-none-any.whl (16.7 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page