Manage training results, weights and data flow of your Tensorflow models
Project description
MLPipe-Trainer
Manage your Data Pipline and Tensorflow & Keras models with MLPipe. It is NOT another "wrapper" around Tensorflow, but rather adds utilities to setup an environment to control data flow and managed trained models (weights & results) with the help of Conda and MongoDB.
Table of Contents:
Application Setup
In case you create a new project, copy the environment.yml into your new folder and you are ready to go. If you want to use an existing tensorflow (v2) project, you will need to copy the dependencies from the this environment file to yours.
Install Conda
You can either install Anaconda or Miniconda (https://conda.io/miniconda.html). It is used for package and environment management. The environment.yml file is specifying all the packages needed. For a new project, copy the file from this repo.
During installation, check the box to add conda
to your PATH in the .bashrc file or do it manually afterwards.
>> conda env create -f environment.yml
# And in case you need to update the environment later on
>> conda env update -f environment.yml
This will create a conda environment and install all the needed packages (as described in environment.yml).
Install MongoDB
MongoDB database is used to store trained Models including their weights and results. Additionally there is also a data reader for MongoDB implemented (basically just a generator as you know and love from using keras). Currenlty that is the only implemented data reader working "out of the box".
Follow the instructions on the MongoDB website for installation e.g. for Linux: https://docs.mongodb.com/manual/administration/install-on-linux/
Install PyCharm (optional)
If you development python applications, PyCharm is most probably your goto editor. For installation, use this link and follow the instructions: https://www.jetbrains.com/pycharm/download
After the conda environment is set up, it can be added to the pycharm. Follow:
- File -> Settings -> Project -> Project Interpreter -> Add
- Chose "Existing environment"
- CONDA_PATH/envs/mlpipe_env/bin/python
- Select added environment
Now pycharm will use this conda environment and can access all installed dependencies while developing MLPipe-Trainer with PyCharm.
Getting Started
Export the MLPipe-Trainer root to the python path either with (can also be added to .bashrc):
# change path accordingly
>> export PYTHONPATH="/home/USER/MLPipe-Trainer"
To activate the conda environment call:
conda activate mlpipe_env
For an example you can check out the Cifar-10 example in the examples folder
Road Map
- Create and generat MkDocs documentation & host documentation
- Add tests
- Redis cache in case data is fetched from remote source
- Better dependency handling...
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for mlpipe_trainer-0.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 59c34688ed423e15c7595beae779e421a726dc4add45116ba35ddb696f01053e |
|
MD5 | 7ed7bb7b09a950d1cad62f050623bf35 |
|
BLAKE2b-256 | 67d25c9c45992a60551aa48366e0690d7e29f32e5f7552de4e342124e197e2e3 |