Training Pipeline

Check out Lesson 2 on Medium to better understand how we built the training pipeline.

Install for Development

Create virtual environment:

cd training-pipeline
poetry shell
poetry install

Check the Set Up Additional Tools and Usage sections to see how to set up the additional tools and credentials you need to run this project.

Usage for Development

Run the scripts in the following order:

1. Start the hyperparameter tuning script:

python -m training_pipeline.hyperparameter_tuning

2. Upload the best config based on the previous hyperparameter tuning step:

python -m training_pipeline.best_config

3. Start the training script using the best configuration uploaded one step before:

python -m training_pipeline.train

NOTE: Be careful to complete the .env file and set the ML_PIPELINE_ROOT_DIR variable as explained in the Set Up the ML_PIPELINE_ROOT_DIR Variable section of the main README.