income-predict-d100-d400 0.1.2

My installable package for income prediction

d100_d400_income_predict

Overview

This repository provides a reproducible Docker environment pre-configured with everything needed to run the GLM and LGBM models for predicting high income basaed on the 1994 US census dataset.

The main analysis can be found at: src/notebooks/final_report.ipynb

There is are other sub-analysis files, they are:

• src/tests/benchmark_pandas_polars.py script that highlights the performance differences between Polars and Pandas on loading and cleaning this specific dataset.
• src/notebooks/eda_cleaning.ipynb exploratory data analysis. Lots more charts and info on how and why certain decisions were made in building the models.

Installation

There are two ways to install and run:

• Directly from PyPI as a package (easiest)
• Docker container (most robust, recommended for development)

Install and Run - Method 1, PyPI

1. Install the package

pip install income_predict_d100_d400

2. Run the Pipeline

python -m income_predict_d100_d400.training_pipeline

or create a your own file and import income_predict_d100_d400:

from income_predict_d100_d400.data import load_data
from income_predict_d100_d400.cleaning import run_cleaning_pipeline
from income_predict_d100_d400.evaluation import run_evaluation
from income_predict_d100_d400.model_training import (
    TARGET,
    load_training_outputs,
    run_split,
    run_training,
)

print("Starting Pipeline...")

file_path = load_data()
df_raw = pd.read_parquet(file_path)
run_cleaning_pipeline(df_raw)
run_split()
run_training()

results = load_training_outputs()

run_evaluation(
    results["test"],
    TARGET,
    results["glm_model"],
    results["lgbm_model"],
    results["train_features"],
)

print("Pipeline finished.")

Install and Run - Method 2, Docker

1. Download and install Docker Desktop (if you don't have it already)

• link: Docker Desktop

2. Clone the Repository

git clone https://github.com/caitpj/d100_d400_income_predict.git
cd d100_d400_income_predict

2. Build the Docker Image

docker build -t conda-uciml . (from root of d100_d400_income_predict)

3. Run the Model Pipeline

This runs the model in the Docker container, including downloading the data, cleaning, training, tuning, and saving key visualisations. It should take a minuite or so to run.

docker run --rm --shm-size=2g \
-e PYTHONWARNINGS=ignore \
-e PYTHONUNBUFFERED=1 \
-e OMP_NUM_THREADS=1 \
conda-uciml python src/income_predict_d100_d400/training_pipeline.py

4. Run Notebooks

docker run --rm -it \
-p 8888:8888 \
-v "$(pwd):/app" \
conda-uciml \
jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root

From the output of the above code, find and paste the URL into a browser. It should start with: http://127.0.0.1:8888/?token=...

Extra Steps for Development

There are a few more steps needed if you want to develop this repo on your local machine.

To ensure code quality, I use pre-commit hooks that run locally on your machine before every commit. As well as pytest for unit tests. These require a local Conda environment on your host machine (not in Docker).

1. Download and install Miniconda (if you don't have it already)

• link: Conda installed on your machine (for local development and git hooks).

2. Install required packages based on environment.yml

conda env update --file environment.yml --prune

3. initialize conda (will need to reset terminal)

conda init zsh

4. Activate the environment

conda activate d100_d300_env

5. Install the git hooks

pre-commit install

Now, every time you run git commit, your local machine will fist check it meets the rules stated in .pre-commit-config.yaml automatically. You can also run pytests in the src/tests directory.

AI Use

Some code was AI generated, notably:

• Visualisations
• Pandas vs Polars benchmark test
• Pretty terminal outputs
• Full docstrings

In other areas, AI was used to help with debugging, notably:

• Docker related issues
• Performence issues with hypertunning

All code generated from AI is understood and reviewed by the author.