packaged-non-std-sai-bandaru 0.0.1

A custom package for the House Price Prediction project

MLE-TRAINING

Median housing value prediction

The housing data can be downloaded from https://raw.githubusercontent.com/ageron/handson-ml/master/. The script has codes to download the data. We have modelled the median house value on given housing data.

The following techniques have been used:

• Linear regression
• Decision Tree
• Random Forest

Steps performed

• We prepare and clean the data. We check and impute for missing values.
• The dataset is consisting of several missing values like. the total_bedrooms column in possing around 150 missing values and the income category column is containing 207 missing values
• We handled the missing values in 'total_bedrooms' by using the mean value for the column and used the mode (most repeated) value of the 'income_cat' column to treat its missing values.
• Features are generated and the variables are checked for correlation.
• Multiple sampling techinuqies are evaluated. The data set is split into train and test.
• All the above said modelling techniques are tried and evaluated. The final metric used to evaluate is mean squared error.

Execution of the script

• Inorder to set up the conda environment and make the execution process isolated, a special environment 'mle-dev' was created with the help of env.yml file generated. The folowing is the command to create the environment using env.yml file

  conda env create -f env.yml

• After creating the environment, the bug free file nonstandardcode.py was executed with the help of the following command.

  python3 nonstandardcode.py

• The screenshot of the output from running the nonstandardcode.py successfully in the mle-dev environment was attached as part of PR description

Packaging and Testing

Breaking the nonstandardcode.py file into three files

• The nonstandardcode.py file was converted into three seperate files namely ingest_data.py, train.py and score.py
• The ingest_data.py file is responsible for preprocessing the data and creation of valid test and train datasets that are ready for training the model
• The ingest_data.py file was run by giving the path to the output folder where the processed files are to be stored using the command prompt.
• The train.py file is solely responsible for training the models. It uses different algorithms to train using the datasets and as an output,it generates the pickle files which are stored in the artifacts folder of the project
• The score.py file is responsible for evaluating the performance of the models by considering the pickle files and prints the scores to the terminal

Running Tests

• Created two unit tests in the tests folder of the project root folder and in that folder, two seperate folders were created, one for unit testing and one for functional testing.
• Ran unit tests for two file, ingest_data.py and train.py files, evaluating whether the the generated outputs are valid
• Ran functional test for the proper installation of the package in a virtualenvironment created using the subprocess inbuilt package in python

Packaging

• Created the pyproject.toml file to create the package.The following command is used to create the package

       **python -m build** 
  

• After converting it to a package, sphinx - a python documentation generating tool is used to create the proper documentation and to generate html document to the prepared package.

• The following are the commands used for generating html content

         **conda install sphinx**
  
         **sphinx-quickstart**
  

• The paths to the source code of the package are included in the conf.py file in the source directory generated by the sphinx.

• Included the 'index.rst' file as master_doc in the conf.py file

• The following is the command used for generating the html

         **sphinx -b html <source code folder path of the package> <source folder generated by the sphinx>**
  

• The package can be installed by using the .whl file which was created while building the package

• The package name is "packaged_non_std" and it can be installed as

          **from packaged_non_std import * **
  

• Inorder to test the successfull installation of the package, try running it using the following steps

  • create a similar kind of environment with the dependencies of the packaged_non_std mentioned in the env.yml file situated in the deploy folder using the following command

    ** conda env create --name <new_env_name> --file

  • after successful installation, test the proper functioning of the package using functional tests mentioned in the tests directory of the project root folder.