DIVER

Diver is the Dataset Inspector, Visualiser and Encoder library, automating and codifying common data science project steps as standardised and reusable methods.

See example-notebooks/house-price-demo.ipynb for a full walkthrough.

`dataset_inspector`

A set of functions which help perform checks for common dataset issues which can impact machine learning model performance.

inspector flow

`dataset_conditioner`

A scikit-learn-formatted module which can perform various data-type encodings in a single go, and save the associated attributes from a train-set encoding to reuse on a test-set encoding:

The .fit_transform method learns various encodings (feature means and variances; categorical feature elements - yellow in the flow chart below) and then performs the various encodings on the feature train set
The .transform method applies train-set encodings to a test set

fit_transform flow

`dataset_visualiser`

Functions for visualising aspects of the dataset

Correlation analysis

Display the correlation matrix for the top n correlating features (n specified by the user) against the dependent variable (at the bottom row of the matrix)

correlation

Future Work

`categorical_excess_cardinality_flagger_and_reducer`

Option for instances where there are no categorical features

`missing_value_conditioner`

Choose between either {use means from train set (default), calculate means for test set}
Missing values for categorical features
Implement missing value imputation: https://measuringu.com/handle-missing-data/
GOOD READING: https://towardsdatascience.com/6-different-ways-to-compensate-for-missing-values-data-imputation-with-examples-6022d9ca0779

`ordinal_encoder`

Create a function to do this

`timestamp_encoder`

is_public_holiday : bool
Update above diagram

Remove warnings

Make robust to non-consecutive indices in input df

Unit test all functions

Extreme values

PCA option?

Label balanced class checker (for classification problems)

https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/

Distribution and correlation analysis

Display correlation matrix for top n correlates alongside target at the bottom
Display pairplot for top n correlates alongside target at the bottom
Or instead of top n correlates, instead threshold of cumulative variance
Option to DROOP lower correlates (lower than threshold) if desired

Useful reading

Release files for diver 0.2.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for diver 0.2.3
File	Size	Uploaded
diver-0.2.3.tar.gz	29.4 kB	Mar 6, 2020	Details

Built distribution (wheel)

Table of built distributions (wheels) for diver 0.2.3
File	Interpreter	ABI	Platform	Reset
diver-0.2.3-py3-none-any.whl 33.0 kB Mar 6, 2020	Python 3	none	any	Details

Total release size: 62.3 kB

Release files / diver-0.2.3.tar.gz

Download URL	diver-0.2.3.tar.gz
Size	29.4 kB
Tags	Source
SHA-256 checksum How to use checksums	`2d68f394a7c72d180fbe2c888b95c8c536d5ab84a03e6d910fcf721713182f24`
BLAKE2b-256 checksum How to use checksums	`1ba78ad21679d46a80ed325e1c3d937f20926a94a0bcd90fffd6abd472b4cf4a`
Upload date	Mar 6, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.3`

Release files / diver-0.2.3-py3-none-any.whl

Download URL	diver-0.2.3-py3-none-any.whl
Size	33.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`e6dd526cc135ca21af7ce6cac6afa896e260081950a9d9db6a0b6c3d1c1b87c1`
BLAKE2b-256 checksum How to use checksums	`0126eecd71793d588711c3ee65bced6191d393b0d460ab90d27ac09816e9cea7`
Upload date	Mar 6, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.3`