AdvancedAnalytics 1.5

Python support for 'The Art and Science of Data Analytics'

AdvancedAnalytics

A collection of python modules, classes and methods for simplifying the use of machine learning solutions. AdvancedAnalytics provides easy access to advanced tools in Sci-Learn, NLTK and other machine learning packages. AdvancedAnalytics was developed to simplify learning python from the book The Art and Science of Data Analytics.

Description

From a high level view, building machine learning applications typically proceeds through three stages:

1. Data Preprocessing

2. Modeling or Analytics

3. Postprocessing

The classes and methods in AdvancedAnalytics primarily support the first and last stages of machine learning applications.

Data scientists report they spend 80% of their total effort in first and last stages. The first stage, data preprocessing, is concerned with preparing the data for analysis. This includes:

1. identifying and correcting outliers,

2. imputing missing values, and

3. encoding data.

The last stage, solution postprocessing, involves developing graphic summaries of the solution, and metrics for evaluating the quality of the solution.

Documentation and Examples

The API and documentation for all classes and examples are available at https://github.com/tandonneur/AdvancedAnalytics .

Usage

Currently the most popular usage is for supporting solutions developed using these advanced machine learning packages:

• Sci-Learn

• StatsModels

• NLTK

The intention is to expand this list to other packages. This is a simple example for linear regression that uses the data map structure to preprocess data:

from AdvancedAnalytics.ReplaceImputeEncode import DT
from AdvancedAnalytics.ReplaceImputeEncode import ReplaceImputeEncode
from AdvancedAnalytics.Tree import tree_regressor
from sklearn.tree import DecisionTreeRegressor, export_graphviz
# Data Map Using DT, Data Types
data_map = {
    “Salary”:         [DT.Interval, (20000.0, 2000000.0)],
    “Department”:     [DT.Nominal, (“HR”, “Sales”, “Marketing”)]
    “Classification”: [DT.Nominal, (1, 2, 3, 4, 5)]
    “Years”:          [DT.Interval, (18, 60)] }
# Preprocess data from data frame df
rie = ReplaceImputeEncode(data_map=data_map, interval_scaling=None,
                          nominal_encoding= “SAS”, drop=True)
encoded_df = rie.fit_transform(df)
y = encoded_df[“Salary”]
X = encoded_df.drop(“Salary”, axis=1)
dt = DecisionTreeRegressor(criterion= “gini”, max_depth=4
                            min_samples_split=5, min_samples_leaf5)
dt = dt.fit(X,y)
tree_regressor.display_importance(dt, encoded_df.columns)
tree_regressor.display_metrics(dt, X, y)

Current Modules and Classes

ReplaceImputeEncode

Classes for Data Preprocessing

• DT defines new data types used in the data dictionary

• ReplaceImputeEncode a class for data preprocessing

Regression

Classes for Linear and Logistic Regression

• linreg support for linear regressino

• logreg support for logistic regression

• stepwise a variable selection class

Tree

Classes for Decision Tree Solutions

• tree_regressor support for regressor decision trees

• tree_classifier support for classification decision trees

Forest

Classes for Random Forests

• forest_regressor support for regressor random forests

• forest_classifier support for classification random forests

NeuralNetwork

Classes for Neural Networks

• nn_regressor support for regressor neural networks

• nn_classifier support for classification neural networks

Text

Classes for Text Analytics

• text_analysis support for topic analysis

• text_plot for word clouds

• sentiment_analysis support for sentiment analysis

Internet

Classes for Internet Applications

• scrape support for web scrapping

• metrics a class for solution metrics

Installation and Dependencies

AdvancedAnalytics is designed to work on any operating system running python 3. It can be installed using pip or conda.

pip install AdvancedAnalytics
# or
conda install -c dr.jones AdvancedAnalytics

General Dependencies

There are dependencies. Most classes import one or more modules from Sci-Learn, referenced as sklearn in module imports, and StatsModels. These are both installed with the current version of anaconda.

Installed with AdvancedAnalytics

Most packages used by AdvancedAnalytics are automatically installed with its installation. These consist of the following packages.

• statsmodels

• scikit-learn

• scikit-image

• nltk

• pydotplus

• python-graphviz

• wordcloud

• newspaper3k

Other Dependencies

The Tree and Forest modules plot decision trees and importance metrics using pydotplus and the graphviz packages. These should also be automatically installed with AdvancedAnalytics.

However, the graphviz install is sometimes not fully complete with the conda install. It may require an additional pip install.

pip install graphviz

Text Analytics Dependencies

The TextAnalytics module uses the NLTK, Sci-Learn, and wordcloud packages. Usually these are also automatically installed automatically with AdvancedAnalytics. You can verify they are installed using the following commands.

conda list nltk
conda list sci-learn
conda list wordcloud

However, when the NLTK package is installed, it does not install the data used by the package. In order to load the NLTK data run the following code once before using the TextAnalytics module.

#The following NLTK commands should be run once
nltk.download(“punkt”)
nltk.download(“averaged_preceptron_tagger”)
nltk.download(“stopwords”)
nltk.download(“wordnet”)

The wordcloud package also uses a little know package tinysegmenter version 0.3. Run the following code to ensure it is installed.

conda install -c conda-forge tinysegmenter==0.3
# or
pip install tinysegmenter==0.3

Internet Dependencies

The Internet module contains a class scrape which has some functions for scraping newsfeeds. Some of these use the newspaper3k package. It should be automatically installed with AdvancedAnalytics.

However, it also uses the package newsapi-python, which is not automatically installed. If you intended to use this news scraping scraping tool, it is necessary to install the package using the following code:

conda install -c conda-forge newsapi
# or
pip install newsapi

In addition, the newsapi service is sponsored by a commercial company www.newsapi.com. You will need to register with them to obtain an API key required to access this service. This is free of charge for developers, but there is a fee if newsapi is used to broadcast news with an application or at a website.

Code of Conduct

Everyone interacting in the AdvancedAnalytics project’s codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PyPA Code of Conduct: https://www.pypa.io/en/latest/code-of-conduct/ .