Skip to main content
Join the official 2020 Python Developers SurveyStart the survey!

Peer into the future of a data science project

Project description

BuildStatus Documentation Status Coverage Code Style License

Foreshadow is an automatic pipeline generation tool that makes creating, iterating, and evaluating machine learning pipelines a fast and intuitive experience allowing data scientists to spend more time on data science and less time on code.

Key Features

  • Scikit-Learn compatible
  • Automatic column intent inference
    • Numerical
    • Categorical
    • Text
    • Droppable (All values in a column are either the same or different)
  • Allow user override on column intent and transformation functions
  • Automatic feature preprocessing depending on the column intent type
    • Numerical: imputation followed by scaling
    • Categorical: a variety of categorical encoding
    • Text: TFIDF followed by SVD
  • Automatic model selection
  • Rapid pipeline development / iteration

Features in the road map

  • Automatic feature engineering
  • Automatic parameter optimization

Foreshadow supports python 3.6+

Installing Foreshadow

$ pip install foreshadow

Read the documentation to set up the project from source.

Getting Started

To get started with foreshadow, install the package using pip install. This will also install the dependencies. Now create a simple python script that uses all the defaults with Foreshadow.

First import foreshadow

from foreshadow.foreshadow import Foreshadow
from foreshadow.estimators import AutoEstimator
from foreshadow.utils import ProblemType

Also import sklearn, pandas, and numpy for the demo

import pandas as pd

from sklearn.datasets import boston_housing
from sklearn.model_selection import train_test_split

Now load in the boston housing dataset from sklearn into pandas dataframes. This is a common dataset for testing machine learning models and comes built in to scikit-learn.

boston = load_boston()
bostonX_df = pd.DataFrame(, columns=boston.feature_names)
bostony_df = pd.DataFrame(, columns=['target'])

Next, exactly as if working with an sklearn estimator, perform a train test split on the data and pass the train data into the fit function of a new Foreshadow object

X_train, X_test, y_train, y_test = train_test_split(bostonX_df,
   bostony_df, test_size=0.2)

problem_type = ProblemType.REGRESSION

estimator = AutoEstimator(
    estimator_kwargs={"max_time_mins": 1},
shadow = Foreshadow(estimator=estimator, problem_type=problem_type), y_train)

Now fs is a fit Foreshadow object for which all feature engineering has been performed and the estimator has been trained and optimized. It is now possible to utilize this exactly as a fit sklearn estimator to make predictions.

shadow.score(X_test, y_test)

Great, you now have a working Foreshaow installation! Keep reading to learn how to export, modify and construct pipelines of your own.


We also have a jupyter notebook tutorial to go through more details under the examples folder.


Read the docs!

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for foreshadow, version 1.0.1
Filename, size File type Python version Upload date Hashes
Filename, size foreshadow-1.0.1-py3-none-any.whl (4.1 MB) File type Wheel Python version py3 Upload date Hashes View
Filename, size foreshadow-1.0.1.tar.gz (3.9 MB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page