Skip to main content

Feature Pipelines for Keras preprocessing layers.

Project description

EasyFlow: Keras Feature Preprocessing Pipelines

Keras logo

Table of Contents

  1. About EasyFlow
  2. Motivation
  3. Installation
  4. Example
  5. Tutorials

About EasyFlow

The EasyFlow package implements an interface similar to SKLearn's Pipeline API that contains easy feature preprocessing pipelines to build a full training and inference pipeline natively in Keras. All pipelines are implemented as Keras layers.


There is a need to have a similar interface for Keras that mimics the SKLearn Pipeline API such as Pipeline, FeatureUnion and ColumnTransformer, but natively in Keras as Keras layers. The usual design pattern especially for tabular data is to first do preprocessing with SKLearn and then feed the data to a Keras model. With EasyFlow you don't need to leave the Tensorflow/Keras ecosystem to build custom pipelines and your preprocessing pipeline is part of your model architecture.

Main interfaces are:

  • FeaturePreprocessor: This layer applies feature preprocessing steps and returns a separate layer for each step supplied. This gives more flexibility to the user and if a more advance network architecture is needed. For example something like a Wide and Deep network.
  • FeatureUnion: This layer is similar to FeaturePreprocessor with an extra step that concatenates all layers into a single layer.


pip install easy-tensorflow


Lets look at a quick example:

import pandas as pd
import tensorflow as tf
from tensorflow.keras.layers import Normalization, StringLookup, IntegerLookup

# local imports
from import TensorflowDataMapper
from easyflow.preprocessing import FeatureUnion
from easyflow.preprocessing import (

Read in data and map as

Use the TensorflowDataMapper class to map pandas data frame to a type.

file_url = ""
dataframe = pd.read_csv(file_url)
labels = dataframe.pop("target")

batch_size = 32
dataset_mapper = TensorflowDataMapper() 
dataset =, labels)
train_data_set, val_data_set = dataset_mapper.split_data_set(dataset)
train_data_set = train_data_set.batch(batch_size)
val_data_set = val_data_set.batch(batch_size)

Set constants

NUMERICAL_FEATURES = ['age', 'trestbps', 'chol', 'thalach', 'oldpeak', 'slope']
CATEGORICAL_FEATURES = ['sex', 'cp', 'fbs', 'restecg', 'exang', 'ca']
# thal is represented as a string

dtype_mapper = {
    "age": tf.float32,
    "sex": tf.float32,
    "cp": tf.float32,
    "trestbps": tf.float32,
    "chol": tf.float32,
    "fbs": tf.float32,
    "restecg": tf.float32,
    "thalach": tf.float32,
    "exang": tf.float32,
    "oldpeak": tf.float32,
    "slope": tf.float32,
    "ca": tf.float32,
    "thal": tf.string,

Setup Preprocessing layer using FeatureUnion

This is the main part where EasyFlow fits in. We can now easily setup a feature preprocessing pipeline as a Keras layer with only a few lines of code.

feature_preprocessor_list = [
    ('numeric_encoder', Normalization(), NUMERICAL_FEATURES),
    ('categorical_encoder', IntegerLookup(output_mode='multi_hot'), CATEGORICAL_FEATURES),
    ('string_encoder', StringToIntegerLookup(), STRING_CATEGORICAL_FEATURES)

preprocessor = FeatureUnion(feature_preprocessor_list)

feature_layer_inputs = FeatureInputLayer(dtype_mapper)
preprocessing_layer = preprocessor(feature_layer_inputs)

Set up network

# setup simple network
x = tf.keras.layers.Dense(128, activation="relu")(preprocessing_layer)
x = tf.keras.layers.Dropout(0.5)(x)
outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)
model = tf.keras.Model(inputs=feature_layer_inputs, outputs=outputs)
    metrics=[tf.keras.metrics.BinaryAccuracy(name='accuracy'), tf.keras.metrics.AUC(name='auc')])

Fit model, validation_data=val_data_set, epochs=10)


Migrate an Sklearn training Pipeline to Tensorflow Keras: Open In Colab

  • In this notebook we look at ways to migrate an Sklearn training pipeline to Tensorflow Keras. There might be a few reasons to move from Sklearn to Tensorflow.

Single Input Multiple Output Preprocessor: Open In Colab

  • In this example we will show case how to apply different transformations and preprocessing steps on the same feature. What we have here is an example of a Single input Multiple output feature transformation scenario.

Preprocessing module quick intro: Open In Colab

  • The easyflow.preprocessing module contains functionality similar to what Sklearn does with its Pipeline, FeatureUnion and ColumnTransformer does. This is a quick introduction.

Tensorflow Feature columns quick intro: Open In Colab

  • Model building Pipeline using EasyFlow feature_encoders module. This module is a fusion between Keras layers and Tensorflow feature columns.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

easy-tensorflow-1.4.3.tar.gz (15.5 kB view hashes)

Uploaded source

Built Distributions

easy_tensorflow-1.4.3-py3.7.egg (48.3 kB view hashes)

Uploaded 1 4 3

easy_tensorflow-1.4.3-py3-none-any.whl (21.9 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page