Piper

Piper is a python module to simplify data wrangling with pandas using a set of wrapper functions. These functions or 'verbs' attempt to provide a simpler interface to standard Pandas functions.

When combined within a Jupyter notebook using the %%piper magic command, a simple data 'pipeline' that looks rather like SQL syntax can be built.

The concept is similar to the way R's tidyverse and magrittr libraries are used.

The main dataframe manipulation functions are:

• select()
• assign()
• relocate()
• where()
• group_by()
• summarise()
• order_by()

For other piper functionality, please see the Goals and Features section.

Alternatives

For a comprehensive alternative, please check out Michael Chow's siuba package.

Table of contents

• Installation
• Basic use
• Documentation
• Goals and Features
• Status
• Inspiration
• Contact

Installation

To install the package, enter the following:

pip install dpiper

Basic use

Suppose you need the following function to trim a given dataframes columnar text data.

def trim_columns(df):
    ''' Trim blanks for given dataframe '''

    str_cols = df.select_dtypes(include='object').columns

    for col in str_cols:
        df[col] = df[col].str.strip()

    return df

Standard Pandas can combine the new function into a pipeline along with other transformation/filtering tasks:

import pandas as pd
from piper.factory import get_sample_data

df = get_sample_data()

# Select all columns EXCEPT 'dates'
subset_cols = ['order_dates', 'regions', 'countries', 'values_1', 'values_2']

criteria1 = ~df['countries'].isin(['Italy', 'Portugal'])
criteria2 = df['values_1'] > 40
criteria3 = df['values_2'] < 25

df2 = (df[subset_cols][criteria1 & criteria2 & criteria3]
       .pipe(trim_columns)
       .sort_values('countries', ascending=False))

df2.head()

Result:

Using piper

Piper tries to improve this pipeline approach. Let's import piper's %%piper magic command and piper 'verbs'.

from piper import piper
from piper.verbs import head, select, where, group_by, summarise, order_by

Using the %%piper magic function, piper verbs can be combined with standard python functions like trim_columns() using the linking symbol '>>' to form a data pipeline.

%%piper
get_sample_data()
>> trim_columns()
>> select('-regions')
>> where(""" ~countries.isin(['Italy', 'Portugal']) &
              values_1 > 40 &
              values_2 < 25 """)
>> order_by('countries', ascending=False)
>> head(5)

Goals and Features

• Enhance working with Excel files through the WorkBook class
  • Exporting high quality formatted Excel Workbooks using xlsxwriter
• Provide access to databases with support for SQL based scripting and connections.

To-do list

• TBD

Documentation

Further examples are available in these jupyter notebooks:

• piper demo jupyter notebooks

Status

Project has just started. I welcome any and all help to improve etc.

Inspiration

Pandas and numpy are amazing data analysis libraries. My goal is to combine their power with the convenience and ease of use of the R tidyverse package suite.

Contact

This is very much a personal library, in that its highly opinionated, flawed and probably of no use to anyone else :). However if it helps anyone else in their endeavours, that would be fantastic to hear about.

If you'd like to contact me, I'm miketarpey@gmx.net.