pdpipe 0.0.7

Easy pipelines for pandas.

Easy pipelines for pandas DataFrames.

>>> df = pd.DataFrame(
        data=[[4, 165, 'USA'], [2, 180, 'UK'], [2, 170, 'Greece']],
        index=['Dana', 'Jack', 'Nick'],
        columns=['Medals', 'Height', 'Born']
    )
>>> pipeline = pdp.ColDrop('Medals').Binarize('Born')
>>> pipeline(df)
            Height  Born_UK  Born_USA
    Dana     165        0         1
    Jack     180        1         0
    Nick     170        0         0

Contents

• 1 Installation

• 2 Features

  • 2.1 Design Decisions

• 3 Use

  • 3.1 Pipeline Stages

    • 3.1.1 Creating Pipeline Stages

    • 3.1.2 Applying Pipelines Stages

    • 3.1.3 Fittable Pipeline Stages

    • 3.1.4 Extending PipelineStage

    • 3.1.5 Ad-Hoc Pipeline Stages

  • 3.2 Pipelines

    • 3.2.1 Creating Pipelines

    • 3.2.2 Pipeline Arithmetics

    • 3.2.3 Pipeline Chaining

    • 3.2.4 Pipeline Slicing

    • 3.2.5 Applying Pipelines

• 4 Types of Pipeline Stages

  • 4.1 Basic Stages

  • 4.2 Column Generation

  • 4.3 Scikit-learn-dependent Stages

• 5 Contributing

  • 5.1 Installing for development

  • 5.2 Running the tests

  • 5.3 Adding documentation

• 6 Credits

1 Installation

Install pdpipe with:

pip install pdpipe

Some pipeline stages require scikit-learn; they will simply not be loaded if scikit-learn is not found on the system, and pdpipe will issue a warning. To use them you must also install scikit-learn.

2 Features

• Pure Python.

• Compatible with Python 3.5+.

• A simple interface.

• Informative prints and errors on pipeline application.

• Chaining pipeline stages constructor calls for easy, one-liners pipelines.

• Pipeline arithmetics.

2.1 Design Decisions

• Data science-oriented naming (rather than statistics).

• A functional approach: Pipelines never change input DataFrames. Nothing is done “in place”.

• Opinionated operations: Help novices avoid mistake by default appliance of good practices; e.g., binarizing (creating dummy variables) a column will drop one of the resulting columns by default, to avoid the dummy variable trap (perfect multicollinearity).

• Machine learning-oriented: The target use case is transforming tabular data into a vectorized dataset on which a machine learning model will be trained; e.g., column transformations will drop the source columns to avoid strong linear dependence.

3 Use

3.1 Pipeline Stages

3.1.1 Creating Pipeline Stages

You can create stages with the following syntax:

import pdpipe as pdp
drop_name = pdp.ColDrop("Name")

All pipeline stages have a predefined precondition function that returns True for dataframes to which the stage can be applied. By default, pipeline stages raise an exception if a DataFrame not meeting their precondition is piped through. This behaviour can be set per-stage by assigning exraise with a bool in the constructor call. If exraise is set to False the input DataFrame is instead returned without change:

drop_name = pdp.ColDrop("Name", exraise=False)

3.1.2 Applying Pipelines Stages

You can apply a pipeline stage to a DataFrame using its apply method:

res_df = pdp.ColDrop("Name").apply(df)

Pipeline stages are also callables, making the following syntax equivalent:

drop_name = pdp.ColDrop("Name")
res_df = drop_name(df)

The initialized exception behaviour of a pipeline stage can be overridden on a per-application basis:

drop_name = pdp.ColDrop("Name", exraise=False)
res_df = drop_name(df, exraise=True)

Additionally, to have an explanation message print after the precondition is checked but before the application of the pipeline stage, pass verbose=True:

res_df = drop_name(df, verbose=True)

3.1.3 Fittable Pipeline Stages

Some pipeline stages can be fitted, meaning that some transformation parameters are set the first time a dataframe is piped through the stage, while later applications of the stage use these now-set parameters without changing them; the Encode stage is a good example.

If you want to re-fit an already fitted pipeline stage use the fit_transform method to re-fit the stage to a new dataframe. Notice that for an unfitted stage apply and fit_transform are equivalent, and only later calls to apply will transform input dataframes without refitting the stage.

Finally, apply and fit_transform are of course equivalent for non-fittable pipeline stages.

3.1.4 Extending PipelineStage

To use other stages than the built-in ones (see Types of Pipeline Stages) you can extend the PipelineStage class. The constructor must pass the PipelineStage constructor the exmsg, appmsg and desc keyword arguments to set the exception message, application message and description for the pipeline stage, respectively. Additionally, the _prec and _op abstract methods must be implemented to define the precondition and the effect of the new pipeline stage, respectively.

Fittable custom pipeline stages should implement, additionally to the _op method, the _transform method, which should apply the fitted pipeline to an input dataframe, while also setting self.is_fitted = True. The _op method then acts as the fit_tranform for the stage.

3.1.5 Ad-Hoc Pipeline Stages

To create a custom pipeline stage without creating a proper new class, you can instantiate the AdHocStage class which takes a function in its op constructor parameter to define the stage’s operation, and the optional prec parameter to define a precondition (an always-true function is the default).

3.2 Pipelines

3.2.1 Creating Pipelines

Pipelines can be created by supplying a list of pipeline stages:

pipeline = pdp.Pipeline([pdp.ColDrop("Name"), pdp.Binarize("Label")])

3.2.2 Pipeline Arithmetics

Alternatively, you can create pipelines by adding pipeline stages together:

pipeline = pdp.ColDrop("Name") + pdp.Binarize("Label")

Or even by adding pipelines together or pipelines to pipeline stages:

pipeline = pdp.ColDrop("Name") + pdp.Binarize("Label")
pipeline += pdp.ApplyToRows("Job", {"Part": True, "Full":True, "No": False})
pipeline += pdp.Pipeline([pdp.ColRename({"Job": "Employed"})])

3.2.3 Pipeline Chaining

Pipeline stages can also be chained to other stages to create pipelines:

pipeline = pdp.ColDrop("Name").Binarize("Label").ValDrop([-1], "Children")

3.2.4 Pipeline Slicing

Pipelines are Python Sequence objects, and as such can be sliced using Python’s slicing notation, just like lists:

pipeline = pdp.ColDrop("Name").Binarize("Label").ValDrop([-1], "Children").ApplyByCols("height", math.ceil)
result_df = pipeline[1:2](df)

3.2.5 Applying Pipelines

Pipelines are pipeline stages themselves, and can be applied to a DataFrame using the same syntax, applying each of the stages making them up, in order:

pipeline = pdp.ColDrop("Name") + pdp.Binarize("Label")
res_df = pipeline(df)

Assigning the exraise parameter to a pipeline apply call with a bool sets or unsets exception raising on failed preconditions for all contained stages:

pipeline = pdp.ColDrop("Name") + pdp.Binarize("Label")
res_df = pipeline.apply(df, exraise=False)

Additionally, passing verbose=True to a pipeline apply call will apply all pipeline stages verbosely:

res_df = pipeline.apply(df, verbose=True)

Finally, to re-fit all fittable pipeline stages in the pipeline use the fit_transform method, which calls the corresponding method for all composing pipeline stages.

4 Types of Pipeline Stages

All built-in stages are thoroughly documented, including examples; if you find any documentation lacking please open an issue. A list of briefly described available built-in stages follows:

4.1 Basic Stages

• AdHocStage - Define custom pipeline stages on the fly.

• ColDrop - Drop columns by name.

• ValDrop - Drop rows by by their value in specific or all columns.

• ValKeep - Keep rows by by their value in specific or all columns.

• ColRename - Rename columns.

• DropNa - Drop null values. Supports all parameter supported by pandas.dropna function.

• FreqDrop - Drop rows by value frequency threshold on a specific column.

4.2 Column Generation

• Bin - Convert a continuous valued column to categoric data using binning.

• Binarize - Convert a categorical column to the several binary columns corresponding to it.

• ApplyToRows - Generate columns by applying a function to each row.

• ApplyByCols - Generate columns by applying an element-wise function to columns.

4.3 Scikit-learn-dependent Stages

• Encode - Encode a categorical column to corresponding number values.

5 Contributing

Package author and current maintainer is Shay Palachy ([e-mail]); You are more than welcome to approach him for help. Contributions are very welcomed, especially since this package is very much in its infancy and many other pipeline stages can be added.

5.1 Installing for development

Clone:

git clone [e-mail]:shaypal5/pdpipe.git

Install in development mode with test dependencies:

cd pdpipe
pip install -e ".[test]"

5.2 Running the tests

To run the tests, use:

python -m pytest --cov=pdpipe

5.3 Adding documentation

This project is documented using the numpy docstring conventions, which were chosen as they are perhaps the most widely-spread conventions that are both supported by common tools such as Sphinx and result in human-readable docstrings (in my personal opinion, of course). When documenting code you add to this project, please follow these conventions.

6 Credits

Created by Shay Palachy ([e-mail]).