• Documentation

Deep Feature Synthesis in Rust

This is a project to implement the Deep Feature Synthesis algorithm in Rust.

Roadmap

• (30JUL2022¹) Parity with Featuretools creating feature definitions (no calculation) on single table
• Parity with Featuretools creating feature definitions (no calculation) on multiple tables
• Explore Calculation on single table using Polars
• Explore Calculation on multiple tables using Polars

Running in Python

pip install rust_dfs

Using from python

# Import Featuretools, rust_dfs, and some other utility functions
import featuretools as ft
from rust_dfs.utils import *
from rust_dfs import generate_features, compare_featuresets, Feature
from rust_dfs.generate_fake_dataframe import generate_pandas_fake_dataframe

# Generate a fake dataset with 4 Numeric columns
df = generate_pandas_fake_dataframe(
    n_rows=10,
    col_defs=[
        ("Numeric", 4),
    ]
)

# pick some primitives
f_primitives = [
    ft.primitives.GreaterThan,
    # ft.primitives.LessThan
]

# or use all of them
# f_primitives = list(ft.primitives.utils.get_transform_primitives().values())

# convert datafame to an entityset
es = df_to_es(df)

# run dfs with features_only=True
ft_feats = ft.dfs(
    entityset=es, 
    target_dataframe_name="nums", 
    trans_primitives=f_primitives, 
    features_only=True,
    max_depth=1
)

# ft_feats = [<Feature: F_0>, <Feature: F_1>, <Feature: F_0 > F_1>, <Feature: F_1 > F_0>]

# Convert back into a format that we can use to compare with rust
c_feats = list(convert_features(ft_feats).values())

# Now run using Rust

# convert featuretools primitives to rust primitives
r_primitives = convert_primitives(f_primitives)

# convert dataframe to rust features
r_features = dataframe_to_features(es.dataframes[0])

# generate engineered features using Rust (create new features only)
r_derived_feats = generate_features(r_features, r_primitives)


a,b = compare_featuresets(c_feats, r_derived_feats)

print("=== Features generated by Featuretools, that were NOT generated by Rust ===")
print(a)
print()
print("=== Features generated by Rust, that were NOT generated by Featuretools ===")
print(b)


# Persist Rust Features to disk
Feature.save_features(r_derived_feats, "all_features.json")

Develop Guide

Install pipenv

pip install pipenv
pipenv install
pipenv shell

Ensure Cargo.toml is configured

[lib]
name = "rust_dfs"
crate-type = ["cdylib"]

Run maturin

maturin develop

Run main.rs

To run as a rust binary:

cargo run --no-default-features

Test

cargo test --no-default-features

Why --no-default-features? Because we are using pyo3 and is describe here:

Finally, don't forget that on MacOS the extension-module feature will cause cargo test to fail without the --no-default-features flag (see the FAQ).