Gators: A Lightning-Fast Data Preprocessing And Feature Engineering Python Library

Package
Quality
Documentation
Code style
Downloads
Community

📚 Full Documentation

What is Gators?

Gators is a library built on top of Polars, designed to streamline your entire ML workflow from raw data to production-ready models, leveraging Polars' blazing-fast multi-core processing.

Built by the PSP Data Team at PayPal, Gators makes data preprocessing and feature engineering both faster and simpler.

⚡ Key Features

• 🚀 Lightning Fast: Built on Polars for multi-core parallel processing
• 🔄 Unified API: Consistent sklearn-style .fit() and .transform() interface
• 📦 Production Ready: Deploy the same Python code from notebook to production
• 🎯 Comprehensive: 75+ preprocessing transformers covering every use case
• 🔗 Pipeline Support: Chain transformers seamlessly with the Pipeline class
• 🎓 Easy to Learn: If you know sklearn, you already know Gators

🛠️ What Can Gators Do?

🧹 Data Cleaning

Clean and prepare your data with powerful transformers:

• CastColumns - Convert column data types
• CorrelationFilter - Remove highly correlated features
• DropColumns - Remove specified columns
• DropConstantColumns - Remove columns with constant values
• DropDuplicateColumns - Remove duplicate columns
• DropDuplicateRows - Remove duplicate rows
• DropHighNaNRatio - Remove columns with high missing value ratio
• DropLowCardinality - Remove low cardinality columns
• HighCardinalityFilter - Filter high cardinality features
• OutlierFilter - Detect and filter outliers
• RenameColumns - Rename columns
• Replace - Replace values in data
• VarianceFilter - Remove low variance features

🔢 Categorical Encoding

Transform categorical variables with advanced encoding techniques:

• BinaryEncoder - Binary representation encoding
• CatBoostEncoder - CatBoost-style encoding
• CountEncoder - Frequency-based encoding
• LeaveOneOutEncoder - Leave-one-out encoding
• OneHotEncoder - Classic one-hot encoding
• OrdinalEncoder - Order-based encoding
• RareCategoryEncoder - Replace rare/infrequent categories by a single category
• TargetEncoder - Target-based encoding for supervised learning
• WOEEncoder - Weight of Evidence encoding

🎯 Feature Generation - Numeric

Create powerful numeric features: Mathematical Operations:

• DistanceFeatures - Calculate distance features
• HHIFeatures - Herfindahl–Hirschman Indexfeatures from groups of columns
• IsNull - Generate null indicator features
• MathFeatures - Apply mathematical operations (add, subtract, multiply, divide)
• RatioFeatures - Create ratio features between columns
• PlanRotationFeatures - Rotate features in feature space
• PolynomialFeatures - Generate polynomial combinations
• ScalarMathFeatures - Apply scalar operations
• WeigtedSumFeatures - Weighted sum of features from groups of columns

Aggregation & Statistics:

• GroupLagFeatures - Generate lag features by group
• GroupStatisticsFeatures - Scale features within groups
• RowStatisticsFeatures - Calculate statistics

Rule-based

• ComparisonFeatures - Generate comparison features
• ConditionFeatures - Create conditional features
• RuleFeatures - Apply custom business rules

📝 Feature Generation - String

Extract insights from text data:

• CharacterStatistics - Extract character-level statistics
• CombineFeatures - Combine string features
• Contains - Check if string contains pattern
• Endswith - Check if string ends with pattern
• ExtractSubstring - Extract substring from text
• InteractionFeatures - Generate string interaction features
• Length - Calculate string length
• Lower - Convert text to lowercase
• NGram - Generate n-gram features
• Occurrences - Count pattern occurrences
• PatternDetector - Detect patterns in text
• Split - Split strings
• SplitExtract - Split and extract from strings
• Startswith - Check if string starts with pattern
• Upper - Convert text to uppercase

📅 Feature Generation - DateTime

Unlock temporal patterns:

• BusinessTimeFeatures - Business hours/days calculations
• CyclicFeatures - Circular encoding for cyclical time features
• DiffFeatures - Calculate time differences
• DurationToDatetime - Convert duration to datetime
• HolidayFeatures - Detect and encode holidays
• OrdinalFeatures - Extract year, month, day, hour, etc.
• TimeBinFeatures - Bin times into categories
• TimeWindowFeatures - Generate time window features

🔄 Missing Value Imputation

Handle missing data intelligently:

• BooleanImputer - Impute boolean columns
• GroupByImputer - Group-based imputation strategies
• NumericImputer - Impute numeric columns (mean, median, mode, constant)
• StringImputer - Impute string columns (mode, constant)

📊 Discretization

Convert continuous variables into bins:

• CustomDiscretizer - Custom bin edges
• EqualLengthDiscretizer - Equal-width binning
• EqualSizeDiscretizer - Equal-frequency binning
• GeometricDiscretizer - Geometric progression binning
• KMeansDiscretizer - K-means clustering-based binning
• QuantileDiscretizer - Quantile-based binning
• TreeBasedDiscretizer - Decision tree-based binning

⚖️ Feature Scalers

Normalize your features:

• ArcsinSquarerootScaler - Arcsine square root transformation
• ArcsinhScaler - Inverse hyperbolic sine transformation
• BoxCox - Box-Cox power transformation
• Log1pScaler - Log1p scaling (log(1+x))
• MinmaxScaler - Min-max normalization
• PowerScaler - Power transformation
• StandardScaler - Standardization (z-score normalization)
• YeoJohnson - Yeo-Johnson power transformation

✨ Feature Selection

Select the most important features for your models:

• CorrelationSelector - Select features based on correlation analysis
• FeatureStabilitySelector - Select stable features across data splits
• InformationValueSelector - Select features by information value
• PermutationImportanceSelector - Select features by permutation importance
• PSIFilter - Filter features by Population Stability Index
• select_k_best_stable_features - Select K most stable features

🔗 Pipeline

Chain all transformers together:

• Pipeline - sklearn-compatible pipeline for chaining transformers

🚀 Quick Start

import polars as pl
from gators.data_cleaning import DropHighNaNRatio, VarianceFilter
from gators.encoders import OneHotEncoder
from gators.imputers import NumericImputer
from gators.scalers import StandardScaler
from gators.pipeline import Pipeline

# Load your data
X = pl.read_csv("data.csv")

# Build a preprocessing pipeline
pipeline = Pipeline(steps=[
    ('drop_nan', DropHighNaNRatio(max_ratio=0.5)),
    ('impute', NumericImputer(strategy='median')),
    ('variance', VarianceFilter(min_var=0.01)),
    ('encode', OneHotEncoder()),  # One-hot encode ALL the String or Categorical columns    
    ('scale', StandardScaler())
])

# Fit and transform
X_processed = pipeline.fit_transform(X)

# Serialize the pipeline with pickle/joblib for production deployment

📦 Installation

Requires Python 3.10 or higher.

pip3 install gators

Or install from source:

git clone https://github.com/paypal/gators.git
cd gators
pip3 install -e .    # Install in editable/development mode

📚 Documentation

For detailed documentation, tutorials, and API reference, visit:

https://paypal.github.io/gators/

🎯 Use Cases

Gators is perfect for:

• Fraud Detection - Extensive feature engineering for anomaly detection
• Risk Modeling - Create powerful predictive features
• Customer Analytics - Transform complex customer data
• Time Series - Rich datetime feature engineering
• NLP Tasks - String feature extraction and encoding

🏢 Used By

Gators powers ML pipelines at:

• PayPal (internal use)

🤝 Contributing

We welcome contributions! Please check out our contributing guidelines.

📄 License

Gators is licensed under the Apache License 2.0. See LICENSE file for details.

🙏 Credits

Developed by the PSP Data Team at PayPal.

Built by data scientists, for data scientists