Skip to main content

LinkedIn Contributors Stargazers MIT License Downloads Month Downloads


Atlantic - Automated Data Preprocessing Framework for Supervised Machine Learning

Framework Contextualization

The Atlantic project constitutes a comprehensive and objective approach to simplify and automate data processing through the integration and validated application of various preprocessing mechanisms, ranging from feature engineering, automated feature selection, multiple encoding versions and null imputation methods. The optimization methodology of this framework follows an evaluation structured in tree-based model ensembles.

This project aims at providing the following application capabilities:

  • General applicability on tabular datasets: The developed preprocessing procedures are applicable on multiple domains associated with Supervised Machine Learning, regardless of the properties or specifications of the data.

  • Automated treatment of tabular data associated with predictive analysis: It implements a global and carefully validated data processing based on the characteristics of the data input columns.

  • Robustness and improvement of predictive results: The implementation of the atlantic automated data preprocessing pipeline aims at improving predictive performance directly associated with the processing methods implemented based on the data properties.

Main Development Tools

Major frameworks used to build this project:

Framework Architecture

Atlantic pipeline: validation and preparation, train/validation split with optional H2O feature selection, selection of encoding, imputation and VIF filtering by validation score, final fit on the training split, transformation of new data, and persistence

The Atlantic pipeline runs in 6 main stages:

1. Validate & prepare

  • Task detection: the task is inferred from the target column.

    Target type Task Evaluation metric
    Numeric Regression MAE
    Other, with 2 classes Binary classification Precision
    Other, with more than 2 classes Multiclass classification F1
  • Data preparation:

    • Rows with no target value are dropped.
    • Boolean columns become 0/1 features.
    • Each datetime column is expanded into nine calendar features.
  • Column removal: high-null, constant and identifier-like columns are removed. Every removal is recorded in dropped_columns.

2. Split & select features

  • Split: the data is divided into a train and a validation split. Classification splits are stratified.
  • Feature selection (optional): when relevance < 1.0, only the features carrying the requested share of H2O AutoML variable importance are kept. This requires atlantic[h2o].

3. Select by validation score

Preprocessing steps are chosen one at a time:

Step Candidates Applied when
Encoding 4 combinations: Standard or MinMax scaler × IFrequency or Label encoder Always
Imputation Simple (mean), KNN, Iterative The data has missing values
VIF filtering Iterative removal of features above vif_ratio It improves the validation score
  • Evaluation: each candidate is fitted on the train split and scored on the validation split, using tuned RandomForest, ExtraTrees and XGBoost models.
  • Trial budget: the number of tuning trials adapts to the data size and to optimization_level.

4. Fit final transformers

The chosen scaler, encoder and imputer are refitted on the training rows only, so validation rows never influence them.

5. Transform new data

data_processing replays the same steps on any future frame:

  • Frames missing a fitted column are rejected with a clear error.
  • Unseen categories and missing values are handled.

6. Inspect & persist

  • Inspect: the pipeline's decisions are exposed as enc_method, imp_method, cols and dropped_columns.
  • Persist: the fitted pipeline can be saved with save and reloaded with Atlantic.load.

Where to get it

Binary installer for the latest released version is available at the Python Package Index (PyPI).

Installation

To install this package from Pypi repository run the following command:

pip install atlantic

Usage Examples

1. Atlantic - Automated Data Preprocessing Pipeline

Import the package, load a dataset, split it, and define your target column name. Customize the fit_processing method with the following parameters:

Parameter Description Default
split_ratio Train/Validation split ratio for preprocessing evaluation (0.5 - 0.98) 0.75
relevance Minimum feature importance percentage for H2O AutoML selection (0.4 - 1.0). 1.0 skips H2O feature selection; lower values require atlantic[h2o] 0.99
h2o_fs_models Number of models for H2O AutoML feature selection 7
encoding_fs Encode categorical features before H2O selection True
vif_ratio Variance Inflation Factor threshold (3 - 30) 10.0
optimization_level Optimization intensity: "fast", "balanced", "thorough" "balanced"

The Atlantic constructor additionally accepts:

Parameter Description Default
random_state Seed for the internal train/validation split and every model evaluation 42
n_jobs Parallelism of the evaluation models. None keeps the tree ensembles single-threaded, -1 uses all cores (same results, faster on mid-sized data) None
verbosity 0 silent, 1 selection summary, 2 detailed 1

Once fitted, use data_processing to transform any future dataframes with the same structure.

Task type. It is inferred from the target column: a numeric target is a regression task, anything else (strings, categories, booleans) a classification task. Integer-coded class labels such as 0/1 are therefore treated as regression (a warning is raised) - cast them first: data["Target_Column"] = data["Target_Column"].astype(str).

import pandas as pd
from sklearn.model_selection import train_test_split
from atlantic.pipeline import Atlantic
import warnings
warnings.filterwarnings("ignore", category=Warning)
    
data = pd.read_csv('csv_directory_path', encoding='latin', delimiter=',')

train, test = train_test_split(data, train_size=0.8)
test, future_data = train_test_split(test, train_size=0.6)

train = train.reset_index(drop=True)
test = test.reset_index(drop=True)
future_data = future_data.reset_index(drop=True)
future_data.drop(columns=["Target_Column"], inplace=True)

### Fit Data Processing
atl = Atlantic(X=train, target="Target_Column")    

atl.fit_processing(
    split_ratio=0.75,
    relevance=0.99,      # H2O feature selection (needs atlantic[h2o]); 1.0 skips it
    h2o_fs_models=7,
    vif_ratio=10.0,
    optimization_level="balanced"
)

### Transform Data Processing
train = atl.data_processing(X=train)
test = atl.data_processing(X=test)
future_data = atl.data_processing(X=future_data)

### Inspect what the pipeline decided
print(atl.enc_method, atl.imp_method)   # selected encoding version / imputation method
print(atl.cols)                         # feature columns kept
print(atl.dropped_columns)              # columns removed, grouped by reason

### Export the fitted pipeline (built-in helper — uses pickle under the hood)
### The training data is not stored in the file. Only load pipeline files you trust.
atl.save('fit_atl.pkl')
loaded_atl = Atlantic.load('fit_atl.pkl')

# Or serialize manually with dill (bundled dependency):
# import dill as pickle
# with open('fit_atl.pkl', 'wb') as output:
#     pickle.dump(atl, output)

2. Atlantic - Builder Pattern (Granular Control)

For fine-grained control over preprocessing steps, use the AtlanticBuilder fluent interface:

from sklearn.model_selection import train_test_split
from atlantic.pipeline import AtlanticBuilder

train, test = train_test_split(data, train_size=0.8)
train = train.reset_index(drop=True)
test = test.reset_index(drop=True)

### Build Custom Pipeline
pipeline = (AtlanticBuilder()
    .with_date_engineering(enabled=True, drop=True)
    .with_null_removal(threshold=0.90)
    .with_feature_selection(
        method="h2o",
        relevance=0.95,
        h2o_models=10,
        encoding_fs=True
    )
    .with_encoding(
        scaler="standard",
        encoder="ifrequency",
        auto_select=True
    )
    .with_imputation(
        method="knn",
        auto_select=True
    )
    .with_vif_filtering(threshold=10.0)
    .with_optimization(optimization_level="balanced")
    .build()
)

### Fit and Transform
train_processed = pipeline.fit_transform(train, target="Target_Column")
test_processed = pipeline.transform(test)

### Inspect the fitted result and persist the pipeline
print(pipeline.atlantic.enc_method, pipeline.atlantic.dropped_columns)
pipeline.save('fit_pipeline.pkl')
# from atlantic.pipeline import AtlanticPipeline
# pipeline = AtlanticPipeline.load('fit_pipeline.pkl')

Notes on the builder options:

  • with_date_engineering(enabled, drop) - enabled=False leaves datetime columns out of the features; drop=False keeps the original datetime columns in the transformed output next to the generated features.
  • with_encoding(..., auto_select=True) / with_imputation(..., auto_select=True) select the best option by validation performance and ignore the explicit scaler / encoder / method; with auto_select=False they are applied as given.
  • with_optimization(optimization_level, random_state, split_ratio, n_jobs, verbosity) controls the evaluation budget, reproducibility and console output.

Builder Configuration Presets

Configuration Use Case Key Settings
Fast Quick prototyping h2o_models=3, method="simple", optimization_level="fast"
Balanced General purpose Default settings
Thorough Best results h2o_models=15, method="iterative", optimization_level="thorough"
High-Null Missing data >20% threshold=0.80, scaler="robust", method="iterative"
No-H2O Skip H2O selection method="none", VIF filtering only
# Fast Prototyping
fast_pipeline = (AtlanticBuilder()
    .with_feature_selection(method="h2o", relevance=0.85, h2o_models=3)
    .with_encoding(scaler="minmax", encoder="label", auto_select=False)
    .with_imputation(method="simple", auto_select=False)
    .with_optimization(optimization_level="fast")
    .build()
)

# Thorough Optimization  
thorough_pipeline = (AtlanticBuilder()
    .with_feature_selection(method="h2o", relevance=0.98, h2o_models=15)
    .with_encoding(auto_select=True)
    .with_imputation(method="iterative", auto_select=True)
    .with_vif_filtering(threshold=8.0)
    .with_optimization(optimization_level="thorough")
    .build()
)

# High-Null Data
high_null_pipeline = (AtlanticBuilder()
    .with_null_removal(threshold=0.80)
    .with_encoding(scaler="robust")
    .with_imputation(method="iterative", auto_select=True)
    .build()
)

3. Atlantic - Preprocessing Components

3.1 Encoding Methods

Encode categorical variables into numerical format. Choose from label encoding (ordinal mapping) or inverse frequency encoding (IDF-based weights) — both preserve the column layout, so you can assign the result back in place. AutoOneHotEncoder expands each categorical column into several binary columns, so it returns a wider frame and must be handled separately (see the note below).

import pandas as pd
from sklearn.model_selection import train_test_split 
from atlantic.preprocessing import AutoLabelEncoder, AutoIFrequencyEncoder, AutoOneHotEncoder

train, test = train_test_split(data, train_size=0.8)
train = train.reset_index(drop=True)
test = test.reset_index(drop=True)

from atlantic.utils import get_categorical_columns, get_numeric_columns

target = "Target_Column"
# Detects object, category and string dtype columns on any pandas version.
cat_cols = get_categorical_columns(data, exclude=[target])

### Create Encoder (label or inverse-frequency — same-shape output)
encoder = AutoLabelEncoder()
# encoder = AutoIFrequencyEncoder()

### Fit and Transform
encoder.fit(train[cat_cols])
train[cat_cols] = encoder.transform(train[cat_cols])
test[cat_cols] = encoder.transform(test[cat_cols])

### Inverse Transform (if needed)
train[cat_cols] = encoder.inverse_transform(train[cat_cols])

### One-hot encoding returns a WIDER frame — do not assign back into cat_cols:
# ohe = AutoOneHotEncoder()
# ohe.fit(train[cat_cols])
# train = ohe.transform(train)   # original cat_cols are replaced by binary columns
# test = ohe.transform(test)

3.2 Scalers

Normalize numerical features to improve model convergence. Standard scaler for normal distributions, MinMax for bounded ranges, or Robust scaler for data with outliers.

from atlantic.preprocessing import AutoStandardScaler, AutoMinMaxScaler, AutoRobustScaler
from atlantic.utils import get_numeric_columns

target = "Target_Column"
# Exclude the target: scaling the label column would distort it for regression.
num_cols = get_numeric_columns(train, exclude=[target])

### Create Scaler (choose one)
scaler = AutoStandardScaler()  # Zero mean, unit variance
# scaler = AutoMinMaxScaler()  # Scale to [0, 1]
# scaler = AutoRobustScaler()  # Median/IQR based, outlier-resistant

### Fit and Transform
scaler.fit(train[num_cols])
train[num_cols] = scaler.transform(train[num_cols])
test[num_cols] = scaler.transform(test[num_cols])

3.3 Imputers

Handle missing values using statistical methods. Simple imputation for speed, KNN for local patterns, or iterative modeling for complex dependencies.

from atlantic.preprocessing import AutoSimpleImputer, AutoKNNImputer, AutoIterativeImputer

### Create Imputer (choose one)
imputer = AutoSimpleImputer(strategy='mean', target=target)
# imputer = AutoKNNImputer(n_neighbors=5, target=target)
# imputer = AutoIterativeImputer(max_iter=10, target=target)

### Fit and Transform
imputer.fit(train)
train = imputer.transform(train)
test = imputer.transform(test)

3.4 Feature Selection

Reduce dimensionality by removing redundant or low-importance features. VIF removes multicollinear features, while H2O AutoML identifies the most predictive ones.

Note: VIFFeatureSelector evaluates numeric, null-free columns — impute nulls first (it raises ValidationError on nulls). Non-numeric columns are passed through untouched. vif_threshold must be within [3.0, 30.0]. The H2OFeatureSelector example below requires a running H2O cluster (Java); install the optional dependency with pip install atlantic[h2o].

from atlantic.feature_selection import VIFFeatureSelector, H2OFeatureSelector

### VIF-based Selection (expects encoded, imputed, numeric features)
vif_selector = VIFFeatureSelector(target=target, vif_threshold=10.0)
vif_selector.fit(train)
train = vif_selector.transform(train)
test = vif_selector.transform(test)

### H2O AutoML Selection
h2o_selector = H2OFeatureSelector(target=target, relevance=0.95, max_models=7)
h2o_selector.fit(train)
train = h2o_selector.transform(train)
test = h2o_selector.transform(test)

Check out the Custom Preprocessing for detailed implementations of all preprocessing methods.

For interactive and step by step guideline walkthrough, feel free to explore the Jupyter Notebooks

Citation

@article{SANTOS2023100532,
  author = {Luis Fernando Santos and Luis Ferreira},
  title = {Atlantic - Automated data preprocessing framework for supervised machine learning},
  journal = {Software Impacts},
  volume = {17},
  year = {2023},
  issn = {2665-9638},
  doi = {http://dx.doi.org/10.1016/j.simpa.2023.100532},
  url = {https://www.sciencedirect.com/science/article/pii/S2665963823000696}
}

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Luis Santos - LinkedIn

Metadata

Release files for atlantic 2.0.31

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for atlantic 2.0.31
File Interpreter ABI Platform Actions
atlantic-2.0.31-py3-none-any.whl Python 3 none any Details for atlantic-2.0.31-py3-none-any.whl

Release files / atlantic-2.0.31-py3-none-any.whl

Download URL atlantic-2.0.31-py3-none-any.whl
Size 84.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8516bb3bdeacd82ba60396383266cedf41bea799f9a5de96c3b66a569e2526f0
BLAKE2b-256 checksum
How to use checksums
780ad0190337216c486e28c80342a9e9fcf080c0eb35e2f2551112426f00eb35
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.18

Security for atlantic 2.0.31

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page