Feature-Gen: A Robust Feature Engineering Framework

Feature-Gen is a Python-based library designed to simplify and optimize feature engineering for classification tasks. By integrating genetic algorithms, ensemble learning, and advanced feature transformations, Feature-Gen enables the discovery of feature subsets that maximize model performance while ensuring interpretability. It supports efficient processing through multithreading and multiprocessing, making it scalable for large datasets.

Key Features

• Automated Feature Engineering: Automatically identifies and optimizes feature subsets for classification tasks.
• Advanced Transformations: Includes transformations like logarithmic, square, cubic, sigmoid, and tanh to uncover complex, non-linear relationships.
• Multi-objective Optimization: Leverages the NSGA-II genetic algorithm to optimize both classification accuracy and feature subset size.
• Ensemble Learning Integration: Combines Logistic Regression, SVM, and XGBoost to ensure diverse model perspectives.
• Flexible Ensemble Methods: Supports strategies like Majority Voting, Weighted Averaging, and Greedy Selection for robust feature evaluation.
• Scalable Architecture: Uses multithreading and multiprocessing to handle large datasets efficiently.
• Extensive Validation: Tested on over 100 datasets, demonstrating robustness and adaptability across domains.

Installation

Install the library directly from PyPI:

pip install feature-gen

Getting Started

Example Usage

The following example demonstrates how to use Feature-Gen to perform feature engineering:

# Example Dataset
import pandas as pd
from feature_gen.feature_gen_master import FeatureGenMaster
from feature_gen.implementation.constants import EnsembleMethod
from sklearn.datasets import load_wine

# Load and prepare dataset
data = load_wine()
df = pd.DataFrame(data.data, columns=data.feature_names)
df['target'] = data.target

# Show the current features
print(df.columns)

# Initialize FeatureGenMaster
f_g = FeatureGenMaster(df, 'target')

# Start the feature engineering process
f_g.start(
    ensemble_methods=[EnsembleMethod.GREEDY, EnsembleMethod.WEIGHTED_AVERAGING]
)

# Retrieve results
print("Best New Features:", f_g.get_best_new_features())
print("Best Original Features:", f_g.get_best_original_features())
print("All Ensemble Methods Scores:", f_g.get_all_ensemble_methods_scores())

The following example demonstrates how to use Feature-Gen to perform feature engineering with full control over the library

import pandas as pd
from sklearn.datasets import load_wine

from feature_gen.feature_gen_master import FeatureGenMaster
from feature_gen.implementation.constants import EnsembleMethod

all_ensemble_methods = [
    EnsembleMethod.GREEDY,
    EnsembleMethod.WEIGHTED_MAJORITY_VOTING
]

# Load the Iris dataset
data = load_wine()

# Create a DataFrame with the features and target
df = pd.DataFrame(data.data, columns=data.feature_names)
df['target'] = data.target

# Show the current features
print(df.columns)

f_g = FeatureGenMaster(df, 'target', min_number_of_target_unique_values=20)

f_g.start(
    ensemble_methods=all_ensemble_methods,
    random_state=42,
    max_iter=50,
    C=1e5,
    solver='liblinear',
    gamma=1,
    n_components=100,
    sgd_loss='hinge',
    sgd_max_iter=1000,
    sgd_tol=1e-2,
    xgb_n_estimators=100,
    generations_num=2,
    bootstrap_samples_count=1,
    first_population_size=4
)

print('Best new features', f_g.get_best_new_features())
print('Best original features', f_g.get_best_original_features())
print('Best all features', f_g.get_all_best_features())
print("All ensemble methods scores", f_g.get_all_ensemble_methods_scores())

Framework Architecture

1. Micro-Step Genetic Algorithm

• Bootstrap Sampling: Generates three independent bootstrap samples to ensure robustness and diversity.
• Population Initialization: Creates a population of binary chromosomes representing feature subsets.
• Evaluation Metrics:
  • Maximizes the F1 score of an ensemble model (Logistic Regression, SVM, XGBoost).
  • Minimizes the number of selected features for interpretability.
• Genetic Operations:
  • Selection: Binary tournament selection to choose the best chromosomes.
  • Crossover: Uniform crossover for generating offspring.
  • Mutation: Flip-bit mutation to introduce diversity.
  • Population Update: Combines parents and offspring using NSGA-II for multi-objective optimization.

2. Macro-Step Genetic Algorithm

• Feature Aggregation: Combines feature subsets from the micro-step using union logic.
• Global Optimization: Refines the macro-feature set using NSGA-II.
• Final Feature Set: Outputs an optimal feature set balancing accuracy and interpretability.

Core Features and Functionality

• Multithreading and Multiprocessing:

  • Uses multithreading for concurrent evaluations and multiprocessing for parallelizing resource-intensive tasks.
  • Ensures scalability and efficient execution for large datasets.

• Built-in Ensemble Methods:

  • Supports flexible aggregation strategies like Majority Voting, Weighted Averaging, and Greedy Selection.

• Advanced Feature Transformations:

  • Includes transformations such as logarithmic, sigmoid, and tanh to capture non-linear relationships.

• Extensive Validation:

  • Tested on over 100 datasets, ensuring robustness and reliability.

Strengths

1. Robust Optimization: Balances competing objectives through the micro-macro genetic algorithm.
2. Integration of Transformations: Enhances predictive performance by uncovering non-linear relationships.
3. Generalizability: Ensures applicability across linear, boundary-based, and non-linear problems.
4. Interpretability: Achieves significant feature set reductions without compromising accuracy.

Future Directions

1. Scalability Enhancements: Expand support for distributed systems to handle even larger datasets.
2. Dynamic Transformation Framework: Introduce dataset-specific transformation selection for enhanced adaptability.
3. Additional Ensemble Methods: Integrate more aggregation strategies to improve robustness and flexibility.
4. User Interface: Develop visualization tools for better insights into feature engineering results.

Resources

• Documentation: Available on PyPI: Feature-Gen Documentation
• Source Code: Hosted on your development repository.

Contributing

Contributions are welcome! For major changes, please open an issue to discuss proposed updates. Ensure all pull requests align with the project's goals and include relevant tests.

License

This project is licensed under the MIT License. See the LICENSE file for more details.