🌟 Keras Data Processor (KDP) - Powerful Data Preprocessing for TensorFlow 🌟

Provided and maintained by 🦄 UnicoLab

Transform your raw data into ML-ready features with just a few lines of code!

KDP provides a state-of-the-art preprocessing system built on TensorFlow Keras. It handles everything from feature normalization to advanced embedding techniques, making your ML pipelines faster, more robust, and easier to maintain. Built with ❤️ by 🦄 UnicoLab, it provides a clean, efficient, and extensible foundation for building sophisticated machine learning models for enterprise AI applications.

✨ Key Features

• 🚀 Efficient Single-Pass Processing: Process all features in one go, dramatically faster than alternatives
• 🧠 Distribution-Aware Encoding: Detects the shape of each numeric column and transforms it accordingly
• 👁️ Tabular Attention: Inter-feature and inter-sample attention, with a multi-resolution mode that treats numeric and categorical columns separately
• 🧮 Advanced Numerical Embeddings: Learned per-feature embeddings, and a global embedding across all numeric columns
• 🔀 Transformer Blocks: Stackable blocks over the categorical features or over everything
• 🔄 Feature-wise Mixture of Experts: Specialised expert networks per feature, routed automatically or by hand
• ⏱️ Time Series Features: Lags, rolling statistics, differencing, moving averages, wavelets, FFT, tsfresh statistics and calendar features
• 📝 Text, Date, Categorical, Cross and Passthrough features: Every column type in one model
• 🎛️ Auto-Configuration: Inspects your data and recommends the preprocessing to use
• 🔍 Feature Selection: Gated variable-selection layers on any subset of your features
• 📦 Production-Ready: Save and reload the preprocessor with your model as a single Keras artefact

🚀 Quick Installation

# Using pip
pip install kdp

# Using Poetry
poetry add kdp

📋 Simple Example

from kdp import PreprocessingModel, FeatureType

# Define your features
features_specs = {
    "age": FeatureType.FLOAT_NORMALIZED,
    "income": FeatureType.FLOAT_RESCALED,
    "occupation": FeatureType.STRING_CATEGORICAL,
    "description": FeatureType.TEXT
}

# Create and build the preprocessor
preprocessor = PreprocessingModel(
    path_data="data/my_data.csv",
    features_specs=features_specs,
    # Enable advanced features
    use_distribution_aware=True,
    tabular_attention=True
)
result = preprocessor.build_preprocessor()
model = result["model"]

# Use the preprocessor with your data
processed_features = model(input_data)

📚 Comprehensive Documentation

We've built an extensive documentation system to help you get the most from KDP:

Core Guides

• 🚀 Quick Start Guide - Get up and running in minutes
• 📊 Feature Processing - Learn about all supported feature types
• 🧙‍♂️ Auto-Configuration - Let KDP configure itself for your data

Advanced Topics

• 📈 Distribution-Aware Encoding - Smart handling of different distributions
• 👁️ Tabular Attention - Capture complex feature interactions
• 🔢 Advanced Numerical Embeddings - Rich representations for numbers
• 🤖 Transformer Blocks - Apply transformer architecture to tabular data
• 🎯 Feature Selection - Focus on what matters in your data
• 🧠 Feature-wise Mixture of Experts - Specialized processing per feature

Integration & Performance

• 🔗 Integration Guide - Use KDP with existing ML pipelines
• 🚀 Tabular Optimization - Supercharge your preprocessing
• 📈 Performance Tips - Handling large datasets efficiently

Background & Resources

• 💡 Motivation - Why we built KDP
• 🤝 Contributing - Help improve KDP

🖼️ Model Architecture

Your preprocessing pipeline is built as a Keras model that can be used independently or as the first layer of any model:

📊 Performance

KDP outperforms alternative preprocessing approaches, especially as data size increases:

🤝 Contributing

We welcome contributions! Please check out our Contributing Guide for guidelines on how to proceed.

💬 Join Our Community

Have questions or want to connect with other KDP users? Join us on Discord:

🛠️ Development Tools

KDP includes tools to help developers:

• Documentation Generation: Automatically generate API docs from docstrings
• Model Diagram Generation: Visualize model architectures with make generate_doc_content or run:

  python scripts/generate_model_diagrams.py
  

  This creates diagram images in docs/features/imgs/models/ for all feature types and configurations.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built with TensorFlow and Keras
• Inspired by modern deep learning research
• Community-driven development
• All contributors who help make KDP better

Built with ❤️ for the ML community by 🦄 UnicoLab.ai