🌟 Keras Data Processor (KDP) - Powerful Data Preprocessing for TensorFlow 🌟
Provided and maintained by 🦄 UnicoLab
Transform your raw data into ML-ready features with just a few lines of code!
KDP provides a state-of-the-art preprocessing system built on TensorFlow Keras. It handles everything from feature normalization to advanced embedding techniques, making your ML pipelines faster, more robust, and easier to maintain. Built with ❤️ by 🦄 UnicoLab, it provides a clean, efficient, and extensible foundation for building sophisticated machine learning models for enterprise AI applications.
✨ Key Features
- 🚀 Efficient Single-Pass Processing: Process all features in one go, dramatically faster than alternatives
- 🧠 Distribution-Aware Encoding: Detects the shape of each numeric column and transforms it accordingly
- 👁️ Tabular Attention: Inter-feature and inter-sample attention, with a multi-resolution mode that treats numeric and categorical columns separately
- 🧮 Advanced Numerical Embeddings: Learned per-feature embeddings, and a global embedding across all numeric columns
- 🔀 Transformer Blocks: Stackable blocks over the categorical features or over everything
- 🔄 Feature-wise Mixture of Experts: Specialised expert networks per feature, routed automatically or by hand
- ⏱️ Time Series Features: Lags, rolling statistics, differencing, moving averages, wavelets, FFT, tsfresh statistics and calendar features
- 📝 Text, Date, Categorical, Cross and Passthrough features: Every column type in one model
- 🎛️ Auto-Configuration: Inspects your data and recommends the preprocessing to use
- 🔍 Feature Selection: Gated variable-selection layers on any subset of your features
- 📦 Production-Ready: Save and reload the preprocessor with your model as a single Keras artefact
🚀 Quick Installation
# Using pip
pip install kdp
# Using Poetry
poetry add kdp
📋 Simple Example
from kdp import PreprocessingModel, FeatureType
# Define your features
features_specs = {
"age": FeatureType.FLOAT_NORMALIZED,
"income": FeatureType.FLOAT_RESCALED,
"occupation": FeatureType.STRING_CATEGORICAL,
"description": FeatureType.TEXT
}
# Create and build the preprocessor
preprocessor = PreprocessingModel(
path_data="data/my_data.csv",
features_specs=features_specs,
# Enable advanced features
use_distribution_aware=True,
tabular_attention=True
)
result = preprocessor.build_preprocessor()
model = result["model"]
# Use the preprocessor with your data
processed_features = model(input_data)
📚 Comprehensive Documentation
We've built an extensive documentation system to help you get the most from KDP:
Core Guides
- 🚀 Quick Start Guide - Get up and running in minutes
- 📊 Feature Processing - Learn about all supported feature types
- 🧙♂️ Auto-Configuration - Let KDP configure itself for your data
Advanced Topics
- 📈 Distribution-Aware Encoding - Smart handling of different distributions
- 👁️ Tabular Attention - Capture complex feature interactions
- 🔢 Advanced Numerical Embeddings - Rich representations for numbers
- 🤖 Transformer Blocks - Apply transformer architecture to tabular data
- 🎯 Feature Selection - Focus on what matters in your data
- 🧠 Feature-wise Mixture of Experts - Specialized processing per feature
Integration & Performance
- 🔗 Integration Guide - Use KDP with existing ML pipelines
- 🚀 Tabular Optimization - Supercharge your preprocessing
- 📈 Performance Tips - Handling large datasets efficiently
Background & Resources
- 💡 Motivation - Why we built KDP
- 🤝 Contributing - Help improve KDP
🖼️ Model Architecture
Your preprocessing pipeline is built as a Keras model that can be used independently or as the first layer of any model:
📊 Performance
KDP outperforms alternative preprocessing approaches, especially as data size increases:
🤝 Contributing
We welcome contributions! Please check out our Contributing Guide for guidelines on how to proceed.
💬 Join Our Community
Have questions or want to connect with other KDP users? Join us on Discord:
🛠️ Development Tools
KDP includes tools to help developers:
- Documentation Generation: Automatically generate API docs from docstrings
- Model Diagram Generation: Visualize model architectures with
make generate_doc_contentor run:python scripts/generate_model_diagrams.pyThis creates diagram images indocs/features/imgs/models/for all feature types and configurations.
📄 License
This project is licensed under the MIT License - see the LICENSE file for details.
🙏 Acknowledgments
- Built with TensorFlow and Keras
- Inspired by modern deep learning research
- Community-driven development
- All contributors who help make KDP better
Built with ❤️ for the ML community by 🦄 UnicoLab.ai
Metadata
Release files for kdp 1.13.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| kdp-1.13.0.tar.gz | 253.6 kB | Without provenance | Details for kdp-1.13.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| kdp-1.13.0-py3-none-any.whl | Python 3 | none | any | Details for kdp-1.13.0-py3-none-any.whl |
Total release size: 429.1 kB
Release files / kdp-1.13.0.tar.gz
| Download URL | kdp-1.13.0.tar.gz |
|---|---|
| Size | 253.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1d80dcbce4c2e50ee962a6643cd7a281364778373a3f591722c0f20e0e6e010a
|
|
BLAKE2b-256 checksum How to use checksums |
844bc369c56ebcbf5f9a37fda648940ea5f851f85249e1c667acedcc138e1e01
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.2 CPython/3.11.16 Linux/6.17.0-1022-azure
|
Release files / kdp-1.13.0-py3-none-any.whl
| Download URL | kdp-1.13.0-py3-none-any.whl |
|---|---|
| Size | 175.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
002afcf4132194d30c1525110cac7093b3fbfee220db24d600f85dc0f2f4027c
|
|
BLAKE2b-256 checksum How to use checksums |
fdd982e692427f99b5dc6f1c10544d07de65782c1b721f1d12d21a2897ed4121
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.2 CPython/3.11.16 Linux/6.17.0-1022-azure
|
Security for kdp 1.13.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.