PyDeevo: Advanced Neural Architecture Evolution Framework

PyDeevo is a comprehensive framework for neural network architecture search, hyperparameter optimization, and efficient training. It integrates modern deep learning tools into a cohesive, modular ecosystem:

• PyTorch Lightning: For structured, scalable deep learning
• Optuna: For efficient hyperparameter optimization
• PyGAD: For evolutionary architecture search
• Polars & DuckDB: For high-performance data processing and analytics
• Advanced Utils: For distributed training, profiling, and deployment

🔥 Key Features

🧬 Architecture Search & Optimization

• Evolutionary Architecture Search: Discover optimal network architectures using genetic algorithms
• Multi-level Optimization: Search both architecture space and hyperparameter space efficiently
• Flexible Encodings: Support for MLP and CNN architecture evolution

🚀 Performance & Scaling

• Distributed Training: Seamless scaling from single GPU to multi-node with FSDP/DDP
• Memory Optimization: Activation checkpointing, mixed precision, and other memory-saving techniques
• Batch Size Optimization: Automatically find optimal batch sizes for your hardware

📊 Data Processing & Analytics

• High-Performance DataFrames: Fast data processing with Polars integration
• SQL Analytics: Powerful SQL-based analytics with DuckDB
• Efficient Memory Management: Stream data from disk to model with minimal memory footprint
• ETL Pipelines: Build data pipelines with SQL and Polars transformations
• Analytical Workflows: Create and execute complex analytical pipelines

🔍 Profiling & Benchmarking

• Model Analysis: Profile inference time, memory usage, and computational complexity (FLOPs)
• Architecture Comparison: Benchmark different architectures with comprehensive metrics
• Training Insights: Monitor performance during training with detailed profiling

📦 Deployment & Inference

• Model Export: Export to ONNX, TorchScript, and SafeTensors formats
• Inference Optimization: Quantization and other performance optimizations
• Security: Protection against deserialization vulnerabilities

🛠️ Installation

pip install pydeevo

Or install from source:

git clone https://github.com/example/pydeevo.git
cd pydeevo
pip install -e .

🚀 Quick Start

Evolutionary Architecture Search

from pydeevo import PyDeevo
from pydeevo.models.base import FlexibleModule

# Initialize PyDeevo
pydeevo = PyDeevo(base_dir="./output")

# Evolutionary search for optimal architecture
result = pydeevo.evolve_architecture(
    input_shape=input_shape,
    output_size=output_size,
    train_loader=train_loader,
    val_loader=val_loader,
    network_type="mlp",
    population_size=20,
    num_generations=10
)

# Get best architecture and train final model
best_architecture = result["best_architecture"]
best_hyperparams = result["best_hyperparameters"]
model = FlexibleModule(architecture=best_architecture, **best_hyperparams)

Data Processing with Polars

from pydeevo.utils.data import (
    PolarsDataProcessor, 
    DatasetBuilder, 
    normalize_features, 
    create_polynomial_features
)

# Create processor and builder
processor = PolarsDataProcessor(cache_dir="./data_cache")
builder = DatasetBuilder(processor)

# Define preprocessing steps
preprocessing_steps = [
    normalize_features(feature_cols, method='z-score'),
    create_polynomial_features(['feature1', 'feature2'], degree=2)
]

# Create PyTorch datasets
train_dataset, val_dataset, test_dataset = builder.from_file(
    file_path="data.csv",
    feature_cols=feature_cols,
    target_col="target",
    preprocessing_steps=preprocessing_steps,
    cache_key="processed_data"
)

SQL Analytics with DuckDB

from pydeevo.utils.database import (
    AnalyticalDataManager,
    sql_to_pytorch_dataset
)

# Set up analytics manager
analytics = AnalyticalDataManager("analytics.duckdb")

# Load data
analytics.load_data("customers.csv", "customers")
analytics.load_data("transactions.csv", "transactions")

# Run analytical query
results = analytics.execute_sql("""
    SELECT c.segment, AVG(t.amount) as avg_spend
    FROM customers c
    JOIN transactions t ON c.customer_id = t.customer_id
    GROUP BY c.segment
""", cache_key="segment_analysis")

# Create PyTorch dataset directly from SQL
dataset = sql_to_pytorch_dataset(
    db_path="analytics.duckdb",
    query="SELECT * FROM prepared_features",
    feature_cols=["feature1", "feature2", "feature3"],
    target_col="target"
)

Distributed Training

from pydeevo.utils.distributed import DistributedTrainingHelper, MemoryOptimization
import torch.nn as nn

# Set up distributed training
dist_helper = DistributedTrainingHelper(
    precision='16-mixed',
    devices='auto',
    strategy='auto'
)

# Apply memory optimizations
model = MemoryOptimization.optimize_memory_usage(
    model, 
    use_channels_last=True,
    use_compile=True
)

# Set up Fabric
fabric = dist_helper.setup_fabric(transformer_modules=[nn.Linear, nn.Conv2d])
model, optimizer = dist_helper.setup_model_and_optimizer(model, optimizer_fn)

Model Profiling and Export

from pydeevo.utils.profiling import ModelProfiler, FlopsCalculator
from pydeevo.utils.export import ModelExporter

# Profile model performance
profiler = ModelProfiler(model, input_shape=(batch_size, *input_dim))
memory_stats = profiler.profile_memory_usage()
inference_stats = profiler.profile_inference_time()
flops = FlopsCalculator(model, input_shape).calculate_flops()

# Export for deployment
exporter = ModelExporter(model, input_shape=(1, *input_dim))
export_paths = exporter.export_all_formats()  # ONNX, TorchScript, SafeTensors

🧩 Framework Components

PyDeevo is built with modularity in mind. Major components include:

models

• FlexibleModule: Adaptable PyTorch Lightning module for MLPs
• CNNModule: Adaptable module for convolutional networks

evolution

• Architecture encodings for genetic algorithms
• Multi-level optimization strategies
• Evolutionary search implementations

optimization

• Hyperparameter optimization with Optuna
• Search space definitions and strategies

training

• Extended Lightning training utilities
• Custom callbacks for visualization and monitoring

utils

• data: High-performance data processing with Polars
• database: SQL analytics with DuckDB integration
• distributed: Utilities for distributed training and memory optimization
• profiling: Performance analysis and benchmarking
• export: Model export and deployment optimization

🔬 Advanced Use Cases

PyDeevo supports advanced deep learning workflows:

1. Large Model Training: Train models too large for a single GPU using memory optimization techniques and model sharding
2. Complex Feature Engineering: Create sophisticated features with Polars before training
3. SQL-Based Data Pipelines: Use DuckDB for efficient data preparation and feature engineering
4. Production Deployment: Export optimized models with quantization for inference
5. Performance Optimization: Profile and optimize models for specific hardware targets
6. End-to-End ML Workflows: From data ingestion to model training and deployment in a single framework

📚 Example Gallery

See the examples directory for full implementations:

• mnist_example.py: Basic architecture search for MNIST
• cifar10_example.py: CNN architecture search for CIFAR-10
• hyperopt_example.py: Hyperparameter optimization with fixed architecture
• distributed_example.py: Distributed training with Lightning Fabric
• polars_example.py: Data processing with Polars integration
• duckdb_example.py: SQL analytics with DuckDB integration

📋 Requirements

• Python 3.8+
• PyTorch 1.9+
• PyTorch Lightning 2.0+
• Optuna 3.0+
• PyGAD 2.18+
• Polars 0.19+
• DuckDB 0.8+
• NumPy 1.20+
• Matplotlib 3.5+

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.