sm-dag-compiler 1.0.0

Automatic SageMaker Pipeline Generation from DAG Specifications

SM-DAG-Compiler: Automatic SageMaker Pipeline Generation

Transform pipeline graphs into production-ready SageMaker pipelines automatically.

SM-DAG-Compiler is an intelligent pipeline generation system that automatically creates complete SageMaker pipelines from user-provided pipeline graphs. Simply define your ML workflow as a graph structure, and SM-DAG-Compiler handles all the complex SageMaker implementation details, dependency resolution, and configuration management automatically.

🚀 Quick Start

Installation

# Core installation
pip install sm-dag-compiler

# With ML frameworks
pip install sm-dag-compiler[pytorch,xgboost]

# Full installation with all features
pip install sm-dag-compiler[all]

30-Second Example

import sm_dag_compiler
from sm_dag_compiler.core.dag import PipelineDAG

# Create a simple DAG
dag = PipelineDAG(name="fraud-detection")
dag.add_node("data_loading", "CRADLE_DATA_LOADING")
dag.add_node("preprocessing", "TABULAR_PREPROCESSING") 
dag.add_node("training", "XGBOOST_TRAINING")
dag.add_edge("data_loading", "preprocessing")
dag.add_edge("preprocessing", "training")

# Compile to SageMaker pipeline automatically
pipeline = sm_dag_compiler.compile_dag(dag)
pipeline.start()  # Deploy and run!

Command Line Interface

# Generate a new project
sm-dag-compiler init --template xgboost --name fraud-detection

# Validate your DAG
sm-dag-compiler validate my_dag.py

# Compile to SageMaker pipeline
sm-dag-compiler compile my_dag.py --name my-pipeline --output pipeline.json

✨ Key Features

🎯 Graph-to-Pipeline Automation

• Input: Simple pipeline graph with step types and connections
• Output: Complete SageMaker pipeline with all dependencies resolved
• Magic: Intelligent analysis of graph structure with automatic step builder selection

⚡ 10x Faster Development

• Before: 2-4 weeks of manual SageMaker configuration
• After: 10-30 minutes from graph to working pipeline
• Result: 95% reduction in development time

🧠 Intelligent Dependency Resolution

• Automatic step connections and data flow
• Smart configuration matching and validation
• Type-safe specifications with compile-time checks
• Semantic compatibility analysis

🛡️ Production Ready

• Built-in quality gates and validation
• Enterprise governance and compliance
• Comprehensive error handling and debugging
• 98% complete with 1,650+ lines of complex code eliminated

📊 Proven Results

Based on production deployments across enterprise environments:

Component	Code Reduction	Lines Eliminated	Key Benefit
Processing Steps	60%	400+ lines	Automatic input/output resolution
Training Steps	60%	300+ lines	Intelligent hyperparameter handling
Model Steps	47%	380+ lines	Streamlined model creation
Registration Steps	66%	330+ lines	Simplified deployment workflows
Overall System	~55%	1,650+ lines	Intelligent automation

🏗️ Architecture

SM-DAG-Compiler follows a sophisticated layered architecture:

• 🎯 User Interface: Fluent API and Pipeline DAG for intuitive construction
• 🧠 Intelligence Layer: Smart proxies with automatic dependency resolution
• 🏗️ Orchestration: Pipeline assembler and compiler for DAG-to-template conversion
• 📚 Registry Management: Multi-context coordination with lifecycle management
• 🔗 Dependency Resolution: Intelligent matching with semantic compatibility
• 📋 Specification Layer: Comprehensive step definitions with quality gates

📚 Usage Examples

Basic Pipeline

from sm_dag_compiler import PipelineDAGCompiler
from sm_dag_compiler.core.dag import PipelineDAG

# Create DAG
dag = PipelineDAG()
dag.add_node("load_data", "DATA_LOADING_SPEC")
dag.add_node("train_model", "XGBOOST_TRAINING_SPEC")
dag.add_edge("load_data", "train_model")

# Compile with configuration
compiler = PipelineDAGCompiler(config_path="config.yaml")
pipeline = compiler.compile(dag, pipeline_name="my-ml-pipeline")

Advanced Configuration

from sm_dag_compiler import create_pipeline_from_dag

# Create pipeline with custom settings
pipeline = create_pipeline_from_dag(
    dag=my_dag,
    pipeline_name="advanced-pipeline",
    config_path="advanced_config.yaml",
    quality_requirements={
        "min_auc": 0.88,
        "max_training_time": "4 hours"
    }
)

Fluent API (Advanced)

from sm_dag_compiler.utils.fluent import Pipeline

# Natural language-like construction
pipeline = (Pipeline("fraud-detection")
    .load_data("s3://fraud-data/")
    .preprocess_with_defaults()
    .train_xgboost(max_depth=6, eta=0.3)
    .evaluate_performance()
    .deploy_if_threshold_met(min_auc=0.85))

🔧 Installation Options

Core Installation

pip install sm-dag-compiler

Includes basic DAG compilation and SageMaker integration.

Framework-Specific

pip install sm-dag-compiler[pytorch]    # PyTorch Lightning models
pip install sm-dag-compiler[xgboost]    # XGBoost training pipelines  
pip install sm-dag-compiler[nlp]        # NLP models and processing
pip install sm-dag-compiler[processing] # Advanced data processing

Development

pip install sm-dag-compiler[dev]        # Development tools
pip install sm-dag-compiler[docs]       # Documentation tools
pip install sm-dag-compiler[all]        # Everything included

🎯 Who Should Use SM-DAG-Compiler?

Data Scientists & ML Practitioners

• Focus on model development, not infrastructure complexity
• Rapid experimentation with 10x faster iteration
• Business-focused interface eliminates SageMaker expertise requirements

Platform Engineers & ML Engineers

• 60% less code to maintain and debug
• Specification-driven architecture prevents common errors
• Universal patterns enable faster team onboarding

Organizations

• Accelerated innovation with faster pipeline development
• Reduced technical debt through clean architecture
• Built-in governance and compliance frameworks

📖 Documentation

• Full Documentation - Complete guide and architecture
• API Reference - Detailed API documentation
• Examples - Ready-to-use pipeline blueprints
• Developer Guide - Contributing and extending AutoPipe

🤝 Contributing

We welcome contributions! See our Contributing Guide for details.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🔗 Links

• GitHub: https://github.com/TianpeiLuke/autopipe
• Issues: https://github.com/TianpeiLuke/autopipe/issues
• PyPI: https://pypi.org/project/autopipe/

---

SM-DAG-Compiler: Making SageMaker pipeline development 10x faster through intelligent automation. 🚀