forge-flow 0.1.0

Engineering-First ML Library for building robust, production-grade machine learning pipelines

ForgeFlow

Production-grade ML library for building reliable, type-safe machine learning pipelines.

ForgeFlow is designed for engineering teams who need robust data validation, reproducible feature engineering, and production-ready ML infrastructure. Built with strict type safety, immutability, and defensive programming principles.

Why ForgeFlow?

Modern ML systems fail in production due to data quality issues, schema drift, and unreliable pipelines. ForgeFlow addresses these challenges with:

• Strict Schema Validation - Pydantic-based contracts that fail fast on invalid data
• Immutable Operations - Pure functions that prevent side effects and ensure reproducibility
• Quality Gates - Automated checks that prevent bad data from reaching production
• Drift Detection - Statistical monitoring to catch distribution shifts early
• Type Safety - 100% type-hinted codebase for better IDE support and fewer runtime errors

Installation

pip install forge-flow

Optional dependencies:

pip install forge-flow[serving]    # Redis-based feature store
pip install forge-flow[database]   # SQL database connectors
pip install forge-flow[all]        # Everything

Quick Start

ForgeFlow follows a simple pipeline pattern: Ingest → Validate → Transform → Serve

1. Define Your Schema

Schemas enforce data contracts and catch issues early:

from forge_flow.schemas import StrictSchema
from pydantic import Field

class Transaction(StrictSchema):
    user_id: int = Field(gt=0)
    amount: float = Field(gt=0, le=1000000)
    timestamp: datetime

2. Validate Data

Automatic validation with Dead Letter Queue for invalid records:

from forge_flow.schemas import SchemaValidator

validator = SchemaValidator(Transaction)
clean_data = validator.validate(raw_data, raise_on_error=False)

3. Engineer Features

Immutable transformations with built-in quality checks:

from forge_flow.features import DataCleaner, FeatureEngineer

cleaner = DataCleaner()
cleaned = cleaner.handle_nulls(clean_data)

engineer = FeatureEngineer()
features = engineer.add_rolling_features(
    cleaned,
    group_col='user_id',
    value_col='amount',
    windows=[7, 30]
)

4. Quality Gates

Prevent bad data from reaching production:

from forge_flow.features import QualityGate

gate = QualityGate(max_null_rate=0.05, min_rows=1000)
gate.validate(features)  # Raises error if quality fails

Core Features

Data Ingestion

• File Connectors - Read from local, S3, GCS, Azure (Parquet, CSV, JSON)
• API Connectors - Resilient HTTP clients with retries and rate limiting
• Database Connectors - SQLAlchemy-based with incremental loading

Feature Engineering

• Data Cleaning - Null handling, outlier detection, type casting
• Transformations - Rolling windows, lag features, encoding, normalization
• Pipelines - Reproducible workflows with Pydantic configuration

Production Features

• Online Store - Redis-based sub-millisecond feature serving
• Drift Detection - KS test for numeric, PSI for categorical features
• Structured Logging - Full observability with structlog

Design Principles

ForgeFlow is built for production systems, not notebooks:

1. Fail Fast - Validate early, catch errors at ingestion boundaries
2. Immutability - Operations return new objects, preventing side effects
3. Type Safety - Comprehensive type hints for better tooling and fewer bugs
4. Explicit Over Implicit - No magic, clear data flows
5. Testability - Pure functions and dependency injection throughout

Documentation

• Examples - Practical usage patterns and integration guides
• Contributing - Development workflow and standards

Use Cases

ForgeFlow excels at:

• Fraud Detection - Real-time feature serving with drift monitoring
• Recommendation Systems - Reproducible feature pipelines at scale
• Credit Risk - Audit trails and regulatory compliance
• Predictive Maintenance - Time-series feature engineering

Requirements

• Python 3.10+
• Core: pandas, numpy, pydantic, structlog
• Optional: redis (serving), sqlalchemy (database)

License

MIT License - see LICENSE for details.

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions

---

Built with ❤️ for production ML teams who value reliability over convenience.