🧬 Formed

Formed is a flexible framework for managing data, experiments, and workflows in both research and production environments. It provides a simple yet powerful DAG-based workflow system with automatic caching, dependency tracking, and seamless integration with popular ML tools.

Key Features

• 📊 DAG-based workflows: Define complex workflows as directed acyclic graphs with automatic dependency resolution
• 💾 Smart caching: Content-based automatic caching that detects code changes via AST analysis
• 🔧 Flexible configuration: Use Jsonnet/JSON for declarative workflow definitions with type safety
• 🔌 Rich integrations: Built-in support for PyTorch, 🤗 Transformers, MLflow, and more
• 🎯 Type-safe: Leverage Python type hints for automatic object construction and validation
• 📦 Extensible: Easy to extend with custom steps, formats, and organizers

Quick Example

Define reusable computation steps with automatic caching:

# mysteps.py
from collections.abc import Iterator
from formed import workflow

@workflow.step
def load_dataset(size: int) -> Iterator[int]:
    for i in range(size):
        yield i

@workflow.step
def square(dataset: Iterator[int]) -> Iterator[int]:
    for i in dataset:
        yield i * i

Connect steps in a workflow configuration:

// workflow.jsonnet
{
  steps: {
    dataset: {
      type: 'load_dataset',
      size: 10
    },
    results: {
      type: 'square',
      dataset: { type: 'ref', ref: 'dataset' }
    }
  }
}

Configure and run:

# formed.yml
workflow:
  organizer:
    type: filesystem

required_modules:
  - mysteps

formed workflow run workflow.jsonnet --execution-id my-experiment

Results are automatically cached - rerunning only executes changed steps!

Installation

pip install formed

With integrations:

pip install formed[mlflow]         # MLflow integration
pip install formed[torch]          # PyTorch integration
pip install formed[transformers]   # 🤗 Transformers integration
pip install formed[all]            # All integrations

Documentation

📖 Full documentation available here

• Quick Start - Get started in minutes
• Tutorials - Practical examples and use cases
• Guides - Deep dives into concepts and features
• API Reference - Complete API documentation

Why Formed?

Formed bridges the gap between experimental notebooks and production pipelines:

• Reproducible: Content-based caching ensures consistent results
• Iterative: Only re-execute what changed, speeding up development
• Collaborative: Declarative configs make workflows easy to share and review
• Production-ready: Same code works in research and deployment

Whether you're prototyping in Jupyter or deploying at scale, formed adapts to your workflow.

License

MIT License - see LICENSE file for details.