Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

Flyte 2 SDK 🚀

The next-generation Python SDK for scalable, distributed workflows

Version Python License

⚡ Pure Python workflows • 🔄 Async-first parallelism • 🛠️ Zero DSL constraints • 📊 Sub-task observability

What is Flyte 2?

Flyte 2 represents a fundamental shift from constrained domain-specific languages to pure Python workflows. Write data pipelines, ML training jobs, and distributed compute exactly like you write Python—because it is Python.

import flyte

env = flyte.TaskEnvironment("hello_world")

@env.task
async def process_data(data: list[str]) -> list[str]:
    # Use any Python construct: loops, conditionals, try/except
    results = []
    for item in data:
        if len(item) > 5:
            results.append(await transform_item(item))
    return results

@env.task
async def transform_item(item: str) -> str:
    return f"processed: {item.upper()}"

if __name__ == "__main__":
    flyte.init()
    result = flyte.run(process_data, data=["hello", "world", "flyte"])

🌟 Why Flyte 2?

No More Workflow DSL

  • ❌ @workflow decorators with Python subset limitations
  • ✅ Pure Python: loops, conditionals, error handling, dynamic structures

Async-First Parallelism

  • ❌ Custom map() functions and workflow-specific parallel constructs
  • ✅ Native asyncio: await asyncio.gather() for distributed parallel execution

True Container Reusability

  • ❌ Cold container starts for every task
  • ✅ Millisecond scheduling with warm, reusable container pools

Fine-Grained Observability

  • ❌ Task-level logging only
  • ✅ Function-level tracing with @flyte.trace for sub-task checkpoints

🚀 Quick Start

Installation

# Install uv package manager
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create virtual environment
uv venv && source .venv/bin/activate

# Install Flyte 2 (beta)
uv pip install --prerelease=allow flyte

Your First Workflow

# hello.py
# /// script
# requires-python = ">=3.10"
# dependencies = ["flyte>=0.2.0"]
# ///

import flyte

env = flyte.TaskEnvironment(
    name="hello_world", 
    resources=flyte.Resources(memory="250Mi")
)

@env.task
def calculate(x: int) -> int:
    return x * 2 + 5

@env.task  
async def main(numbers: list[int]) -> float:
    # Parallel execution across distributed containers
    results = await asyncio.gather(*[
        calculate.aio(num) for num in numbers
    ])
    return sum(results) / len(results)

if __name__ == "__main__":
    flyte.init_from_config("config.yaml")
    run = flyte.run(main, numbers=list(range(10)))
    print(f"Result: {run.result}")
    print(f"View at: {run.url}")
# Run locally, execute remotely
uv run --prerelease=allow hello.py

🏗️ Core Concepts

TaskEnvironments: Container Configuration Made Simple

# Group tasks with shared configuration
env = flyte.TaskEnvironment(
    name="ml_pipeline",
    image=flyte.Image.from_debian_base().with_pip_packages(
        "torch", "pandas", "scikit-learn"
    ),
    resources=flyte.Resources(cpu=4, memory="8Gi", gpu=1),
    reusable=flyte.ReusePolicy(replicas=3, idle_ttl=300)
)

@env.task
def train_model(data: flyte.File) -> flyte.File:
    # Runs in configured container with GPU access
    pass

@env.task  
def evaluate_model(model: flyte.File, test_data: flyte.File) -> dict:
    # Same container configuration, different instance
    pass

Pure Python Workflows: No More DSL Constraints

@env.task
async def dynamic_pipeline(config: dict) -> list[str]:
    results = []
    
    # ✅ Use any Python construct
    for dataset in config["datasets"]:
        try:
            # ✅ Native error handling
            if dataset["type"] == "batch":
                result = await process_batch(dataset)
            else:
                result = await process_stream(dataset)
            results.append(result)
        except ValidationError as e:
            # ✅ Custom error recovery
            result = await handle_error(dataset, e)
            results.append(result)
    
    return results

Async Parallelism: Distributed by Default

@env.task
async def parallel_training(hyperparams: list[dict]) -> dict:
    # Each model trains on separate infrastructure  
    models = await asyncio.gather(*[
        train_model.aio(params) for params in hyperparams
    ])
    
    # Evaluate all models in parallel
    evaluations = await asyncio.gather(*[
        evaluate_model.aio(model) for model in models  
    ])
    
    # Find best model
    best_idx = max(range(len(evaluations)), 
                   key=lambda i: evaluations[i]["accuracy"])
    return {"best_model": models[best_idx], "accuracy": evaluations[best_idx]}

🎯 Advanced Features

Sub-Task Observability with Tracing

@flyte.trace
async def expensive_computation(data: str) -> str:
    # Function-level checkpointing - recoverable on failure
    result = await call_external_api(data)
    return process_result(result)

@env.task(cache=flyte.Cache(behavior="auto"))
async def main_task(inputs: list[str]) -> list[str]:
    results = []
    for inp in inputs:
        # If task fails here, it resumes from the last successful trace
        result = await expensive_computation(inp)  
        results.append(result)
    return results

Remote Task Execution

import flyte.remote

# Reference tasks deployed elsewhere
torch_task = flyte.remote.Task.get("torch_env.train_model", auto_version="latest")
spark_task = flyte.remote.Task.get("spark_env.process_data", auto_version="latest")

@env.task
async def orchestrator(raw_data: flyte.File) -> flyte.File:
    # Execute Spark job on big data cluster
    processed = await spark_task(raw_data)
    
    # Execute PyTorch training on GPU cluster  
    model = await torch_task(processed)
    
    return model

High-Performance Container Reuse

env = flyte.TaskEnvironment(
    name="high_throughput",
    reusable=flyte.ReusePolicy(
        replicas=10,      # Keep 10 warm containers
        idle_ttl=600,     # 10-minute idle timeout
    ),
    resources=flyte.Resources(cpu=2, memory="4Gi")
)

# Tasks scheduled in milliseconds on warm containers
@env.task
async def process_thousands(items: list[str]) -> list[str]:
    return await asyncio.gather(*[
        process_item.aio(item) for item in items
    ])

📊 Native Jupyter Integration

Run and monitor workflows directly from notebooks:

# In Jupyter cell
import flyte

flyte.init_from_config()
run = flyte.run(my_workflow, data=large_dataset)

# Stream logs in real-time
run.logs.stream()

# Get outputs when complete
results = run.wait()

🔧 Configuration & Deployment

Configuration File

# config.yaml
endpoint: https://my-flyte-instance.com
project: ml-team
domain: production
image:
  builder: remote
  registry: ghcr.io/my-org
auth:
  type: oauth2

Deploy and Run

# Deploy tasks to remote cluster
flyte deploy my_workflow.py

# Run deployed workflow  
flyte run my_workflow --input-file params.json

# Monitor execution
flyte logs <execution-id>

🆚 Migration from Flyte 1

Flyte 1 Flyte 2
@workflow + @task @env.task only
flytekit.map() await asyncio.gather()
@dynamic workflows Regular @env.task with loops
flytekit.conditional() Python if/else
LaunchPlan schedules @env.task(on_schedule=...)
Workflow failure handlers Python try/except

Example Migration

# Flyte 1
@flytekit.workflow  
def old_workflow(data: list[str]) -> list[str]:
    return [process_item(item=item) for item in data]

# Flyte 2  
@env.task
async def new_workflow(data: list[str]) -> list[str]:
    return await asyncio.gather(*[
        process_item.aio(item) for item in data
    ])

🌍 Ecosystem & Resources

🤝 Contributing

We welcome contributions! Whether it's:

  • 🐛 Bug fixes
  • ✨ New features
  • 📚 Documentation improvements
  • 🧪 Testing enhancements

See CONTRIBUTING.md for guidelines.

📄 License

Flyte 2 is licensed under the Apache 2.0 License.


Ready to build the future of distributed computing with pure Python?

⭐ Star this repo | 🚀 Get Started Now | 💬 Join our Community

Metadata

Release files for flyte 0.2.0b37

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for flyte 0.2.0b37
File Interpreter ABI Platform Actions
flyte-0.2.0b37-py3-none-any.whl Python 3 none any Details for flyte-0.2.0b37-py3-none-any.whl

Release files / flyte-0.2.0b37-py3-none-any.whl

Download URL flyte-0.2.0b37-py3-none-any.whl
Size 396.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6e1c649d40c7363fdc832c10e20a4c0292b3785bc577820ab674b7cca105266b
BLAKE2b-256 checksum
How to use checksums
0decd4970bd75ad2ba807e14491aeabdd70ef33b36fe7cb1f9ebdda4b15d8778
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.1

Security for flyte 0.2.0b37

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page