dagster-ray 0.4.2

Dagster integration library for Ray

dagster-ray

---

Ray integration for Dagster.

dagster-ray enables working with distributed Ray compute from Dagster pipelines, combining Dagster's excellent orchestration capabilities and Ray's distributed computing power together.

[!NOTE] This project is ready for production use, but some APIs may change between minor releases.

Learn more in the docs

🚀 Key Features

• 🎯 Run Launchers & Executors: Submit Dagster runs or individual steps by submitting Ray jobs
• 🔧 Ray Resources: Automatically create and destroy ephemeral Ray clusters and connect to them in client mode
• 📡 Dagster Pipes Integration: Submit external scripts as Ray jobs, stream back logs and rich Dagster metadata
• ☸️ KubeRay Support: Utilize RayJob and RayCluster custom resources in client or job submission mode (tutorial)
• 🏭 Production Ready: Tested against a matrix of core dependencies, integrated with Dagster+
• ⚡ Instant Startup: Leverage RayCluster with Cluster Sharing for lightning-fast development cycles with zero cold start times (intended for development environments)

Installation

pip install dagster-ray

📚 Docs

📖 Full Documentation

• Tutorial: Step-by-step guide with examples
• API Reference: Complete API reference

🤝 Contributing

Contributions are very welcome! To get started:

uv sync --all-extras --all-groups
uv run pre-commit install

🧪 Testing

uv run pytest

Running KubeRay tests requires the following tools to be present:

• docker
• kubectl
• helm
• minikube

❄️ Nix users will find them provided in the dev shell:

nix develop

Documentation

To build and serve the documentation locally:

# Serve documentation locally
uv run --group docs mkdocs serve