Skip to main content

No project description provided

Project description

Dagster Scaleway (WIP)

This repository contains a Dagster integration for Scaleway.

It allows you to run Dagster pipelines on Scaleway Serverless Jobs.

Basic usage

Build a docker image containing your Dagster code and push it to the Scaleway Registry (or any other registry of your choice):

dagster project scaffold --name my-dagster-project
cd my-dagster-project

cat <<EOF > Dockerfile
FROM python:3.12-slim-bookworm
WORKDIR /app
COPY . .
RUN pip install pendulum==2.0.3
RUN pip install .
# Install the Dagster Scaleway module. You can also specify it in your "setup.py" file
RUN pip install dagster_scaleway
EOF

Build and push the image:

docker build -t rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest .
docker push rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest

Then, configure the dagster.yaml file to use this image:

run_launcher:
  module: dagster_scaleway
  class: ScalewayServerlessJobRunLauncher
  config:
    docker_image: rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest

Run Dagster locally:

pip install -e ".[dev]" "dagster-scaleway"
dagster dev

Your Dagster ops will be run as Scaleway Serverless Jobs! :tada:

See the Dagster documentation for more information on how to get started with Dagster.

Examples

See the examples folder for examples of how to use this integration.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dagster_scaleway-0.1.3.tar.gz (6.4 kB view hashes)

Uploaded Source

Built Distribution

dagster_scaleway-0.1.3-py3-none-any.whl (7.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page