Skip to main content

No project description provided

Project description

Dagster Scaleway (WIP)

This repository contains a Dagster integration for Scaleway.

It allows you to run Dagster pipelines on Scaleway Serverless Jobs.

Basic usage

Build a docker image containing your Dagster code and push it to the Scaleway Registry (or any other registry of your choice):

dagster project scaffold --name my-dagster-project
cd my-dagster-project

cat <<EOF > Dockerfile
FROM python:3.12-slim-bookworm
WORKDIR /app
COPY . .
RUN pip install pendulum==2.0.3
RUN pip install .
# Install the Dagster Scaleway module. You can also specify it in your "setup.py" file
RUN pip install dagster_scaleway
EOF

Build and push the image:

docker build -t rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest .
docker push rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest

Then, configure the dagster.yaml file to use this image:

run_launcher:
  module: dagster_scaleway
  class: ScalewayServerlessJobRunLauncher
  config:
    docker_image: rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest

Run Dagster locally:

pip install -e ".[dev]" "dagster-scaleway"
dagster dev

Your Dagster ops will be run as Scaleway Serverless Jobs! :tada:

See the Dagster documentation for more information on how to get started with Dagster.

Examples

See the examples folder for examples of how to use this integration.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dagster_scaleway-0.1.3.tar.gz (6.4 kB view details)

Uploaded Source

Built Distribution

dagster_scaleway-0.1.3-py3-none-any.whl (7.9 kB view details)

Uploaded Python 3

File details

Details for the file dagster_scaleway-0.1.3.tar.gz.

File metadata

  • Download URL: dagster_scaleway-0.1.3.tar.gz
  • Upload date:
  • Size: 6.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: poetry/1.8.2 CPython/3.11.9 Linux/6.5.0-1018-azure

File hashes

Hashes for dagster_scaleway-0.1.3.tar.gz
Algorithm Hash digest
SHA256 c119794694e1dabfc67c8951083706279c67c2f26a36a5c802b0d87a1e88810c
MD5 a5d5b6c34dfdec20002ef5209a27156e
BLAKE2b-256 d2ac28b8e4f257cdab7e0720eb2fbf27e255a1806cf9ba3c76c956be95505509

See more details on using hashes here.

File details

Details for the file dagster_scaleway-0.1.3-py3-none-any.whl.

File metadata

  • Download URL: dagster_scaleway-0.1.3-py3-none-any.whl
  • Upload date:
  • Size: 7.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: poetry/1.8.2 CPython/3.11.9 Linux/6.5.0-1018-azure

File hashes

Hashes for dagster_scaleway-0.1.3-py3-none-any.whl
Algorithm Hash digest
SHA256 3e8800949661862ee0e66578e54628de1467fb81eddc77f054a5f1d38fdcaa9e
MD5 35e50898d16c1fdfeba42e75287f08c3
BLAKE2b-256 a0893924faffdccfcaddf8ef43fa4c7df9eddea12558f4499461a2807e20480b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page