No project description provided
Project description
Dagster Scaleway (WIP)
This repository contains a Dagster integration for Scaleway.
It allows you to run Dagster pipelines on Scaleway Serverless Jobs.
Basic usage
Build a docker image containing your Dagster code and push it to the Scaleway Registry (or any other registry of your choice):
dagster project scaffold --name my-dagster-project
cd my-dagster-project
cat <<EOF > Dockerfile
FROM python:3.12-slim-bookworm
WORKDIR /app
COPY . .
RUN pip install pendulum==2.0.3
RUN pip install .
# Install the Dagster Scaleway module. You can also specify it in your "setup.py" file
RUN pip install dagster_scaleway
EOF
Build and push the image:
docker build -t rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest .
docker push rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest
Then, configure the dagster.yaml
file to use this image:
run_launcher:
module: dagster_scaleway
class: ScalewayServerlessJobRunLauncher
config:
docker_image: rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest
Run Dagster locally:
pip install -e ".[dev]" "dagster-scaleway"
dagster dev
Your Dagster ops will be run as Scaleway Serverless Jobs! :tada:
See the Dagster documentation for more information on how to get started with Dagster.
Examples
See the examples folder for examples of how to use this integration.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
dagster_scaleway-0.1.3.tar.gz
(6.4 kB
view hashes)
Built Distribution
Close
Hashes for dagster_scaleway-0.1.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e8800949661862ee0e66578e54628de1467fb81eddc77f054a5f1d38fdcaa9e |
|
MD5 | 35e50898d16c1fdfeba42e75287f08c3 |
|
BLAKE2b-256 | a0893924faffdccfcaddf8ef43fa4c7df9eddea12558f4499461a2807e20480b |