No project description provided
Project description
Dagster Scaleway (WIP)
This repository contains a Dagster integration for Scaleway.
It allows you to run Dagster pipelines on Scaleway Serverless Jobs.
Basic usage
Build a docker image containing your Dagster code and push it to the Scaleway Registry (or any other registry of your choice):
dagster project scaffold --name my-dagster-project
cd my-dagster-project
cat <<EOF > Dockerfile
FROM python:3.12-slim-bookworm
WORKDIR /app
COPY . .
RUN pip install pendulum==2.0.3
RUN pip install .
# Install the Dagster Scaleway module. You can also specify it in your "setup.py" file
RUN pip install dagster_scaleway
EOF
Build and push the image:
docker build -t rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest .
docker push rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest
Then, configure the dagster.yaml
file to use this image:
run_launcher:
module: dagster_scaleway
class: ScalewayServerlessJobRunLauncher
config:
docker_image: rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest
Run Dagster locally:
pip install -e ".[dev]" "dagster-scaleway"
dagster dev
Your Dagster ops will be run as Scaleway Serverless Jobs! :tada:
See the Dagster documentation for more information on how to get started with Dagster.
Examples
See the examples folder for examples of how to use this integration.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file dagster_scaleway-0.1.3.tar.gz
.
File metadata
- Download URL: dagster_scaleway-0.1.3.tar.gz
- Upload date:
- Size: 6.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: poetry/1.8.2 CPython/3.11.9 Linux/6.5.0-1018-azure
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c119794694e1dabfc67c8951083706279c67c2f26a36a5c802b0d87a1e88810c |
|
MD5 | a5d5b6c34dfdec20002ef5209a27156e |
|
BLAKE2b-256 | d2ac28b8e4f257cdab7e0720eb2fbf27e255a1806cf9ba3c76c956be95505509 |
File details
Details for the file dagster_scaleway-0.1.3-py3-none-any.whl
.
File metadata
- Download URL: dagster_scaleway-0.1.3-py3-none-any.whl
- Upload date:
- Size: 7.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: poetry/1.8.2 CPython/3.11.9 Linux/6.5.0-1018-azure
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e8800949661862ee0e66578e54628de1467fb81eddc77f054a5f1d38fdcaa9e |
|
MD5 | 35e50898d16c1fdfeba42e75287f08c3 |
|
BLAKE2b-256 | a0893924faffdccfcaddf8ef43fa4c7df9eddea12558f4499461a2807e20480b |