README for Dagster Integration with Perian
Introduction
This Dagster integration allows you to easily dockerize your codebase and execute it on the Perian platform, our serverless GPU environment. By leveraging this integration, you can streamline your workflow, enabling efficient code deployment and execution without the need for managing infrastructure.
Features
- Dockerization: Automatically packages your codebase into a Docker container.
- Serverless Execution: Runs your Docker containers on Perian’s scalable GPU infrastructure.
- Ease of Use: Simple commands to get your project up and running quickly.
Prerequisites
- Docker installed on your local machine.
- Access to the Perian platform (ensure your account is set up).
- Dagster installed in your project (follow the Dagster installation guide).
Ops
This integration introduces the following Dagster ops:
-
containerize_codebase:- Packages your codebase into a Docker container, including the installation of any necessary dependencies.
- Parameters:
path: Directory of the codebase.dependency_file: Path to a requirements file (optional).python_version: Version of Python to use in the Docker image.image_name: Name for the Docker image.tag: Tag for the Docker image (default is "latest").docker_username,docker_password,docker_repository: Credentials for Docker Hub.command: Command to run in the container.parameters: Parameters for the command.
-
create_perian_job:- Creates a job on the Perian platform using the Docker image built from your codebase.
- Parameters:
image_name: Name of the Docker image.tag: Tag of the Docker image.docker_username,docker_password,docker_repository: Credentials for Docker Hub.cores,memory,accelerators,accelerator_type,country_code,provider: Configuration for the job instance.
-
get_perian_job:- Retrieves the status and logs of a job running on the Perian platform.
- Input:
job_metadata: Metadata containing the job ID.
-
cancel_perian_job:- Cancels a running job on the Perian platform.
- Input:
job_metadata: Metadata containing the job ID.
Installation
-
Install dagster-perian package:
pip install dagster-perian
or
poetry add dagster-perian
Example
Here’s an example of how to define a Dagster job that utilizes the containerize_codebase and create_perian_job ops:
from dagster import job
from perian_job.resources import perian_job_resource
from perian_job.operations import create_perian_job, containerize_codebase
@job(resource_defs={"perian_job_manager": perian_job_resource})
def perian_job_flow():
containerize_codebase()
create_perian_job()
```yaml
resources:
perian_job_manager:
config:
api_url: "https://api.perian.cloud"
organization: "<your-perian-organization-name>"
token: "<your-perian-token>"
ops:
create_perian_job:
config:
accelerators: 1
accelerator_type:"A100"
image_name: "<your-image-name>"
tag: "<your-image-tag>"
docker_username: "<your-docker-username>"
docker_password: "<your-docker-password>"
containerize_codebase:
config:
path: "<your-code-base-to-be-dockerized>"
image_name: "<your-image-name>"
tag: "<your-image-tag>"
python_version: "<python-version-to-be-used>"
docker_username: "<your-docker-username>"
docker_password: "<your-docker-password>"
command: "<running-script-or-module>"
parameters: "<command-parameters>"
Metadata
Release files for dagster-perian 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dagster_perian-0.1.1.tar.gz | 4.9 kB | Without provenance | Details for dagster_perian-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dagster_perian-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for dagster_perian-0.1.1-py3-none-any.whl |
Total release size: 10.6 kB
Release files / dagster_perian-0.1.1.tar.gz
| Download URL | dagster_perian-0.1.1.tar.gz |
|---|---|
| Size | 4.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
60dc28a77d2960a7990d045933684dd3af7a9461248c3220a6e429c7b52ed029
|
|
BLAKE2b-256 checksum How to use checksums |
9277b37b93ef345513f08640452990e018511d315380c0dc87dff9efed4bc36b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.7
|
Release files / dagster_perian-0.1.1-py3-none-any.whl
| Download URL | dagster_perian-0.1.1-py3-none-any.whl |
|---|---|
| Size | 5.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2dbb7ae8e13fc1fc204463b696941e5dd5760e8707739f5436ee0c30120e48f3
|
|
BLAKE2b-256 checksum How to use checksums |
3b73d0df40101eb47e4af2f4abc86be7b69cb895ab5fbf515ac0d7c46372c11f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.7
|
Security for dagster-perian 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.