Skip to main content

Flyte agent for Perian Job Platform (perian.io)

Project description

Flytekit Perian Job Platform Plugin

Flyte Agent plugin for executing Flyte tasks on Perian Job Platform (perian.io).

Perian Job Platform is still in closed beta. Contact support@perian.io if you are interested in trying it out.

To install the plugin, run the following command:

pip install flytekitplugins-perian-job

Getting Started

This plugin allows executing PythonFunctionTask on Perian.

An ImageSpec need to be built with the perian agent plugin installed.

Parameters

The following parameters can be used to set the requirements for the Perian task. If any of the requirements are skipped, it is replaced with the cheapest option. At least one requirement value should be set.

  • cores: Number of CPU cores
  • memory: Amount of memory in GB
  • accelerators: Number of accelerators
  • accelerator_type: Type of accelerator (e.g. 'A100'). For a full list of supported accelerators, use the perian CLI list-accelerators command.
  • country_code: Country code to run the job in (e.g. 'DE')

Credentials

The following secrets are required to be defined for the agent server:

  • Perian credentials:
    • perian_organization
    • perian_token
  • For accessing the Flyte storage bucket, you need to add either AWS or GCP credentials. These credentials are never logged by Perian and are only stored until then are used, then immediately deleted.
    • AWS credentials:
      • aws_access_key_id
      • aws_secret_access_key
    • GCP credentials:
      • google_application_credentials. This should be the full json credentials.
  • (Optional) Custom docker registry for pulling the Flyte image:
    • docker_registry_url
    • docker_registry_username
    • docker_registry_password

Example

example.py workflow example:

from flytekit import ImageSpec, task, workflow
from flytekitplugins.perian_job import PerianConfig

image_spec = ImageSpec(
    name="flyte-test",
    registry="my-registry",
    python_version="3.11",
    apt_packages=["wget", "curl", "git"],
    packages=[
        "flytekitplugins-perian-job",
    ],
)

@task(container_image=image_spec,
      task_config=PerianConfig(
        accelerators=1,
        accelerator_type="A100",
      ))
def perian_hello(name: str) -> str:
    return f"hello {name}!"

@workflow
def my_wf(name: str = "world") -> str:
    return perian_hello(name=name)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

flytekitplugins_perian_job-1.14.0b1.tar.gz (6.4 kB view details)

Uploaded Source

Built Distribution

File details

Details for the file flytekitplugins_perian_job-1.14.0b1.tar.gz.

File metadata

File hashes

Hashes for flytekitplugins_perian_job-1.14.0b1.tar.gz
Algorithm Hash digest
SHA256 2f9acefded6939be397ce1a2a2451a16c467cffdd61c43fc431c6513d32093e6
MD5 c0d0322c9820688f73d45fb58f4f7480
BLAKE2b-256 7ff8c528a43a846c6301907363252f14b8f8dcf135606921912674d50351283b

See more details on using hashes here.

File details

Details for the file flytekitplugins_perian_job-1.14.0b1-py3-none-any.whl.

File metadata

File hashes

Hashes for flytekitplugins_perian_job-1.14.0b1-py3-none-any.whl
Algorithm Hash digest
SHA256 794c763518b86a4dac3f425f78f6d5e054c0436390c1f9193c95a75a7b073dbd
MD5 80afa393be017bdc311718d14ea1a66d
BLAKE2b-256 7c2d13e18316dcd49dee9152fff15199849b7323f0df9b5b354d5685d7e581c2

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page