Skip to main content

Flytekit Memory Machine Cloud Plugin

Flyte Connector plugin to allow executing Flyte tasks using MemVerge Memory Machine Cloud.

To install the plugin, run the following command:

pip install flytekitplugins-mmcloud

To get started with MMCloud, refer to the MMCloud User Guide.

Getting Started

This plugin allows executing PythonFunctionTask using MMCloud without changing any function code.

Resource (cpu and mem) requests and limits, container images, and environment variable specifications are supported.

ImageSpec may be used to define images to run tasks.

Credentials

The following secrets are required to be defined for the connector server:

  • mmc_address: MMCloud OpCenter address
  • mmc_username: MMCloud OpCenter username
  • mmc_password: MMCloud OpCenter password

Defaults

Compute resources:

  • If only requests are specified, there are no limits.
  • If only limits are specified, the requests are equal to the limits.
  • If neither resource requests nor limits are specified, the default requests used for job submission are cpu="1" and mem="1Gi", and there are no limits.

Example

example.py workflow example:

import pandas as pd
from flytekit import ImageSpec, Resources, task, workflow
from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression

from flytekitplugins.mmcloud import MMCloudConfig

image_spec = ImageSpec(packages=["scikit-learn"], registry="docker.io/memverge")


@task
def get_data() -> pd.DataFrame:
    """Get the wine dataset."""
    return load_wine(as_frame=True).frame


@task(task_config=MMCloudConfig(), container_image=image_spec)  # Task will be submitted as MMCloud job
def process_data(data: pd.DataFrame) -> pd.DataFrame:
    """Simplify the task from a 3-class to a binary classification problem."""
    return data.assign(target=lambda x: x["target"].where(x["target"] == 0, 1))


@task(
    task_config=MMCloudConfig(submit_extra="--migratePolicy [enable=true]"),
    requests=Resources(cpu="1", mem="1Gi"),
    limits=Resources(cpu="2", mem="4Gi"),
    container_image=image_spec,
    environment={"KEY": "value"},
)
def train_model(data: pd.DataFrame, hyperparameters: dict) -> LogisticRegression:
    """Train a model on the wine dataset."""
    features = data.drop("target", axis="columns")
    target = data["target"]
    return LogisticRegression(max_iter=3000, **hyperparameters).fit(features, target)


@workflow
def training_workflow(hyperparameters: dict) -> LogisticRegression:
    """Put all of the steps together into a single workflow."""
    data = get_data()
    processed_data = process_data(data=data)
    return train_model(
        data=processed_data,
        hyperparameters=hyperparameters,
    )

Connector Image

Install flytekitplugins-mmcloud in the connector image.

A float binary (obtainable via the OpCenter) is required. Copy it to the connector image PATH.

Sample Dockerfile for building an connector image:

FROM python:3.11-slim-bookworm

WORKDIR /root
ENV PYTHONPATH /root

# flytekit will autoload the connector if package is installed.
RUN pip install flytekitplugins-mmcloud
COPY float /usr/local/bin/float

CMD pyflyte serve connector --port 8000

Metadata

Release files for flytekitplugins-mmcloud 1.16.32

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for flytekitplugins-mmcloud 1.16.32
File Size Uploaded Provenance Actions
flytekitplugins_mmcloud-1.16.32.tar.gz 9.0 kB Without provenance Details for flytekitplugins_mmcloud-1.16.32.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for flytekitplugins-mmcloud 1.16.32
File Interpreter ABI Platform Actions
flytekitplugins_mmcloud-1.16.32-py3-none-any.whl Python 3 none any Details for flytekitplugins_mmcloud-1.16.32-py3-none-any.whl

Total release size: 17.9 kB

Release files / flytekitplugins_mmcloud-1.16.32.tar.gz

Download URL flytekitplugins_mmcloud-1.16.32.tar.gz
Size 9.0 kB
Tags Source
SHA-256 checksum
How to use checksums
7e10f75ba6470efe75378e8d4d586ac8a000b9e3de9fce08a6abcbbc3f086410
BLAKE2b-256 checksum
How to use checksums
68a0bfd46e230e8c4863a30505ca6ebad26dd0d913f8c008d9b2b44cf9d8d708
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / flytekitplugins_mmcloud-1.16.32-py3-none-any.whl

Download URL flytekitplugins_mmcloud-1.16.32-py3-none-any.whl
Size 8.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
206bda4de0a1a324687f18bfd539ea6b0d1e25775d2a8e490f7520414a6acaea
BLAKE2b-256 checksum
How to use checksums
94fd563cd2f203173b6a5b1e62df9763b5183f9c4110fb6413e4c02d82e157fb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Security for flytekitplugins-mmcloud 1.16.32

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.16.32 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page