Flyte SDK for Python
Project description
Flytekit
Python Library for easily authoring, testing, deploying, and interacting with Flyte tasks, workflows, and launch plans. To understand more about flyte refer to,
Installation
Flytekit is designed for minimal footprint, and thus some features must be installed as extras.
Base Installation
This is the lightest-weight SDK install. This installation includes everything you need to interact with Flyte.
Modules include:
- The full Flyte IDL and an additional model layer for easier extension of the data model.
- gRPC client for communicating with the platform.
- Implementations for authoring and extending all Flyte entities (including tasks, workflows, and launch plans).
Tools include:
- flyte-cli (Command-Line Interface for Interacting with the Flyte Platform)
- pyflyte (Command-Line tool for easing the registration of Flyte entities)
pip install flytekit
Plugin Installation
Spark
If @spark_task
is to be used, one should install the spark
plugin.
pip install "flytekit[spark]" for Spark 2.4.x
pip install "flytekit[spark3]" for Spark 3.x
Schema
If Types.Schema()
is to be used for computations involving large dataframes, one should install the schema
extension.
pip install "flytekit[schema]"
Sidecar
If @sidecar_task
is to be used, one should install the sidecar
plugin.
pip install "flytekit[sidecar]"
Pytorch
If @pytorch_task
is to be used, one should install the pytorch
plugin.
pip install "flytekit[pytorch]"
TensorFlow
If @tensorflow_task
is to be used, one should install the tensorflow
plugin.
pip install flytekit[tensorflow]
Full Installation
To install all or multiple available plugins, one can specify them individually:
pip install "flytekit[sidecar,spark,schema]"
Or install them with the all
or all-spark2.4
or all-spark3
directives which will install all the plugins and a specific Spark version.
Please note that all
currently defaults to Spark 2.4.x. In a future release (starting 0.15.x), all
will be switched to use Spark 3.x.
pip install "flytekit[all]"
Development
Recipes
$ make
Available recipes:
setup Install requirements
fmt Format code with black and isort
lint Run linters
test Run tests
requirements Compile requirements
Setup (Do Once)
virtualenv ~/.virtualenvs/flytekit
source ~/.virtualenvs/flytekit/bin/activate
make setup
Formatting
We use black and isort to autoformat code. Run the following command to execute the formatters:
source ~/.virtualenvs/flytekit/bin/activate
make fmt
Testing
Unit Testing
source ~/.virtualenvs/flytekit/bin/activate
make test
Updating requirements
Update requirements in setup.py
, or update requirements for development in dev-requirements.in
. Then, validate, pin and freeze all requirements by running:
source ~/.virtualenvs/flytekit/bin/activate
make requirements
This will re-create the requirements.txt
and dev-requirements.txt
files which will be used for testing. You will have also have to re-run make setup
to update your local environment with the updated requirements.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for flytekit-0.13.0b7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bf4c02c6fbe5eb6800eb2c214847471c7f20fb82d2c53ab7f96fe8615de2ff76 |
|
MD5 | beb46fd91653793714074a0888c88349 |
|
BLAKE2b-256 | 320cda64e794c91a59454a3c6b6db631508180a1149f9a47add98993abb1ef84 |