Databricks Plugin for Flyte
This plugin provides Databricks integration for Flyte, enabling you to run Spark jobs on Databricks as Flyte tasks.
Installation
pip install flyteplugins-databricks
Usage
from flyteplugins.databricks import Databricks, DatabricksConnector
@task(task_config=Databricks(
databricks_conf={
"run_name": "flyte databricks plugin",
"new_cluster": {
"spark_version": "13.3.x-scala2.12",
"autoscale": {
"min_workers": 1,
"max_workers": 1,
},
"node_type_id": "m6i.large",
"num_workers": 1,
"aws_attributes": {
"availability": "SPOT_WITH_FALLBACK",
"instance_profile_arn": "arn:aws:iam::339713193121:instance-profile/databricks-demo",
"ebs_volume_type": "GENERAL_PURPOSE_SSD",
"ebs_volume_count": 1,
"ebs_volume_size": 100,
"first_on_demand": 1,
},
},
# "existing_cluster_id": "1113-204018-tb9vr2fm", # use existing cluster id if you want
"timeout_seconds": 3600,
"max_retries": 1,
},
databricks_instance="mycompany.cloud.databricks.com",
))
def my_spark_task() -> int:
# Your Spark code here
return 42
1 maintainer
Metadata
1 maintainer
Release files for flyteplugins-databricks 2.11.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| flyteplugins_databricks-2.11.3-py3-none-any.whl | Python 3 | none | any | Details |
Release files / flyteplugins_databricks-2.11.3-py3-none-any.whl
| Download URL | flyteplugins_databricks-2.11.3-py3-none-any.whl |
|---|---|
| Size | 6.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e697bd6dc546f7b45bcdca6f7c742b1da649483ba787168ace91d350a21442b3
|
|
BLAKE2b-256 checksum How to use checksums |
cb4e5ab19ec049ad68d080aaf724161cb5866ea6ec1d42b90cf46fb50792ce8d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
