Databricks Plugin for Flyte
This plugin provides Databricks integration for Flyte, enabling you to run Spark jobs on Databricks as Flyte tasks.
Installation
pip install flyteplugins-databricks
Usage
from flyteplugins.databricks import Databricks, DatabricksConnector
@task(task_config=Databricks(
databricks_conf={
"run_name": "flyte databricks plugin",
"new_cluster": {
"spark_version": "13.3.x-scala2.12",
"autoscale": {
"min_workers": 1,
"max_workers": 1,
},
"node_type_id": "m6i.large",
"num_workers": 1,
"aws_attributes": {
"availability": "SPOT_WITH_FALLBACK",
"instance_profile_arn": "arn:aws:iam::339713193121:instance-profile/databricks-demo",
"ebs_volume_type": "GENERAL_PURPOSE_SSD",
"ebs_volume_count": 1,
"ebs_volume_size": 100,
"first_on_demand": 1,
},
},
# "existing_cluster_id": "1113-204018-tb9vr2fm", # use existing cluster id if you want
"timeout_seconds": 3600,
"max_retries": 1,
},
databricks_instance="mycompany.cloud.databricks.com",
))
def my_spark_task() -> int:
# Your Spark code here
return 42
1 maintainer
Metadata
1 maintainer
Release files for flyteplugins-databricks 2.11.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| flyteplugins_databricks-2.11.1-py3-none-any.whl | Python 3 | none | any | Details |
Release files / flyteplugins_databricks-2.11.1-py3-none-any.whl
| Download URL | flyteplugins_databricks-2.11.1-py3-none-any.whl |
|---|---|
| Size | 6.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
420928703dcad32233fa4070f83f5e2f8d36dfd69cf8bf11bb2790c97eb9a799
|
|
BLAKE2b-256 checksum How to use checksums |
47d62ba18e1d0bd9dd5a87b6f4586da3ad2c53a5c94f7e28361661fc4ea9778c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
