Databricks Plugin for Flyte
This plugin provides Databricks integration for Flyte, enabling you to run Spark jobs on Databricks as Flyte tasks.
Installation
pip install flyteplugins-databricks
Usage
from flyteplugins.databricks import Databricks, DatabricksConnector
@task(task_config=Databricks(
databricks_conf={
"run_name": "flyte databricks plugin",
"new_cluster": {
"spark_version": "13.3.x-scala2.12",
"autoscale": {
"min_workers": 1,
"max_workers": 1,
},
"node_type_id": "m6i.large",
"num_workers": 1,
"aws_attributes": {
"availability": "SPOT_WITH_FALLBACK",
"instance_profile_arn": "arn:aws:iam::339713193121:instance-profile/databricks-demo",
"ebs_volume_type": "GENERAL_PURPOSE_SSD",
"ebs_volume_count": 1,
"ebs_volume_size": 100,
"first_on_demand": 1,
},
},
# "existing_cluster_id": "1113-204018-tb9vr2fm", # use existing cluster id if you want
"timeout_seconds": 3600,
"max_retries": 1,
},
databricks_instance="mycompany.cloud.databricks.com",
))
def my_spark_task() -> int:
# Your Spark code here
return 42
1 maintainer
Metadata
1 maintainer
Release files for flyteplugins-databricks 2.10.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| flyteplugins_databricks-2.10.4-py3-none-any.whl | Python 3 | none | any | Details |
Release files / flyteplugins_databricks-2.10.4-py3-none-any.whl
| Download URL | flyteplugins_databricks-2.10.4-py3-none-any.whl |
|---|---|
| Size | 6.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e34892708e66b8936a033aac15070a28c4154c07f22485e20d31a25d73705244
|
|
BLAKE2b-256 checksum How to use checksums |
22bb2a79b3bbf55c106743b415164600ff5c18bd3f910fee4c0354e8647696cc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
