Databricks Plugin for Flyte
This plugin provides Databricks integration for Flyte, enabling you to run Spark jobs on Databricks as Flyte tasks.
Installation
pip install flyteplugins-databricks
Usage
from flyteplugins.databricks import Databricks, DatabricksConnector
@task(task_config=Databricks(
databricks_conf={
"run_name": "flyte databricks plugin",
"new_cluster": {
"spark_version": "13.3.x-scala2.12",
"autoscale": {
"min_workers": 1,
"max_workers": 1,
},
"node_type_id": "m6i.large",
"num_workers": 1,
"aws_attributes": {
"availability": "SPOT_WITH_FALLBACK",
"instance_profile_arn": "arn:aws:iam::339713193121:instance-profile/databricks-demo",
"ebs_volume_type": "GENERAL_PURPOSE_SSD",
"ebs_volume_count": 1,
"ebs_volume_size": 100,
"first_on_demand": 1,
},
},
# "existing_cluster_id": "1113-204018-tb9vr2fm", # use existing cluster id if you want
"timeout_seconds": 3600,
"max_retries": 1,
},
databricks_instance="mycompany.cloud.databricks.com",
))
def my_spark_task() -> int:
# Your Spark code here
return 42
1 maintainer
Metadata
1 maintainer
Release files for flyteplugins-databricks 2.11.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| flyteplugins_databricks-2.11.2-py3-none-any.whl | Python 3 | none | any | Details |
Release files / flyteplugins_databricks-2.11.2-py3-none-any.whl
| Download URL | flyteplugins_databricks-2.11.2-py3-none-any.whl |
|---|---|
| Size | 6.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d9a75826e8c192613540acaf873ea9af07cc1c525ff87c610f285dd83c2cedab
|
|
BLAKE2b-256 checksum How to use checksums |
4cd1eeb6174b772d35822802f77207b15b78e5a954a4e48597f8c245f837d8eb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
