pydataframer-databricks

Databricks connector for DataFramer.

Installation

pip install pydataframer-databricks

Usage

Inside a Databricks notebook the connector authenticates as whoever runs the notebook and finds the workspace's SQL warehouse itself. scope is the Databricks secret scope that holds your DataFramer API key.

from pydataframer_databricks import DatabricksConnector, DatasetType, FileType

databricks_connector = DatabricksConnector(scope="dataframer")
api_key = databricks_connector.get_secret("DATAFRAMER_API_KEY")

Anywhere else there is no workspace identity and no secret scope, so pass the connection details in. The SQL warehouse's HTTP path is on its Connection details tab.

databricks_connector = DatabricksConnector(
    server_hostname="dbc-xxxxx.cloud.databricks.com",
    http_path="/sql/1.0/warehouses/xxxxxxxx",
    client_id="<service principal client ID>",
    client_secret="<service principal secret>",
)

Fetch sample data

df = databricks_connector.fetch_sample_data(
    num_items_to_select=100,
    table_name="catalog.schema.table"
)

Load to table (SINGLE_FILE: CSV, JSON, JSONL)

databricks_connector.load_to_table(
    table_name="catalog.schema.generated_samples",
    downloaded_zip=downloaded_zip,
    dataset_type=DatasetType.SINGLE_FILE,
    file_type=FileType.CSV,  # or FileType.JSON, FileType.JSONL
)

Load to volume (any dataset type)

databricks_connector.load_to_volume(
    volume_path="/Volumes/catalog/schema/volume_name",
    downloaded_zip=downloaded_zip,
    output_folder="generated_samples",  # optional
)