REST client for Databricks
Project description
databricks-client
About
A REST client for the Databricks REST API.
This module is a thin layer allowing to build HTTP Requests. It does not expose API operations as distinct methods, but rather exposes generic methods allowing to build API calls.
The Databricks API sometimes returns 200 error codes and HTML content when the request is not properly authenticated. The client intercepts such occurrences (detecting non-JSON returned content) and wraps them into an exception.
This open-source project is not developed by nor affiliated with Databricks.
Installing
pip install databricks-client
Usage
import databricks_client
client = databricks_client.create("https://northeurope.azuredatabricks.net/api/2.0")
client.auth_pat_token(pat_token)
client.ensure_available()
clusters_list = client.get('clusters/list')
for cluster in clusters_list["clusters"]:
print(cluster)
Usage with a newly provisioned workspace
If using this module as part of a provisioning job, you need to call client.ensure_available()
.
When the first user logs it to a new Databricks workspace, workspace provisioning is triggered, and the API is not available until that job has completed (that usually takes under a minute, but could take longer depending on the network configuration).
The method client.ensure_available(url="instance-pools/list", retries=100, delay_seconds=6)
attempts connecting to the provided URL and retries as long as the workspace is in provisioning
state, or until the given number of retries has elapsed.
Usage with Azure Active Directory
Note: Azure AD authentication for Databricks is currently in preview.
The client generates short-lived Azure AD tokens. If you need to use your client for longer
than the lifetime (typically 30 minutes), rerun client.auth_azuread
periodically.
Azure AD authentication with Azure CLI
pip install databricks-client[azurecli]
az login
import databricks_client
client = databricks_client.create("https://northeurope.azuredatabricks.net/api/2.0")
client.auth_azuread("/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-rg/providers/Microsoft.Databricks/workspaces/my-workspace")
# or client.auth_azuread(resource_group="my-rg", workspace_name="my-workspace")
client.ensure_available()
clusters_list = client.get('clusters/list')
for cluster in clusters_list["clusters"]:
print(cluster)
This is recommended with Azure DevOps Pipelines using the Azure CLI task.
Azure AD authentication with ADAL
pip install databricks-client
pip install adal
import databricks_client
import adal
authority_host_uri = 'https://login.microsoftonline.com'
authority_uri = authority_host_uri + '/' + tenant_id
context = adal.AuthenticationContext(authority_uri)
def token_callback(resource):
return context.acquire_token_with_client_credentials(resource, client_id, client_secret)["accessToken"]
client = databricks_client.create("https://northeurope.azuredatabricks.net/api/2.0")
client.auth_azuread("/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-rg/providers/Microsoft.Databricks/workspaces/my-workspace", token_callback)
# or client.auth_azuread(resource_group="my-rg", workspace_name="my-workspace", token_callback=token_callback)
client.ensure_available()
clusters_list = client.get('clusters/list')
for cluster in clusters_list["clusters"]:
print(cluster)
Example usages
Generating a PAT token
response = client.post(
'token/create',
json={"lifetime_seconds": 60, "comment": "Unit Test Token"}
)
pat_token = response['token_value']
Uploading a notebook
import base64
with open(notebook_file, "rb") as f:
file_content = f.read()
client.post(
'workspace/import',
json={
"content": base64.b64encode(file_content).decode('ascii'),
"path": notebook_path,
"overwrite": False,
"language": "PYTHON",
"format": "SOURCE"
}
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for databricks_client-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f847d9abde84c7532838130bf379025b31b2c8af94f128a98d529620b9f0f833 |
|
MD5 | 039629cf8c5761987aed8f6d7b4548c9 |
|
BLAKE2b-256 | adc3cf0972f47927486a2669b5fa46d827e4c9e86634a478d72d8a222447dc04 |