Skip to main content

High-level Databricks client

Project description

PyPI - Version GitHub Build

License PyPI - Python Version PyPI - Downloads

Ruff Checked with mypy

dbxio: High-level Databricks client

Overview

dbxio is a high-level client for Databricks that simplifies working with tables and volumes. It provides a simple interface for reading and writing data, creating and deleting objects, and running SQL queries and fetching results.

Why dbxio?

  1. dbxio connects the power of Databricks SQL and Python for local data manipulation.
  2. dbxio provides a simple and intuitive interface for working with Databricks Tables and Volumes. Now it's possible to read/write data with just a few lines of code.
  3. For large amounts of data, dbxio uses intermediate object storage of your choice to perform bulk upload later (see COPY INTO for more details). So, you can upload any amount of data, and dbxio will take care of synchronizing the data with the table in Databricks.

Alternatives

Currently, we are not aware of any alternatives that offer the same functionality as dbxio. If you come across any, we would be interested to learn about them. Please let us know by opening an issue in our GitHub repository.


Installation

dbxio requires Python 3.9 or later. You can install dbxio using pip:

pip install dbxio

dbxio by Example

import dbxio

client = dbxio.DbxIOClient.from_cluster_settings(
    cluster_type=dbxio.ClusterType.SQL_WAREHOUSE,
    http_path='<YOUR_HTTP_PATH>',
    server_hostname='<YOUR_SERVER_HOSTNAME>',
    settings=dbxio.Settings(cloud_provider=dbxio.CloudProvider.AZURE),
)

# read table
table = list(dbxio.read_table('catalog.schema.table', client=client))

# write table
data = [
    {'col1': 1, 'col2': 'a', 'col3': [1, 2, 3]},
    {'col1': 2, 'col2': 'b', 'col3': [4, 5, 6]},
]
schema = dbxio.TableSchema.from_obj(
    {
        'col1': dbxio.types.IntType(),
        'col2': dbxio.types.StringType(),
        'col3': dbxio.types.ArrayType(dbxio.types.IntType()),
    }
)
dbxio.bulk_write_table(
    dbxio.Table('domain.schema.table', schema=schema),
    data,
    client=client,
    abs_name='blob_storage_name',
    abs_container_name='container_name',
    append=True,
)

Cloud Support

dbxio supports the following cloud providers:

  • Azure
  • AWS (in plans)
  • GCP (in plans)

Project Information

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dbxio-0.5.2.tar.gz (36.2 kB view details)

Uploaded Source

Built Distribution

dbxio-0.5.2-py3-none-any.whl (46.9 kB view details)

Uploaded Python 3

File details

Details for the file dbxio-0.5.2.tar.gz.

File metadata

  • Download URL: dbxio-0.5.2.tar.gz
  • Upload date:
  • Size: 36.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for dbxio-0.5.2.tar.gz
Algorithm Hash digest
SHA256 cfe2aa0a513ed74244d6758e2d2fbeaecf26ade989bede7453a3797f2a3c469d
MD5 ba9aa1546b88fe16efc0d342e919be76
BLAKE2b-256 819b9e99d2cd8fb715c61047c761e4538746384e6fabc9281035b202e1b1da5c

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbxio-0.5.2.tar.gz:

Publisher: release.yml on Toloka/dbxio

Attestations:

File details

Details for the file dbxio-0.5.2-py3-none-any.whl.

File metadata

  • Download URL: dbxio-0.5.2-py3-none-any.whl
  • Upload date:
  • Size: 46.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for dbxio-0.5.2-py3-none-any.whl
Algorithm Hash digest
SHA256 290578cf14980bbac66ba2aaf9ef69c6d8e703edabfd64cf13f6ec392306b7c0
MD5 293eedb3ad056f66f5989edc27f26eba
BLAKE2b-256 efdddeac49c5e9ffc7ae9d56fb8b45e71c66cf7b6cdf4a96655999290bdd4c83

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbxio-0.5.2-py3-none-any.whl:

Publisher: release.yml on Toloka/dbxio

Attestations:

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page