Skip to main content

databuck-sdk

PyPI version Python versions Publish to PyPI License: customer license

Python wrapper for the DataBuck Spark SDK. It discovers data-quality rules from an existing Spark DataFrame or a registered Spark table and exports expectations that Databricks Lakeflow Declarative Pipelines can execute before downstream data is published.

Install

python -m pip install databuck-sdk

Give each customer one signed databucksdk.properties file:

customer_id=your-customer-id
issued_on=2026-11-02
expires_on=2027-12-31
license_key=signed-key-created-by-the-issuer

Set DATABUCK_SDK_CONFIG_FILE to the full local file path before importing the SDK. If the file is absent, import downloads databucksdk.properties from the DataBuck S3 URL to that path and validates its signature and dates. An existing file is reused. Without a configured path, the SDK uses databuck/jars/databucksdk.properties in the installed package. The license is valid through its expires_on date, inclusive. Renew it with a newly issued config file when it expires. Editing the dates or customer ID invalidates the signature.

import os
os.environ["DATABUCK_SDK_CONFIG_FILE"] = "/path/to/databucksdk.properties"
from databuck import DataBuck

If databuck-spark-sdk is installed, remove it before installing this package; both distributions provide the same databuck Python import:

python -m pip uninstall databuck-spark-sdk
python -m pip install --upgrade databuck-sdk

Importing databuck first downloads the signed properties file if needed, validates it, and then downloads databuck-sdk.jar from the DataBuck S3 URL to databuck/jars/ in the installed Python package, with progress output. The downloaded path is set in DATABUCK_SDK_JAR for the current Python process. A later import reuses the existing JAR. The JAR is not included in the wheel because it exceeds PyPI's default per-file upload limit. An internet connection and write access to the installed package directory are required for the first download.

python -m databuck is an explicit download command that also prints the local path. Set DATABUCK_SDK_AUTO_DOWNLOAD=0 before import to defer both downloads when offline; a valid local properties file is still required. DataBuck.jar_path() returns the path after download. Downloading the JAR does not install it as a Databricks compute library.

To use an existing JAR or another writable destination, set DATABUCK_SDK_JAR to that file path. If the download URL changes, set DATABUCK_SDK_JAR_URL to the new HTTPS URL before importing or running python -m databuck. The download checks that the file is a valid JAR archive. Use a trusted JAR URL because the package does not verify a checksum. Set DATABUCK_SDK_CONFIG_FILE_URL before import if the signed properties file must come from another trusted URL.

Licensing

The SDK requires a signed databucksdk.properties file. It checks the customer ID, issue date, expiry date, and Ed25519 signature at import and before public SDK operations. The file remains valid through its expiry date, inclusive. See the license notice for the repository's licensing status and contact DataBuck for customer terms or renewal.

Databricks notebook setup

The SDK notebook was run on Databricks Runtime 13.3 ML (Spark 3.4.1, Scala 2.12). Check attached connector libraries separately if Spark fails while loading a data source.

Run this once in the notebook to install the Python package:

%pip install databuck-sdk

After installing or upgrading, restart Python in the next cell so the notebook loads the new package version:

dbutils.library.restartPython()

Then verify the version and set a writable Unity Catalog Volume file path before importing databuck:

import os
from importlib.metadata import version

print(version("databuck-sdk"))

os.environ["DATABUCK_SDK_JAR"] = (
    "/Volumes/catalog/schema/volume/databuck-sdk.jar"
)
os.environ["DATABUCK_SDK_CONFIG_FILE"] = (
    "/Volumes/catalog/schema/volume/databucksdk.properties"
)

from databuck import DataBuck  # Downloads the JAR to that path on first import.

print(DataBuck.jar_path())

Use the full filename ending in .jar, not just the Volume directory. Importing databuck downloads or reuses the JAR at that path. Setting the variable after the import is too late to choose the initial download location. A Volume keeps the JAR available beyond the current notebook driver session.

Before calling DataBuck.discover(...), attach the downloaded JAR to the compute that runs discovery: in Databricks, open Compute > select that compute > Libraries > Install New, select the Volume as the library source, choose the JAR path printed above, and install it. If the compute is already running, restart it or reattach the notebook so Spark loads the new library. Then read your Spark DataFrame and run the discovery example below. If the old databuck-spark-sdk.jar is attached to that compute, remove it before attaching databuck-sdk.jar. The Python import downloads the file; it does not attach the JAR to compute. See Databricks compute-scoped library instructions.

Local PySpark usage

from pyspark.sql import SparkSession
import os

from databuck import DataBuck

spark = (
    SparkSession.builder
    .config("spark.jars", DataBuck.jar_path())
    .getOrCreate()
)

df = spark.read.csv("customers.csv", header=True)

print(DataBuck.count(df))

Discover and export rules for Databricks

Run discovery in a Databricks notebook or Spark job against an existing DataFrame, or pass a Spark table name and the SDK will read it into a DataFrame internally. Pass PDF and DOCX policy files for context-aware BuckGPT rules. The SDK exports generated rules as JSON (.json), YAML (.yaml / .yml), or a Great Expectations suite (.gx.json). A Lakeflow Declarative Pipeline then loads and executes those rules as expectations. Choose one of the two JSON inputs below; both use the same discovery and export flow. The YAML workflow appears below.

from databuck import DataBuck

context = {
    "gemini_api_key": "YOUR_GEMINI_API_KEY",
    "domain_name": "banking",
    "pdf_paths": [
        "/Volumes/catalog/schema/volume/customer_policy.pdf",
        "/Volumes/catalog/schema/volume/data_contract.docx",
    ],
}

# Option 1: use a DataFrame already available in your pipeline.
json_path = DataBuck.discover(
    df, "/Volumes/catalog/schema/volume/expectations.json", context=context
)
print(json_path)

Or pass a registered Spark table name. The SDK calls spark.read.table(...) internally, then passes that DataFrame through the same discovery and export:

json_path = DataBuck.discover(
    "catalog.schema.telco_customer_subscription",
    "/Volumes/catalog/schema/volume/expectations.json",
    context=context,
)
print(json_path)

Replace the example Volume paths and table name with files and a table available in your workspace. The table-name option uses the notebook's Spark session and requires read access to the registered table; it does not take separate database connection details. Passing an existing DataFrame avoids reading the table again.

The call combines automatic and policy-informed rules. The output extension selects JSON or YAML. With domain_name, JSON and YAML contain selected HIGH and MEDIUM rules from both sources.

pdf_paths accepts PDF and DOCX files available to the notebook. For a dataset-level uniqueness rule, the output includes a dataset_checks section. Run its SQL against the full dataset separately, replacing {{DATAFRAME}} with your source table or view. Dataset checks are not Lakeflow row expectations.

To export only automatic profiling rules, omit context:

json_path = DataBuck.discover(
    df, "/Volumes/catalog/schema/volume/expectations.json"
)

Without context, the output contains automatic rules and JSON places them in the warn group. To include policy-informed rules, pass a business statement or policy document along with the Gemini API key.

The JSON contains three action dictionaries: warn, drop, and fail. In a Databricks Lakeflow Declarative Pipeline, use the dictionaries with dp.expect_all, dp.expect_all_or_drop, and dp.expect_all_or_fail, respectively.

For example, add this to the Lakeflow Declarative Pipeline source file. The expectations run when the pipeline updates; the SDK's discovery call above generates the file beforehand:

import json
from pyspark import pipelines as dp

with open("/Volumes/catalog/schema/volume/expectations.json", encoding="utf-8") as stream:
    expectations = json.load(stream)

@dp.materialized_view(name="telco_customers_checked")
@dp.expect_all(expectations["warn"])
@dp.expect_all_or_drop(expectations["drop"])
@dp.expect_all_or_fail(expectations["fail"])
def customers_checked():
    return spark.read.table("catalog.schema.telco_customer_subscription")

What the pipeline results look like

DataBuck.discover(df, output_path, ...) prints the rules and returns the path to the rules file. The row results appear after the Lakeflow Declarative Pipeline runs, in the validated table's Expectations view. For example, one telco demo run showed:

Pipeline result Records
Written 58 (58%)
Dropped 42 (42%)
Expectation Action shown in Databricks Failed records
BuckGPT_Rule_006 ALLOW 60
BuckGPT_Rule_007 DROP 40
BuckGPT_Rule_009 DROP 20
valid_length_network_type DROP 4
valid_pattern_network_type DROP 4

ALLOW corresponds to dp.expect_all: the violation is recorded without dropping the row. DROP corresponds to dp.expect_all_or_drop: invalid rows are excluded from the validated output. A FAIL expectation stops the pipeline update when violated. A row can fail more than one rule, so the per-rule failed record counts do not add up to the dropped-row total. These numbers are from one demo update; actual results depend on the source data and exported rules.

The pipeline must read a DataFrame with the columns used by the exported expectations. Regenerate the rules file when the source schema or business policy changes, then refresh the pipeline.

{
  "warn": {"not_null_customer_email": "`customer_email` IS NOT NULL"},
  "drop": {"not_null_order_id": "`order_id` IS NOT NULL"},
  "fail": {"valid_pattern_currency_code": "`currency_code` IS NULL OR CAST(`currency_code` AS STRING) RLIKE '(?:^[A-Z][A-Z][A-Z]$)'"}
}

The example shows the file shape; actual actions depend on the generated rules.

Export flat YAML expectations

Pass a .yaml or .yml output file to write a flat YAML document. Pass either an existing DataFrame or a table name. Choose one of these calls:

from databuck import DataBuck

context = {
    "gemini_api_key": "YOUR_GEMINI_API_KEY",
    "pdf_paths": [
        "/Volumes/catalog/schema/volume/customer_policy.pdf",
        "/Volumes/catalog/schema/volume/data_contract.docx",
    ],
}

# Option 1: use a DataFrame already available in your pipeline.
yaml_path = DataBuck.discover(
    df,
    "/Volumes/catalog/schema/volume/telco_expectations.yaml",
    context=context,
)
print(yaml_path)

Or pass a registered Spark table name. The SDK reads it into a DataFrame internally before discovering and exporting the rules:

yaml_path = DataBuck.discover(
    "catalog.schema.telco_customer_subscription",
    "/Volumes/catalog/schema/volume/telco_expectations.yaml",
    context=context,
)
print(yaml_path)

No separate table_name= argument is needed. When you pass a table name, YAML includes a table: field. When you pass a DataFrame, YAML contains expectations: without a table: field because the original table name is not reliably available from a DataFrame.

For example, the generated file has this shape (the names and SQL depend on the discovered rules):

table: catalog.schema.telco_customer_subscription

expectations:
  not_null_subscriber_id: "`subscriber_id` IS NOT NULL"
  BuckGPT_Rule_001: "NOT COALESCE((subscription_status_cd = 'INVALID'), FALSE)"

YAML combines automatic and policy-informed rules under expectations without warn, drop, or fail groups. It does not call the action classifier. Use JSON when the pipeline needs those action groups. Omit context to write only automatically discovered rules.

To run the flat YAML expectations in a Lakeflow Declarative Pipeline, add PyYAML to the pipeline's Python dependencies and load the expectations mapping:

import yaml
from pyspark import pipelines as dp

with open("/Volumes/catalog/schema/volume/telco_expectations.yaml", encoding="utf-8") as stream:
    rules_doc = yaml.safe_load(stream)

source_table = rules_doc.get("table") or "catalog.schema.telco_customer_subscription"

@dp.materialized_view(name="telco_customers_checked")
@dp.expect_all(rules_doc["expectations"])
def telco_customers_checked():
    return spark.read.table(source_table)

If you discovered from a DataFrame, set the fallback source_table to the table or transformation that produces the same columns and rows.

In this example, YAML rules are monitored with dp.expect_all: invalid rows remain in the output and their failures are recorded. Use the action-grouped JSON format if rules must drop rows or fail the pipeline update.

DataBuck.discover_rules(df) still returns automatic profiling rules without exporting. For compatibility, DataBuck.discover(df, context_dict) still returns BuckGPT rules without exporting. Pass an output file path as the second argument to DataBuck.discover(...) to combine discovery and export.

In JSON, each action dictionary maps expectation names to Spark SQL pass conditions, such as {"not_null_subscriber_id": "subscriber_id IS NOT NULL"}.

Export a Great Expectations suite

After installing databuck-sdk, install Great Expectations separately in the Databricks notebook:

%pip install "great-expectations>=1.0,<2"

Restart Python, set the SDK config and JAR paths again, then give DataBuck.discover a path ending in .gx.json:

gx_path = DataBuck.discover(
    "catalog.schema.loan_table",
    "/Volumes/catalog/schema/volume/loan_expectations.gx.json",
    context=context,
)
print(gx_path)

The GX file contains the selected HIGH and MEDIUM rules when domain_name is provided. With no domain, it contains all exportable rules. It includes dataset-level checks. Each row rule keeps its DataBuck action (warn, drop, or fail) in GX metadata. GX validation reports failures; it does not apply Lakeflow drop or fail actions to the data.

Metadata

Release files for databuck-sdk 0.5.57

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for databuck-sdk 0.5.57
File Size Uploaded Provenance Actions
databuck_sdk-0.5.57.tar.gz 65.4 kB With provenance Details for databuck_sdk-0.5.57.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for databuck-sdk 0.5.57
File Interpreter ABI Platform Actions
databuck_sdk-0.5.57-py3-none-any.whl Python 3 none any Details for databuck_sdk-0.5.57-py3-none-any.whl

Total release size: 110.2 kB

Release files / databuck_sdk-0.5.57.tar.gz

Download URL databuck_sdk-0.5.57.tar.gz
Size 65.4 kB
Tags Source
SHA-256 checksum
How to use checksums
955971172d6c1b51dd36fa3daad95e0bf911fb6997d73a6c1808202b32b00dcd
BLAKE2b-256 checksum
How to use checksums
658aa3353a83740d344bf3b803c769a4c73c60847c33e7a72dc259ff1d556515
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

Release files / databuck_sdk-0.5.57-py3-none-any.whl

Download URL databuck_sdk-0.5.57-py3-none-any.whl
Size 44.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ad40425f96f3917e2ffcf0d91a443000289dc205696b7b79e5db124899242355
BLAKE2b-256 checksum
How to use checksums
5716bea0df1ea41607417d427931119afe3d84aaecb712e90de260ef2bbcadf3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

Security for databuck-sdk 0.5.57

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for databuck_sdk-0.5.57.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.5.57 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page