databuck-sdk
Python wrapper for the DataBuck Spark SDK. It discovers data-quality rules from an existing Spark DataFrame or a registered Spark table and exports expectations that Databricks Lakeflow Declarative Pipelines can execute before downstream data is published.
Install
python -m pip install databuck-sdk
Give each customer one signed databucksdk.properties file:
customer_id=your-customer-id
issued_on=2026-11-02
expires_on=2027-12-31
license_key=signed-key-created-by-the-issuer
Set DATABUCK_SDK_CONFIG_FILE to the full local file path before importing the
SDK. If the file is absent, import downloads databucksdk.properties from the
DataBuck S3 URL to that path and validates its signature and dates. An existing
file is reused. Without a configured path, the SDK uses
databuck/jars/databucksdk.properties in the installed package.
The license is valid through its expires_on date, inclusive. Renew it with
a newly issued config file when it expires. Editing the dates or customer ID
invalidates the signature.
import os
os.environ["DATABUCK_SDK_CONFIG_FILE"] = "/path/to/databucksdk.properties"
from databuck import DataBuck
If databuck-spark-sdk is installed, remove it before installing this package;
both distributions provide the same databuck Python import:
python -m pip uninstall databuck-spark-sdk
python -m pip install --upgrade databuck-sdk
Importing databuck first downloads the signed properties file if needed,
validates it, and then downloads databuck-sdk.jar from the DataBuck S3
URL to databuck/jars/ in the installed Python package, with progress output.
The downloaded path is set in DATABUCK_SDK_JAR for the current Python
process. A later import reuses the existing JAR.
The JAR is not included in the wheel because it exceeds PyPI's
default per-file upload limit. An internet connection and write access to the
installed package directory are required for the first download.
python -m databuck is an explicit download command that also prints the
local path. Set DATABUCK_SDK_AUTO_DOWNLOAD=0 before import to defer both
downloads when offline; a valid local properties file is still required.
DataBuck.jar_path() returns the path after download.
Downloading the JAR does not install it as a Databricks compute library.
To use an existing JAR or another writable destination, set
DATABUCK_SDK_JAR to that file path. If the download URL changes, set
DATABUCK_SDK_JAR_URL to the new HTTPS URL before importing or running
python -m databuck. The download checks that the file is a valid JAR archive.
Use a trusted JAR URL because the package does not verify a checksum.
Set DATABUCK_SDK_CONFIG_FILE_URL before import if the signed properties file
must come from another trusted URL.
Licensing
The SDK requires a signed databucksdk.properties file. It checks the customer
ID, issue date, expiry date, and Ed25519 signature at import and before public
SDK operations. The file remains valid through its expiry date, inclusive.
See the license notice for the repository's licensing status and
contact DataBuck for customer terms or renewal.
Databricks notebook setup
The SDK notebook was run on Databricks Runtime 13.3 ML (Spark 3.4.1, Scala 2.12). Check attached connector libraries separately if Spark fails while loading a data source.
Run this once in the notebook to install the Python package:
%pip install databuck-sdk
After installing or upgrading, restart Python in the next cell so the notebook loads the new package version:
dbutils.library.restartPython()
Then verify the version and set a writable Unity Catalog Volume file path
before importing databuck:
import os
from importlib.metadata import version
print(version("databuck-sdk"))
os.environ["DATABUCK_SDK_JAR"] = (
"/Volumes/catalog/schema/volume/databuck-sdk.jar"
)
os.environ["DATABUCK_SDK_CONFIG_FILE"] = (
"/Volumes/catalog/schema/volume/databucksdk.properties"
)
from databuck import DataBuck # Downloads the JAR to that path on first import.
print(DataBuck.jar_path())
Use the full filename ending in .jar, not just the Volume directory. Importing
databuck downloads or reuses the JAR at that path. Setting the variable after
the import is too late to choose the initial download location. A Volume keeps
the JAR available beyond the current notebook driver session.
Before calling DataBuck.discover(...), attach the downloaded JAR to the
compute that runs discovery: in Databricks, open Compute > select that
compute > Libraries > Install New, select the Volume as the library
source, choose the JAR path printed above, and install it. If the compute is
already running, restart it or reattach the notebook so Spark loads the new
library. Then read your Spark DataFrame and run the discovery example below.
If the old databuck-spark-sdk.jar is attached to that compute, remove it
before attaching databuck-sdk.jar.
The Python import downloads the file; it does not attach the JAR to compute.
See Databricks compute-scoped library instructions.
Local PySpark usage
from pyspark.sql import SparkSession
import os
from databuck import DataBuck
spark = (
SparkSession.builder
.config("spark.jars", DataBuck.jar_path())
.getOrCreate()
)
df = spark.read.csv("customers.csv", header=True)
print(DataBuck.count(df))
Discover and export rules for Databricks
Run discovery in a Databricks notebook or Spark job against an existing DataFrame,
or pass a Spark table name and the SDK will read it into a DataFrame internally.
Pass PDF and DOCX policy files for context-aware BuckGPT rules.
The SDK exports generated rules as JSON (.json), YAML (.yaml / .yml),
or a Great Expectations suite (.gx.json).
A Lakeflow Declarative Pipeline then loads and executes those rules as
expectations. Choose one of the two JSON inputs below; both use the same
discovery and export flow. The YAML workflow appears below.
from databuck import DataBuck
context = {
"gemini_api_key": "YOUR_GEMINI_API_KEY",
"domain_name": "banking",
"pdf_paths": [
"/Volumes/catalog/schema/volume/customer_policy.pdf",
"/Volumes/catalog/schema/volume/data_contract.docx",
],
}
# Option 1: use a DataFrame already available in your pipeline.
json_path = DataBuck.discover(
df, "/Volumes/catalog/schema/volume/expectations.json", context=context
)
print(json_path)
Or pass a registered Spark table name. The SDK calls spark.read.table(...)
internally, then passes that DataFrame through the same discovery and export:
json_path = DataBuck.discover(
"catalog.schema.telco_customer_subscription",
"/Volumes/catalog/schema/volume/expectations.json",
context=context,
)
print(json_path)
Replace the example Volume paths and table name with files and a table available in your workspace. The table-name option uses the notebook's Spark session and requires read access to the registered table; it does not take separate database connection details. Passing an existing DataFrame avoids reading the table again.
The call combines automatic and policy-informed rules. The output extension
selects JSON or YAML. With domain_name, JSON and YAML contain selected HIGH
and MEDIUM rules from both sources.
pdf_paths accepts PDF and DOCX files available to the notebook. For a
dataset-level uniqueness rule, the output includes a dataset_checks section.
Run its SQL against the full dataset separately, replacing {{DATAFRAME}} with
your source table or view. Dataset checks are not Lakeflow row expectations.
To export only automatic profiling rules, omit context:
json_path = DataBuck.discover(
df, "/Volumes/catalog/schema/volume/expectations.json"
)
Without context, the output contains automatic rules and JSON places them in
the warn group. To include policy-informed rules, pass a business statement
or policy document along with the Gemini API key.
The JSON contains three action dictionaries: warn, drop, and fail.
In a Databricks Lakeflow Declarative Pipeline, use the dictionaries with
dp.expect_all, dp.expect_all_or_drop, and dp.expect_all_or_fail,
respectively.
For example, add this to the Lakeflow Declarative Pipeline source file. The expectations run when the pipeline updates; the SDK's discovery call above generates the file beforehand:
import json
from pyspark import pipelines as dp
with open("/Volumes/catalog/schema/volume/expectations.json", encoding="utf-8") as stream:
expectations = json.load(stream)
@dp.materialized_view(name="telco_customers_checked")
@dp.expect_all(expectations["warn"])
@dp.expect_all_or_drop(expectations["drop"])
@dp.expect_all_or_fail(expectations["fail"])
def customers_checked():
return spark.read.table("catalog.schema.telco_customer_subscription")
What the pipeline results look like
DataBuck.discover(df, output_path, ...) prints the rules and returns the path to the rules
file. The row results appear after the Lakeflow Declarative Pipeline runs,
in the validated table's Expectations view. For example, one telco demo run
showed:
| Pipeline result | Records |
|---|---|
| Written | 58 (58%) |
| Dropped | 42 (42%) |
| Expectation | Action shown in Databricks | Failed records |
|---|---|---|
BuckGPT_Rule_006 |
ALLOW | 60 |
BuckGPT_Rule_007 |
DROP | 40 |
BuckGPT_Rule_009 |
DROP | 20 |
valid_length_network_type |
DROP | 4 |
valid_pattern_network_type |
DROP | 4 |
ALLOW corresponds to dp.expect_all: the violation is recorded without
dropping the row. DROP corresponds to dp.expect_all_or_drop: invalid rows
are excluded from the validated output. A FAIL expectation stops the pipeline
update when violated. A row can fail more than one rule, so the per-rule failed
record counts do not add up to the dropped-row total. These numbers are from one
demo update; actual results depend on the source data and exported rules.
The pipeline must read a DataFrame with the columns used by the exported expectations. Regenerate the rules file when the source schema or business policy changes, then refresh the pipeline.
{
"warn": {"not_null_customer_email": "`customer_email` IS NOT NULL"},
"drop": {"not_null_order_id": "`order_id` IS NOT NULL"},
"fail": {"valid_pattern_currency_code": "`currency_code` IS NULL OR CAST(`currency_code` AS STRING) RLIKE '(?:^[A-Z][A-Z][A-Z]$)'"}
}
The example shows the file shape; actual actions depend on the generated rules.
Export flat YAML expectations
Pass a .yaml or .yml output file to write a flat YAML document. Pass either
an existing DataFrame or a table name. Choose one of these calls:
from databuck import DataBuck
context = {
"gemini_api_key": "YOUR_GEMINI_API_KEY",
"pdf_paths": [
"/Volumes/catalog/schema/volume/customer_policy.pdf",
"/Volumes/catalog/schema/volume/data_contract.docx",
],
}
# Option 1: use a DataFrame already available in your pipeline.
yaml_path = DataBuck.discover(
df,
"/Volumes/catalog/schema/volume/telco_expectations.yaml",
context=context,
)
print(yaml_path)
Or pass a registered Spark table name. The SDK reads it into a DataFrame internally before discovering and exporting the rules:
yaml_path = DataBuck.discover(
"catalog.schema.telco_customer_subscription",
"/Volumes/catalog/schema/volume/telco_expectations.yaml",
context=context,
)
print(yaml_path)
No separate table_name= argument is needed. When you pass a table name, YAML
includes a table: field. When you pass a DataFrame, YAML contains
expectations: without a table: field because the original table name is
not reliably available from a DataFrame.
For example, the generated file has this shape (the names and SQL depend on the discovered rules):
table: catalog.schema.telco_customer_subscription
expectations:
not_null_subscriber_id: "`subscriber_id` IS NOT NULL"
BuckGPT_Rule_001: "NOT COALESCE((subscription_status_cd = 'INVALID'), FALSE)"
YAML combines automatic and policy-informed rules under expectations without
warn, drop, or fail groups. It does not call the action classifier. Use
JSON when the pipeline needs those action groups.
Omit context to write only automatically discovered rules.
To run the flat YAML expectations in a Lakeflow Declarative Pipeline, add
PyYAML to the pipeline's Python dependencies and load the expectations
mapping:
import yaml
from pyspark import pipelines as dp
with open("/Volumes/catalog/schema/volume/telco_expectations.yaml", encoding="utf-8") as stream:
rules_doc = yaml.safe_load(stream)
source_table = rules_doc.get("table") or "catalog.schema.telco_customer_subscription"
@dp.materialized_view(name="telco_customers_checked")
@dp.expect_all(rules_doc["expectations"])
def telco_customers_checked():
return spark.read.table(source_table)
If you discovered from a DataFrame, set the fallback source_table to the
table or transformation that produces the same columns and rows.
In this example, YAML rules are monitored with dp.expect_all: invalid rows
remain in the output and their failures are recorded. Use the action-grouped
JSON format if rules must drop rows or fail the pipeline update.
DataBuck.discover_rules(df) still returns automatic profiling rules without
exporting. For compatibility, DataBuck.discover(df, context_dict) still
returns BuckGPT rules without exporting. Pass an output file path as the second
argument to DataBuck.discover(...) to combine discovery and export.
In JSON, each action dictionary maps expectation names to Spark SQL pass
conditions, such as {"not_null_subscriber_id": "subscriber_id IS NOT NULL"}.
Export a Great Expectations suite
After installing databuck-sdk, install Great Expectations separately in the
Databricks notebook:
%pip install "great-expectations>=1.0,<2"
Restart Python, set the SDK config and JAR paths again, then give
DataBuck.discover a path ending in .gx.json:
gx_path = DataBuck.discover(
"catalog.schema.loan_table",
"/Volumes/catalog/schema/volume/loan_expectations.gx.json",
context=context,
)
print(gx_path)
The GX file contains the selected HIGH and MEDIUM rules when domain_name is
provided. With no domain, it contains all exportable rules. It includes
dataset-level checks. Each row rule keeps its DataBuck action (warn, drop,
or fail) in GX metadata. GX validation reports failures; it does not apply
Lakeflow drop or fail actions to the data.
Metadata
Release files for databuck-sdk 0.5.57
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| databuck_sdk-0.5.57.tar.gz | 65.4 kB | With provenance | Details for databuck_sdk-0.5.57.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| databuck_sdk-0.5.57-py3-none-any.whl | Python 3 | none | any | Details for databuck_sdk-0.5.57-py3-none-any.whl |
Total release size: 110.2 kB
Release files / databuck_sdk-0.5.57.tar.gz
| Download URL | databuck_sdk-0.5.57.tar.gz |
|---|---|
| Size | 65.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
955971172d6c1b51dd36fa3daad95e0bf911fb6997d73a6c1808202b32b00dcd
|
|
BLAKE2b-256 checksum How to use checksums |
658aa3353a83740d344bf3b803c769a4c73c60847c33e7a72dc259ff1d556515
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logRelease files / databuck_sdk-0.5.57-py3-none-any.whl
| Download URL | databuck_sdk-0.5.57-py3-none-any.whl |
|---|---|
| Size | 44.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ad40425f96f3917e2ffcf0d91a443000289dc205696b7b79e5db124899242355
|
|
BLAKE2b-256 checksum How to use checksums |
5716bea0df1ea41607417d427931119afe3d84aaecb712e90de260ef2bbcadf3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logSecurity for databuck-sdk 0.5.57
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for databuck_sdk-0.5.57.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.