DCS SDK v1.9.5
SDK for DataChecks
Installation
Python version
>=3.10,<3.13
$ pip install dcs-sdk[all-dbs]
Supported Databases
Availability Status
| Database | Code Name | Supported |
|---|---|---|
| PostgreSQL | postgres |
✅ |
| Snowflake | snowflake |
✅ |
| Trino | trino |
✅ |
| Presto | presto |
✅ |
| Databricks | databricks |
✅ |
| Oracle | oracle |
✅ |
| MSSQL | mssql |
✅ |
| MySQL | mysql |
✅ |
| SAP Sybase IQ/ASE | sybase |
✅ |
| SAP HANA | sap_hana |
✅ |
| IBM DB2 | db2 |
✅ |
| Vertica | vertica |
✅ |
| ClickHouse | clickhouse |
✅ |
| File | file |
✅ |
| BigQuery | bigquery |
✅ |
Available Commands
| Option | Short Option | Required | Default | Description | Example |
|---|---|---|---|---|---|
| --config-path | -C | Yes | None | Specify the file path for the configuration | dcs-sdk run --config-path config.yaml --compare comp_name |
| --compare | Yes | None | Run only specific comparison using comparison name | dcs-sdk run --config-path config.yaml --compare comp_name | |
| --save-json | -j | No | False | Save the data into a JSON file | dcs-sdk run --config-path config.yaml --compare comp_name --save-json |
| --json-path | -jp | No | dcs_report.json | Specify the file path for JSON file | dcs-sdk run --config-path config.yaml --compare comp_name --save-json --json-path ouput.json |
| --stats | No | False | Print stats about data diff | dcs-sdk run --config-path config.yaml --compare comp_name --stats | |
| --url | No | None | Specify url to send data to server | dcs-sdk run --config-path config.yaml --compare comp_name --url=https://comapre/send/data | |
| --html-report | No | False | Save table as HTML | dcs-sdk run --config-path config.yaml --compare comp_name --html-report | |
| --report-path | No | dcs_report.html | Specify the file path for HTML report | dcs-sdk run --config-path config.yaml --compare comp_name --html-report --report-path table.html | |
| --table | No | False | Display Comparison in table format | dcs-sdk run --config-path config.yaml --compare comp_name --html-report --report-path table.html --table |
Example Command [CLI]
$ dcs-sdk --version
$ dcs-sdk --help
$ dcs-sdk run -C example.yaml --compare comparison_one --stats -j -jp output.json --html-report --report-path result.html --table --url=https://comapre/send/data
File Comparisons
dcs-sdk supports file-backed comparisons through DuckDB for:
.csv.parquet- mixed-format comparisons such as
csv ↔ parquet
Supported file datasource types:
fileazure_blob
Notes:
- File paths must point to concrete
.csvor.parquetfiles or globs. - Query-backed file comparisons are supported. When
source.queryortarget.queryis provided, the SDK loads the file into DuckDB and compares against the filtered/projected query view.
Local File Example
data_sources:
- name: source_file
type: file
file_path: sample_data/parquet/one_source.parquet
- name: target_file
type: file
file_path: sample_data/parquet/two_target.parquet
comparisons:
parquet_file_diff:
source:
data_source: source_file
table: one_source
target:
data_source: target_file
table: two_target
key_columns: [id]
columns: [customer_name, status, amount, region]
Run it with:
dcs-sdk run -C parquet_file_comparison.yaml --compare parquet_file_diff --stats
Databricks Query-Backed Comparisons
Databricks comparisons can use either:
- a table name
- a SQL query
For Parquet files stored on Databricks, use a query with read_files(...).
Databricks Table vs Parquet Example
data_sources:
- name: databricks_demo
type: databricks
connection:
host: your-workspace.cloud.databricks.com
port: 443
http_path: /sql/1.0/warehouses/your-warehouse
access_token: ${DATABRICKS_TOKEN}
catalog: dcs_demo_databricks
schema: source
temporary_schema: temp_schema
comparisons:
databricks_table_vs_parquet:
source:
data_source: databricks_demo
table: source_table
target:
data_source: databricks_demo
query: |
SELECT *
FROM read_files(
'/Volumes/dcs_demo_databricks/source/dcs-test-volumne/two_target.parquet',
format => 'parquet'
)
view_name: datachecks_target_file
materialization_type: table
key_columns: [id]
columns: [customer_name, status, amount, region]
Notes:
- Query-backed Databricks comparisons require
temporary_schema. - Generated temp views/tables use the
datachecks_prefix. - Prefer Unity Catalog volume paths such as
/Volumes/...for Databricks file queries. - Legacy DBFS root paths such as
dbfs:/raw/...are not the recommended path for this flow.
1 maintainer
Release files for dcs-sdk 1.9.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| dcs_sdk-1.9.6.tar.gz | 1.4 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| dcs_sdk-1.9.6-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 2.9 MB
Release files / dcs_sdk-1.9.6.tar.gz
| Download URL | dcs_sdk-1.9.6.tar.gz |
|---|---|
| Size | 1.4 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
78c4554000ed5e18eca5e9089bcb4ced753ef2a3bd73c39c5ca526cee8d43ace
|
|
BLAKE2b-256 checksum How to use checksums |
80186cd880b0a9355e852e41d2098c246aaa64e8eeb2c0ec9eb5560e3e841964
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.12.13 Linux/6.17.0-1020-azure
|
Release files / dcs_sdk-1.9.6-py3-none-any.whl
| Download URL | dcs_sdk-1.9.6-py3-none-any.whl |
|---|---|
| Size | 1.5 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7afb441e5da54dcf1ad9b4b8f5dd081d14079a959974488cf52f9db27924c053
|
|
BLAKE2b-256 checksum How to use checksums |
ad3b534de24b289fece12e0c6347f5fa200d071884af0dcb31a37d51a7911627
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.12.13 Linux/6.17.0-1020-azure
|
