dcs-sdk 1.8.4

SDK for DataChecks

DCS SDK v1.8.4

SDK for DataChecks

Installation

Python version >=3.10,<3.13

$ pip install dcs-sdk[all-dbs]

Supported Databases

Availability Status

Database	Code Name	Supported
PostgreSQL	postgres	✅
Snowflake	snowflake	✅
Trino	trino	✅
Databricks	databricks	✅
Oracle	oracle	✅
MSSQL	mssql	✅
MySQL	mysql	✅
SAP Sybase IQ/ASE	sybase	✅
File	file	✅
BigQuery	bigquery	✅

Available Commands

Option	Short Option	Required	Default	Description	Example
--config-path	-C	Yes	None	Specify the file path for the configuration	dcs-sdk run --config-path config.yaml --compare comp_name
--compare		Yes	None	Run only specific comparison using comparison name	dcs-sdk run --config-path config.yaml --compare comp_name
--save-json	-j	No	False	Save the data into a JSON file	dcs-sdk run --config-path config.yaml --compare comp_name --save-json
--json-path	-jp	No	dcs_report.json	Specify the file path for JSON file	dcs-sdk run --config-path config.yaml --compare comp_name --save-json --json-path ouput.json
--stats		No	False	Print stats about data diff	dcs-sdk run --config-path config.yaml --compare comp_name --stats
--url		No	None	Specify url to send data to server	dcs-sdk run --config-path config.yaml --compare comp_name --url=https://comapre/send/data
--html-report		No	False	Save table as HTML	dcs-sdk run --config-path config.yaml --compare comp_name --html-report
--report-path		No	dcs_report.html	Specify the file path for HTML report	dcs-sdk run --config-path config.yaml --compare comp_name --html-report --report-path table.html
--table		No	False	Display Comparison in table format	dcs-sdk run --config-path config.yaml --compare comp_name --html-report --report-path table.html --table

Example Command [CLI]

$ dcs-sdk --version

$ dcs-sdk --help

$ dcs-sdk run -C example.yaml --compare comparison_one --stats -j -jp output.json --html-report --report-path result.html --table --url=https://comapre/send/data

File Comparisons

dcs-sdk supports file-backed comparisons through DuckDB for:

• .csv
• .parquet
• mixed-format comparisons such as csv ↔ parquet

Supported file datasource types:

• file
• azure_blob

Notes:

• File paths must point to concrete .csv or .parquet files or globs.
• Query-backed file comparisons are supported. When source.query or target.query is provided, the SDK loads the file into DuckDB and compares against the filtered/projected query view.

Local File Example

data_sources:
  - name: source_file
    type: file
    file_path: sample_data/parquet/one_source.parquet

  - name: target_file
    type: file
    file_path: sample_data/parquet/two_target.parquet

comparisons:
  parquet_file_diff:
    source:
      data_source: source_file
      table: one_source
    target:
      data_source: target_file
      table: two_target
    key_columns: [id]
    columns: [customer_name, status, amount, region]

Run it with:

dcs-sdk run -C parquet_file_comparison.yaml --compare parquet_file_diff --stats

Databricks Query-Backed Comparisons

Databricks comparisons can use either:

• a table name
• a SQL query

For Parquet files stored on Databricks, use a query with read_files(...).

Databricks Table vs Parquet Example

data_sources:
  - name: databricks_demo
    type: databricks
    connection:
      host: your-workspace.cloud.databricks.com
      port: 443
      http_path: /sql/1.0/warehouses/your-warehouse
      access_token: ${DATABRICKS_TOKEN}
      catalog: dcs_demo_databricks
      schema: source
    temporary_schema: temp_schema

comparisons:
  databricks_table_vs_parquet:
    source:
      data_source: databricks_demo
      table: source_table
    target:
      data_source: databricks_demo
      query: |
        SELECT *
        FROM read_files(
          '/Volumes/dcs_demo_databricks/source/dcs-test-volumne/two_target.parquet',
          format => 'parquet'
        )
      view_name: datachecks_target_file
      materialization_type: table
    key_columns: [id]
    columns: [customer_name, status, amount, region]

Notes:

• Query-backed Databricks comparisons require temporary_schema.
• Generated temp views/tables use the datachecks_ prefix.
• Prefer Unity Catalog volume paths such as /Volumes/... for Databricks file queries.
• Legacy DBFS root paths such as dbfs:/raw/... are not the recommended path for this flow.