snowflake-data-migration-orchestrator 1.26.0

Data migration orchestrator for Snowflake

Snowflake Data Migration Orchestrator

The Cloud Data Migration feature of SnowConvert provides a fault-tolerant, scalable solution for moving data from external sources into Snowflake. This tool is specifically designed for cases where a user is moving data from a system they plan to decommission. For replication purposes, other solutions are available that might better fit your use case.

Architecture

• 1 Orchestrator is connected to the Snowflake Account.
  • It requires privileges to create/operate the SNOWCONVERT_AI database, in which metadata is stored.
• 1 or more Workers connect to the Source System and to the Snowflake Account.
  • Workers read data from the Source System and upload it to a Snowflake Stage.
  • Workers pick up tasks created by the Orchestrator and process them in parallel.
• Files uploaded to the Snowflake Stage are copied into the Target Tables using a COPY INTO statement.
  • The COPY INTO statement is submitted and monitored by the Orchestrator.

Where to Deploy Orchestrator and Worker(s)?

The Orchestrator and Worker(s) can be deployed in multiple ways:

1. Both on Snowpark Container Services (in the Snowflake Account).
2. Both on the Customer's Environment (custom hardware, virtual machines, containers, etc.).
3. Orchestrator on Snowpark Container Services and Worker(s) on the Customer's Environment (or the other way around).

Requirements for the environment:

• The Orchestrator and Worker(s) are Python packages, so Python must be installed.
• The Worker(s) will typically require an ODBC driver to connect to the Source System.
• The Orchestrator needs to be able to connect to the Snowflake Account. The connection used must have privileges to create the SNOWCONVERT_AI database and create schemas/objects on that database.

Setup

Additional Configuration on Snowflake Account

When starting the Orchestrator, it will automatically try to set up resources in your Snowflake Account in the SNOWCONVERT_AI database (if it does not exist yet, it will be created). This is a one-time step and transparent to the user. Some considerations:

• The Orchestrator should connect with a role that has privileges to create the SNOWCONVERT_AI database and its objects.
• Whenever the Orchestrator starts, it should use a role that allows it to interact with the SNOWCONVERT_AI database and its resources. The easiest way to guarantee this is to always run it with the same role that was used for creating SNOWCONVERT_AI in the first place.

Usage

In general, for migrating data using this solution, you will need to:

1. Start the Orchestrator.
2. Start the Worker(s).
3. Create a Data Migration Workflow.
4. Monitor the Data Migration Workflow (asynchronously) until completion.

A Data Migration Workflow is essentially an action/goal for the system to complete, such as migrating a specific set of tables with a given configuration. You can submit multiple workflows simultaneously and monitor them. The Orchestrator breaks Data Migration Workflows into smaller tasks. Normally, this also involves splitting a table into partitions before extracting its data and loading it to Snowflake.

Starting the Orchestrator

After installation, start the Orchestrator by running:

python -m data_migration_orchestrator start

When invoked without a subcommand, start is assumed for backward compatibility.

The start command accepts optional flags:

Flag	Default	Description
--log-destination	both	Where to send logs: stdout, file, or both.
--log-file	logs/data_migration_orchestrator.log	Path to the log file (used when destination includes file).
--log-level	INFO	Logging level: DEBUG, INFO, WARNING, ERROR, or CRITICAL.

You can also set the DATA_MIGRATION_ORCHESTRATOR_LOGS_DIR environment variable to write the log file to a specific directory (the default filename is used) — convenient in containers/SPCS where env vars are the norm. Precedence: --log-file (explicit file path) > DATA_MIGRATION_ORCHESTRATOR_LOGS_DIR (directory) > the default logs/ under the working directory. A leading ~ is expanded and the path is resolved to an absolute location.

Before running, make sure that the SNOWFLAKE_CONNECTION_NAME environment variable is set to a value that matches one of the connection names in your Snowflake config.toml or connections.toml. That is the name of the connection used to connect to the Target Snowflake Account.

Connection session defaults use SNOWFLAKE_DATABASE and SNOWFLAKE_SCHEMA where your environment provides them (for example in SPCS). Those are separate from metadata object locations: by default, workflow and task-queue objects live in SNOWCONVERT_AI.DATA_MIGRATION and data-validation objects in SNOWCONVERT_AI.DATA_VALIDATION. To deploy metadata under different database or schema names, set:

• CUSTOM_SNOWFLAKE_DATABASE_FOR_METADATA (default SNOWCONVERT_AI)
• CUSTOM_SNOWFLAKE_SCHEMA_FOR_DATA_MIGRATION_METADATA (default DATA_MIGRATION)
• CUSTOM_SNOWFLAKE_SCHEMA_FOR_DATA_VALIDATION_METADATA (default DATA_VALIDATION)

Metadata storage mode (Hybrid vs standard): On bootstrap, the orchestrator probes whether the Snowflake account supports Hybrid Tables and chooses DDL accordingly (HYBRID TABLE vs TRANSIENT, task-queue pull semantics, metadata MERGE idempotency). Override the probe with:

• SNOWFLAKE_USE_HYBRID_TABLES=1 — force hybrid metadata DDL (production-capable accounts).
• SNOWFLAKE_USE_HYBRID_TABLES=0 — force standard/TRANSIENT metadata DDL (trial accounts without hybrid support).

When unset, the orchestrator runs a one-time DDL probe at SchemaManager.migrate() and caches the result for the process lifetime. If an existing schema already has a hybrid TASK_QUEUE but the resolved mode is standard, bootstrap fails fast with an actionable error (use SNOWFLAKE_USE_HYBRID_TABLES=1 or a fresh schema). See docs/NonHybridModeImplementationSpec.md.

Metadata profile triage (support / on-call)

Phase 2 records deployment mode in SCHEMA_DEPLOYMENT_PROFILE (warehouse table in the metadata schema). Use this when Datadog or bootstrap logs look inconsistent with expectations:

Signal	What it means	Action
SCHEMA_DEPLOYMENT_PROFILE row	Insert-only audit history (at most one valid row). Not updated when env overrides disagree.	Do not treat SELECT * alone as the live runtime mode when SNOWFLAKE_USE_HYBRID_TABLES is set.
WARNING — env overrides stored profile	Route A forced a mode different from the warehouse row (metadata_storage_mode=… in logs).	Verify deployment env / pipeline vars; confirm intentional override. Profile row stays historical.
MetadataStorageModeMismatchError (§8)	Resolved mode incompatible with physical TASK_QUEUE DDL (GET_DDL).	Fix env vs schema mismatch, or use a fresh metadata schema.
MetadataStorageModePermissionError (§8, hybrid or standard)	Role lacks GET_DDL on TASK_QUEUE during SchemaManager.migrate(). Required on every migrate for the §8 guard (bidirectional matrix).	Grant GET_DDL (or equivalent metadata read) on TASK_QUEUE in the migration schema, or use a role that can classify existing DDL.
InvalidSchemaDeploymentProfileError	Corrupt profile enums or multiple profile rows (concurrent greenfield insert).	Inspect profile table; fix manual edits or redeploy on clean schema.
Mode changed via env without restart	Resolution is cached per process for the orchestrator lifetime.	Restart the pod/process after changing SNOWFLAKE_USE_HYBRID_TABLES or metadata DB env vars.

Manual smoke/worksheet runbooks: tests/manual/README_SMOKE_TRIAL_AND_HYBRID.md, tests/manual/WORKSHEET_BLIND_SPOTS.md.

Workers that call task-queue stored procedures must use the same CUSTOM_* values as the orchestrator.

The Orchestrator will run until you stop it. Data Migration Workflows need an active Orchestrator to be completed. However, the Orchestrator can be safely stopped at any point and resumed later (ongoing Data Migration Workflows will be resumed at that point).

Starting the Worker(s)

After installation (pip install snowflake-data-exchange-agent, or pip install snowflake-data-exchange-agent[teradata] when the Worker connects to Teradata with the native driver), start a Worker by running:

data-exchange-agent run -c <configuration-file-path>

The -c flag can be omitted; in that case, the worker will look for a file called configuration.toml in your current directory. When invoked without a subcommand (data-exchange-agent -c ...), run is assumed for backward compatibility. See the Worker Configuration section below for the full specification.

You can also verify connectivity before starting:

data-exchange-agent test -c <configuration-file-path>

This executes SELECT 1 on every configured source and target connection and reports the results.

Workers will run until you stop them. Data Migration Workflows and Cloud Data Validation Workflows need at least one active Worker to be completed. However, the Workers can be safely stopped at any point and resumed later (ongoing workflows will be resumed at that point).

Creating a Data Migration Workflow

After installation, create workflows by running:

python -m data_migration_orchestrator create-data-migration-workflow <workflow-config-file-path> --source-platform <source-platform> [--name <workflow-name>] [--connection-name <connection-name>]

• The Workflow Configuration specification can be found in the Workflow Configuration Reference section.
• --source-platform is required for this subcommand. Supported values are sqlserver, redshift, oracle, teradata, postgresql, azure_synapse, and azure_synapse_serverless.
• The workflow name must be composed of alphanumerical characters and cannot start with a digit. Defaults to MY_WORKFLOW when omitted.
• --connection-name is optional. When omitted, the orchestrator uses the default Snowflake connection from environment variables. When provided, it should match a named connection in your config.toml or connections.toml file.

Monitoring a Data Migration Workflow

Each Workflow will go through different stages through its lifecycle:

1. Pending: No tasks have been created for this workflow yet.
2. Executing: Tasks have been created for this workflow and there are still tasks that haven't reached a terminal state (COMPLETED or FAILED).
3. Completed: All tasks have reached a terminal state (COMPLETED or FAILED).

In the data migration metadata schema (by default SNOWCONVERT_AI.DATA_MIGRATION) there are tables/views that can be queried to understand the status of one or more Workflows:

View/Table	Description
WORKFLOW	One row per workflow. Includes start/end time, status, and configuration.
TABLE_PROGRESS_WITH_EXAMPLE_ERROR	One row per table being migrated. Shows how many partitions are in each stage (extraction, loading, completed, or failed), along with related errors. Filterable by WORKFLOW_ID.
DATA_MIGRATION_ERROR	For each failed partition, contains the first known error. Filterable by WORKFLOW_ID.
DATA_MIGRATION_WARNING	Non-fatal warnings emitted during migration (e.g. type fallbacks, truncated columns). Filterable by WORKFLOW_ID.

In the same schema, there is a Streamlit dashboard called DATA_MIGRATION_DASHBOARD that can be used to monitor the workflows. The dashboard is organized around tables (the primary user-facing concept) rather than workflow executions. Its default tabs are:

• 📋 Tables — primary view. One row per distinct table aggregated across every workflow that migrated it, with drill-down to per-execution detail. Surfaces per-table TOTAL_ROWS and the target CLOUD_STAGE bucket.
• ⚠️ Errors — table-grouped expanders (and a flat-list fallback) over recent migration errors.
• 📊 Overview — high-level KPIs including total rows migrated.

Every tab has a ⬇ Download CSV button for its on-screen dataframe, and the sidebar exposes a 📦 Export snapshot (CSV) that bundles every active section into one multi-section file. The sidebar table search prefilters every table-driven view. Toggle Show advanced / debug views in the sidebar to reveal the Workflows and Tasks (task queue) debug tabs plus raw internal columns.

Managing Workflows

The migration metadata schema (default SNOWCONVERT_AI.DATA_MIGRATION) exposes stored procedures for pausing, resuming, and cancelling work. They can be called directly from a Snowflake worksheet or any SQL client.

Pause

Pausing moves all pending and executing tasks to paused status and revokes any active leases. Paused tasks cannot be picked up by executors until they are resumed.

-- Pause an entire workflow
CALL DATA_MIGRATION.PAUSE_WORKFLOW(<workflow_id>);

-- Pause a single table within a workflow
CALL DATA_MIGRATION.PAUSE_TABLE(<workflow_id>, '<TABLE_SOURCE_IDENTIFIER>');

PAUSE_WORKFLOW also sets the workflow status to paused. PAUSE_TABLE leaves the workflow status unchanged.

Resume

Resuming moves all paused tasks back to pending so executors can pick them up again.

-- Resume an entire workflow
CALL DATA_MIGRATION.RESUME_WORKFLOW(<workflow_id>);

-- Resume a single table within a workflow
CALL DATA_MIGRATION.RESUME_TABLE(<workflow_id>, '<TABLE_SOURCE_IDENTIFIER>');

RESUME_WORKFLOW also sets the workflow status back to running.

Cancel

Cancelling moves all pending, executing, and paused tasks to failed with the error message Manually cancelled.. The failure is cascaded to any blocked successor tasks, and cleanup tasks are unblocked so they can run.

-- Cancel an entire workflow
CALL DATA_MIGRATION.CANCEL_WORKFLOW(<workflow_id>);

-- Cancel a single table within a workflow
CALL DATA_MIGRATION.CANCEL_TABLE(<workflow_id>, '<TABLE_SOURCE_IDENTIFIER>');

CANCEL_WORKFLOW also sets the workflow status to cancelled.

Note: The TABLE_SOURCE_IDENTIFIER parameter is the fully qualified source table name as it appears in the task scope (e.g., MY_DB.MY_SCHEMA.MY_TABLE). You can find it in the SCOPE column of the TASK_QUEUE table inside the Table[...] fragment.

Cloud Data Validation Workflows

The orchestrator can run Cloud Data Validation workflows in addition to data migration. Validation work is queued as data_validation tasks; the same Worker package (snowflake-data-exchange-agent) executes them when the optional snowflake-data-validation dependency is available in the worker environment. Create a validation workflow with:

python -m data_migration_orchestrator create-data-validation-workflow <validation-config-file-path> --source-platform <source-platform> [--name <workflow-name>] [--connection-name <connection-name>]

• --source-platform is required for this subcommand. Supported values are sqlserver, redshift, teradata, oracle, and postgresql. Its value must match the source_platform field in the JSON configuration file.
• --name defaults to MY_VALIDATION_WORKFLOW when omitted.
• --connection-name is optional. When omitted, the orchestrator uses the default Snowflake connection from environment variables.
• New workflow rows are inserted into the WORKFLOW table in the data migration metadata schema (default SNOWCONVERT_AI.DATA_MIGRATION) with WORKFLOW_TYPE set to data-validation. Validation results and related objects are stored under the data validation metadata schema (default SNOWCONVERT_AI.DATA_VALIDATION, configurable with CUSTOM_SNOWFLAKE_SCHEMA_FOR_DATA_VALIDATION_METADATA).

Monitoring a Data Validation Workflow

In the data validation metadata schema (by default SNOWCONVERT_AI.DATA_VALIDATION) there are views that can be queried to understand the status of validation workflows:

View	Description
TABLE_PROGRESS	One row per validated table. Summarizes overall validation status. Filterable by WORKFLOW_ID.
TABLE_PROGRESS_DETAIL	Per-table breakdown with partition-level L2/L3 status (VALID, INVALID, EXECUTION_ERROR). Filterable by WORKFLOW_ID.
DATA_VALIDATION_ERROR	Errors encountered during validation. Filterable by WORKFLOW_ID.
DATA_VALIDATION_WARNING	Non-fatal warnings (e.g. unsupported column types, metric exclusions). Filterable by WORKFLOW_ID.

In the same schema, there is a Streamlit dashboard called DATA_VALIDATION_DASHBOARD that provides a visual overview of validation progress and results. Like the migration dashboard it is organized around tables: the 📋 Tables tab is the primary view (cross-workflow aggregate + per-table drill-down with quick-links that pre-filter Schema/Metrics/Rows/Cell tabs). The Schema / Metrics / Rows / Cell / Table progress / Errors tabs are retained; a sidebar Show advanced / debug views toggle reveals the Tasks (task queue) debug tab. Every tab has a ⬇ Download CSV button, and the sidebar exposes a 📦 Export snapshot (CSV) that bundles every active section into one file.

Validating Views

In addition to tables, Cloud Data Validation supports validating views. Place view entries in the top-level views array (same shape as tables). Entries under views are automatically tagged with object_type = "VIEW", so there is no need to set object_type explicitly on each entry; however, you can also set object_type to "VIEW" directly on a tables entry if you prefer a flat list.

Views go through the same L1 (schema), L2 (metrics), and L3 (row/cell/hybrid) validation pipeline as tables, with one platform-specific difference:

• Teradata views: L1 schema validation uses a basic comparison (column existence and datatype only) because Teradata exposes view column metadata through HELP COLUMN rather than DBC.Columns. Precision, scale, length, nullable, and ordinal checks are not available for Teradata views. L2 metrics and L3 row/cell validation are fully supported.
• Oracle views: Validated identically to Oracle tables at all levels. Oracle exposes view column metadata through ALL_TAB_COLUMNS the same way as table metadata, so no materialization or special handling is needed.
• Oracle identifiers: Workflow schemaName / tableName map to data-dictionary spelling for catalog queries; see oracle-identifiers.md.
• Other platforms: Views are validated identically to tables at all levels, since their catalogs expose view column metadata the same way as table metadata.

Partitioning (column_names_to_partition_by, target_partition_size_rows, target_partition_size_mb) works the same way for views as it does for tables. See the Partitioning section and the view validation example below.

Unified objects section (optional type auto-detection)

The top-level objects array is an additive alternative to tables / views that lets you list objects without committing to a type. Each entry uses the same shape as a tables / views entry, plus an optional objectType:

• When objectType is set to TABLE or VIEW, the entry behaves exactly like the corresponding tables / views entry.
• When objectType is omitted, the orchestrator resolves the type at runtime against the target Snowflake catalog. Because data validation always runs on Snowflake, a single INFORMATION_SCHEMA query shape works for every source platform — no Data Exchange Agent round-trip or staging is needed. A single dispatch task groups the untyped objects by their target (database, schema), issues one query per schema, then fans out one per-object check task carrying the resolved type, which records it before validation queries are generated.
• Only TABLE and VIEW are accepted. If detection finds something else (for example a materialized view) or the object does not exist on the target, only that object's check task fails; other objects continue.
• A detected type is treated exactly as if you had written objectType yourself — it feeds the same logic on every platform, so there is no special-case behavior for detected vs. configured types.

tables, views, and objects may be combined freely; the configuration must define at least one entry across all three. See example-data-validation-objects-workflow.json for an example mixing an explicitly-typed object and an auto-detected one.

Data Validation Workflow Configuration (Top Level)

Property	Type	Required	Description
source_platform	String	Yes	Source dialect identifier (for example sqlserver, redshift, teradata, oracle, postgresql). Must match the --source-platform argument when creating the workflow from the CLI.
target_platform	String	No	Defaults to Snowflake.
target_database	String	No	Default target database name for tables when not specified per table.
validation_configuration	Object	No	Global validation levels and options (see below).
comparison_configuration	Object	No	Numeric tolerance and optional type mapping file.
database_mappings	Object	No	Map of source database names to Snowflake database names.
schema_mappings	Object	No	Map of source schema names to Snowflake schema names.
tables	Array	No*	Table entries (each tagged object_type = "TABLE").
views	Array	No*	View entries using the same shape as tables (each tagged object_type = "VIEW").
objects	Array	No*	Unified entries with an optional objectType (TABLE/VIEW); omitting it triggers runtime type detection. See Unified objects section.
use_snowflake_compute	Boolean	No	When true, enables Snowflake-side computation paths where supported. Default false.
target_partition_size_rows	Integer	No	Desired rows per partition. Mutually exclusive with target_partition_size_mb. Must be greater than 0 when set. When both targets are omitted, Data Validation defaults to 200 MB per partition. See Partitioning below. Overridable per table.
target_partition_size_mb	Integer	No	Desired MB per partition. Mutually exclusive with target_partition_size_rows. Must be greater than 0 when set. When both targets are omitted, Data Validation defaults to 200 MB per partition. See Partitioning below. Overridable per table.
use_snowpipe_for_results	Boolean	No	When true (default), L2/L3 validation results are ingested into the shared results tables via Snowpipe. Workers issue ALTER PIPE REFRESH after uploading each partition's files and the orchestrator waits for SYSTEM$PIPE_STATUS to report zero pending files before running the evaluate step. Set to false to fall back to the legacy per-partition COPY INTO tasks.
acceptedTransformations	Object[]	No	Global list of accepted source-to-target value transformations applied to all tables. See acceptedTransformations.

* tables, views, and objects are each individually optional, but the configuration must define at least one entry across the three.

validation_configuration (global defaults)

When validation_configuration is omitted, the orchestrator applies these defaults: schema and metrics validation are enabled; row validation is disabled; continue_on_failure defaults to false; max_failed_rows_number defaults to 1000; row_validation_mode defaults to hybrid when L3 is enabled; early_stopping_for_row_hashing defaults to false; early_stopping_for_cell_by_cell_comparison defaults to true; exclude_metrics defaults to false; apply_metric_column_modifier defaults to true. When either early-stop flag is enabled and early_stop_check_interval_minutes is omitted, the parser defaults the interval to 5. At least one validation level must be enabled; configurations that disable schema, metrics, and row validation simultaneously are rejected at parse time. Legacy row and cell values for row_validation_mode are rejected at parse time. text_comparison_mode defaults to logical (Teradata text columns). Any field set here can be overridden per table via a nested validation_configuration on that table entry.

Property	Type	Description
schema_validation	Boolean	Level 1: schema / column consistency checks.
metrics_validation	Boolean	Level 2: statistical metrics comparison.
row_validation	Boolean	Level 3: row-level data comparison.
row_validation_mode	String	L3 mode: hybrid only (default). Legacy row and cell are rejected. Hybrid runs row-hash first, then cell drill-down for mismatched partitions.
continue_on_failure	Boolean	Whether to continue to the next validation level after a failure.
max_failed_rows_number	Integer	Maximum mismatch rows per partition and orchestrator early-stop threshold (must be greater than 0 when set). Default 1000.
exclude_metrics	Boolean	Whether to exclude unsupported metric columns.
apply_metric_column_modifier	Boolean	Whether to apply metric column modifiers.
early_stopping_for_row_hashing	Boolean	When true, hybrid L3 row-hashing may stop early once ingested mismatch rows reach max_failed_rows_number (see Early Stopping). Default false.
early_stopping_for_cell_by_cell_comparison	Boolean	When true, hybrid L3 cell drill-down may stop early once ingested mismatch rows reach max_failed_rows_number. Default true.
early_stop_check_interval_minutes	Integer	Minutes between orchestrator poll ticks that check the mismatch count. Default 5 when either early-stop flag is enabled and interval is omitted. Mutually exclusive with early_stop_check_interval_seconds.
early_stop_check_interval_seconds	Integer	Seconds between orchestrator poll ticks (finer-grained alternative to early_stop_check_interval_minutes; at most one of the two may be set). Note: on the Snowflake task queue the value is rounded up to whole minutes (1-minute floor); sub-minute polling only applies to the synthetic SQLite test queue.
text_comparison_mode	String	How text columns are compared in L3 (row-hash and cell). logical (default) canonicalizes to UTF-8 with TRIM/normalization; raw compares native storage bytes byte-for-byte. Teradata source only; other platforms ignore it. See Text comparison mode.
acceptedTransformations	Object[]	Accepted transformations that apply globally (within this validation_configuration block). Unioned with root-level and per-table lists. See acceptedTransformations.

Text comparison mode

text_comparison_mode controls how Teradata text columns (CHAR, VARCHAR, CLOB, GRAPHIC, LONG VARCHAR, VARGRAPHIC) are compared in Level 3, applied consistently to both the row-hash and cell-by-cell phases:

• logical (default): the historical behaviour — compares with the existing TRIM/normalization (no extra encoding), so the same logical text matches across platforms regardless of source charset.
• raw: compares the source's native storage bytes as hex (via Teradata CHAR2HEXINT; UTF-16 for UNICODE-charset columns, single-byte code page for LATIN), after trimming trailing whitespace (TRIM(TRAILING FROM …) on Teradata, RTRIM(…) on Snowflake) so fixed-width CHAR padding does not cause false mismatches. Byte-exact otherwise — catches invisible characters, leading and internal whitespace, and encoding differences. Because Teradata's native bytes differ from Snowflake's UTF-8 (HEX_ENCODE), the same logical text can still be reported as a mismatch (a UNICODE column differs even for ASCII; a LATIN column matches ASCII but not accented characters). Use when you need to verify exact byte fidelity.

Settable globally and overridable per table/view via the nested validation_configuration. Precedence: per-table → global → default logical.

comparison_configuration

Property	Type	Description
tolerance	Number	Numeric comparison tolerance for metrics (must be greater than 0 when set). Default applied by the orchestrator when omitted is 0.001.
type_mapping_file_path	String	Optional path to a custom type mapping file for comparisons.

Per-table / per-view entry (tables and views)

Property	Type	Required	Description
fully_qualified_name	String	Yes	Source object name (format depends on source platform).
use_column_selection_as_exclude_list	Boolean	No	Default false.
column_selection_list	String[]	No	Columns to include or exclude per use_column_selection_as_exclude_list.
target_name	String	No	Target object name override.
target_database	String	No	Per-table target database override.
target_schema	String	No	Per-table target schema override.
where_clause	String	No	Filter on the source side.
target_where_clause	String	No	Filter on the target side.
index_column_list	String[]	No	Columns used to align rows on the source.
target_index_column_list	String[]	No	Columns used to align rows on the target.
column_mappings	Object	No	Map of source column name to target column name.
is_case_sensitive	Boolean	No	Case sensitivity for identifiers.
max_failed_rows_number	Integer	No	Overrides global cap for this object.
exclude_metrics	Boolean	No	Per-object metrics exclusion override.
apply_metric_column_modifier	Boolean	No	Per-object modifier override.
object_type	String	No	Typically TABLE or VIEW.
column_names_to_partition_by	String[]	No	Ordered columns for lexicographic range (NTILE) partitioning during validation. Multiple columns are supported (e.g. ["year", "month"]). Without this, the table is processed as a single partition.
target_partition_size_rows	Integer	No	Per-table override for desired rows per partition. Mutually exclusive with target_partition_size_mb. Must be greater than 0.
target_partition_size_mb	Integer	No	Per-table override for desired MB per partition. Mutually exclusive with target_partition_size_rows. Must be greater than 0.
validation_configuration	Object	No	Nested object with the same fields as global validation_configuration to override defaults for this object only.
acceptedTransformations	Object[]	No	Per-table accepted transformations. Unioned with global list. See acceptedTransformations.

Partitioning (column_names_to_partition_by)

When column_names_to_partition_by is set, the orchestrator splits the object into range-based partitions using lexicographic order on the listed columns. Data Migration and Data Validation share sizing and interval rules; DM also creates edge partitions (below min / above max) for full extraction coverage.

Details:

• How partition count and NTILE boundary discovery work (including sampling on 100M+ row tables)
• What is stored in PARTITION_METADATA (scalar vs multi-column delimited bounds)
• Edge partitions (DM only) vs interior-only buckets (DV)
• Multi-column examples and limitations

Partition size targets (target_partition_size_rows / target_partition_size_mb): when both are omitted, Data Validation defaults to 200 MB per partition. Specify at most one of the two fields; both must be greater than 0 when set.

Early Stopping (L3)

When a table has many partitions, L3 validation (row or cell comparison) can be expensive because every partition must be fully processed before the orchestrator evaluates results. Early stopping allows the workflow to abort remaining L3 partition work once enough mismatches have been detected, saving compute on both the source system and Snowflake.

How it works

1. When hybrid L3 tasks are created and the relevant early-stop flag is enabled for the stage (early_stopping_for_row_hashing or early_stopping_for_cell_by_cell_comparison), the orchestrator pushes an internal poll task that periodically checks how many mismatch rows have already been ingested into the Snowflake results table for that stage.
2. On each poll tick the orchestrator counts rows in the appropriate results table (ROW_VALIDATION_RESULTS for the row-hash poll, CELL_VALIDATION_RESULTS for the cell drill-down poll).
3. If the count reaches or exceeds max_failed_rows_number, all pending L3 partition tasks for that stage are bulk-completed (skipped) so the workflow can converge without processing every partition.
4. If the count is still below the threshold, the poll reschedules itself after early_stop_check_interval_minutes minutes and checks again.
5. Polling also stops automatically when every monitored L3 task for that stage reaches a terminal state.

When to use it

• Large tables with many partitions where you expect mismatches: early stopping avoids running thousands of partition comparisons when the first few already confirm a data discrepancy.
• Hybrid mode (row_validation_mode = "hybrid", the default when L3 is enabled): configure row-hash and cell drill-down early stopping independently via early_stopping_for_row_hashing (default false) and early_stopping_for_cell_by_cell_comparison (default true).

Configuration

"validation_configuration": {
  "row_validation": true,
  "max_failed_rows_number": 500,
  "early_stopping_for_row_hashing": true,
  "early_stopping_for_cell_by_cell_comparison": true,
  "early_stop_check_interval_minutes": 2
}

Field	Description
max_failed_rows_number	Caps mismatch rows collected per partition and triggers bulk skip when ingested total reaches this value.
early_stopping_for_row_hashing	Enable early stopping during hybrid row-hashing (default false).
early_stopping_for_cell_by_cell_comparison	Enable early stopping during hybrid cell drill-down (default true).
early_stop_check_interval_minutes	How often the orchestrator checks the mismatch count. Shorter intervals react faster but add more metadata queries.

When either early-stop flag is true, early_stop_check_interval_minutes defaults to 5 if omitted. The mismatch threshold is always max_failed_rows_number.

Inheritance

Early stopping fields follow the same inheritance rules as other validation_configuration properties: per-table values override global values. The orchestrator resolves the effective values from the first source (table-level, then global-level) that provides them.

Interaction with hybrid mode

For hybrid mode (row_validation_mode = "hybrid"), there are two independent polls: the row-hash poll counts mismatches in ROW_VALIDATION_RESULTS; the cell drill-down poll counts mismatches in CELL_VALIDATION_RESULTS. Once the row-hash phase converges — either because all partitions finished or because early stopping triggered — the orchestrator evaluates which partitions had mismatches and runs targeted cell comparisons only for those.

acceptedTransformations

Defines source-to-target value pairs that are considered acceptable mismatches during row comparison. Useful when known, benign differences exist between source and target (for example, encoding changes or NULL-to-empty-string coercions).

Each entry requires exactly one of column or columnPattern, plus sourceValue and targetValue:

Field	Type	Required	Description
column	String	One of column/columnPattern	Exact column name to match.
columnPattern	String	One of column/columnPattern	Regex pattern matching one or more column names.
sourceValue	String or null	Yes	Expected source-side value (use null to represent SQL NULL).
targetValue	String or null	Yes	Expected target-side value (use null to represent SQL NULL).

acceptedTransformations can appear at three levels; the orchestrator unions all applicable lists for each table:

1. Root level of the workflow config (applies to all tables).
2. Inside the global validation_configuration block (applies to all tables, merged with root list).
3. Inside a per-table entry, either at the top level or inside its nested validation_configuration (applies to that table only, merged with global list).

Example:

{
  "acceptedTransformations": [
    { "column": "status", "sourceValue": "ACTIVE", "targetValue": "1" },
    { "columnPattern": "^flag_", "sourceValue": null, "targetValue": "false" }
  ]
}

Advanced Features

⏱️ Idle Shutdown Operational Constraints & SLO

The orchestrator may self-terminate after prolonged idle (DM_IDLE_SHUTDOWN_MINUTES, default 60). Before exit, CHECK_IDLE_SHUTDOWN_SAFETY runs once as a pre-exit safety net. See Explicit Idle Shutdown & Safety for the full engineering spec.

• Orchestrator SLO (<10 min): Orchestrator-owned tasks are metadata-bound and expected to complete in <10 minutes (including partition boundary sampling queries). Longer orchestrator execution is considered an architectural bug. Heavy-lift or long-running computations must be delegated to Data Exchange Agents (DEA).
• Multi-Tenant Scope Extension: While recent_task_activity tracks churn exclusively for EXECUTOR_TYPE = 'orchestrator', the pre-exit safety gate (active_upcoming_tasks) evaluates active workloads globally. It intercepts tasks in an executing state for ANY executor type (including DEA) to ensure the orchestrator remains alive to perform bookkeeping when long-running extractions finish.
• Anti-Zombie Guard Clause: A workflow in running without active rows in TASK_QUEUE will NOT block the idle shutdown clock. This prevents abandoned or corrupted long-running workflow metadata from keeping orchestrator infrastructure alive indefinitely. Workflows in initializing still block shutdown immediately even when the queue is empty.

Redshift UNLOAD

For Redshift, it is recommended to use the UNLOAD extraction strategy. The main idea behind this is:

• Large query results are written directly to an S3 Bucket instead of being downloaded to the machine in which the Worker is running.
• On Snowflake side, an External Stage is set up to reference the corresponding S3 Bucket, so that COPY INTO statements can be done directly from that stage.

See the Extraction Strategy section for configuration details.

Incremental Synchronization

It is possible to migrate some tables and then re-migrate them in the future, moving only the data that has changed. See the Synchronization Strategies section for the available strategies and their configuration.

Query Tagging

Both the Orchestrator and the Worker automatically set Snowflake's QUERY_TAG session parameter on every query they submit. Tags are compact JSON strings containing identifiers such as the workflow ID, task ID, and component version. You can use these tags to filter and attribute queries in QUERY_HISTORY:

SELECT query_text, query_tag, start_time
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
WHERE TRY_PARSE_JSON(query_tag):DMVF_WORKFLOW_ID IS NOT NULL
ORDER BY start_time DESC;

Tag key	Present on	Description
DMVF_VERSION	Infrastructure queries	Component package version.
DMVF_WORKFLOW_ID	Task-processing queries	Workflow that originated the task.
DMVF_TASK_ID	Task-processing queries	Individual task identifier.
DMVF_ORCHESTRATOR_VERSION	Orchestrator task-processing queries	Orchestrator package version.
DMVF_WORKER_VERSION	Worker task-processing queries	Worker package version.

Considerations and Recommendations

Connecting to Snowflake with a PAT

It is recommended to use Programmatic Access Tokens for connections used by the Orchestrator and Workers. This ensures there won't be a need to constantly authenticate through the browser or with an Authenticator app. You will need to establish a Network Policy or temporarily bypass the requirement for a Network Policy (this can be done from Snowsight).

Running Orchestrator and/or Workers on SPCS

If you want to leverage Snowflake compute for these tasks, you can:

1. Prepare Docker images that use the Python modules and have the appropriate configuration.
2. Push those Docker images to an Image Repository in Snowflake.
3. Execute the Orchestrator and/or Worker(s) images using Snowpark Container Services.

Some considerations:

• It is recommended to execute them as Services, not Jobs.
• It is possible to run only one component (Orchestrator or Workers) in SPCS and the other on another platform.
• It is a good practice to monitor the SPCS service and suspend it when it is not being used.
• Depending on the network configuration of the Source System, you might need to configure an External Access Integration so that these services can connect to your Source System.

Initial Testing

It is recommended to deploy the DDL for the tables you want to migrate before starting data migration. This ensures the target types match the behavior you want in those tables and their related views/procedures. Converting the DDL from your source dialect into Snowflake SQL can be done through the Code Conversion capabilities of SnowConvert AI and/or Cortex Code. If you don't deploy the DDL before starting data migration, the types will be inferred and might not be as accurate as desired.

Additionally, it is a good practice to move a few rows from each table as a test before starting the full migration. This helps detect configuration or connectivity issues early.

Managing Workers

The time it takes to complete a workflow depends heavily on many variables. One of the variables that affects the most is the number of workers (and the number of threads per worker), since that determines how many extraction tasks can be executed in parallel. Consider:

1. It is not necessary to run two workers on the same machine. If you want more parallelism on one machine, increase the thread count instead.
2. Network bandwidth greatly affects the speed of workers and is effectively shared between threads of a worker.
3. Even with many workers/threads processing tasks in parallel, your source system might not have enough resources to handle the load.
4. You might want to keep a low worker count to avoid overloading your source system.
5. You might want to stop some (or all) of your workers at times when your source system is already overloaded by unrelated operations, to avoid disrupting those operations.

Workflow Configuration Reference

The workflow configuration file is a JSON object. Its structure is described below using named models -- each model's properties reference other models by name.

WorkflowConfiguration (Top Level)

Property	Type	Required	Description
schemaVersion	String	No	Version of the configuration schema (e.g. "1.0.0"). Accepts formats "major", "major.minor", or "major.minor.patch". Defaults to "1.0.0" if omitted.
tables	TableConfiguration[]	Yes	An array of table-specific configurations defining which tables to migrate and how.
defaultTableConfiguration	TableConfiguration	No	Shared settings inherited by all tables. Table-specific values override these defaults (see merging rules below).

TableConfiguration

Defines the settings for migrating a single table.

Property	Type	Required	Description
source	SourceTargetIdentifier	Yes	Identifies the source table.
target	SourceTargetIdentifier	Yes	Identifies the target table in Snowflake.
columnNamesToPartitionBy	String[]	No	Ordered columns for lexicographic range partitioning during extraction (e.g. ["year", "month"]). When omitted or empty, the table is extracted as a single unit (recommended only for very small tables).
extraction	ExtractionStrategy	No	Configures how data is extracted from the source database.
synchronization	SynchronizationStrategy	No	Configures incremental synchronization behavior.
columnTypeMappings	ColumnTypeMapping[]	No	Type conversions applied during migration.
columnNameMappings	ColumnNameMapping[]	No	Column renaming mappings.
primaryKeyColumns	String[]	No	Primary key columns for the source table. Required when using trackModifications or trackDeletions in the watermark synchronization strategy. Can be omitted and inferred automatically at runtime.
targetPartitionSizeMb	Integer	No	Target partition size in MB. Mutually exclusive with targetPartitionSizeRows. Must be greater than 0 when set. When both targetPartitionSizeMb and targetPartitionSizeRows are omitted, the orchestrator picks sizes automatically (auto mode). See Partition Size.
targetPartitionSizeRows	Integer	No	Target partition size in rows. Mutually exclusive with targetPartitionSizeMb. Must be greater than 0 when set. When both targetPartitionSizeMb and targetPartitionSizeRows are omitted, the orchestrator picks sizes automatically (auto mode). See Partition Size.
whereClauseCriteria	String	No	SQL-like filter to select a subset of rows (e.g., "is_deleted = 0").
loadSegmentation	LoadSegmentation	No	Splits a single COPY INTO into multiple parallel statements, each targeting a subset of staged files.
loading	Object	No	(Experimental) Loading strategy override. Accepts { "strategy": "snowpipe" } to use Snowpipe for ingestion instead of the default warehouse-based COPY INTO. Behavior and configuration may change in future releases.

Default Table Configuration Merging Rules

When defaultTableConfiguration is provided, its values are merged into each table entry using these rules:

• Nested objects (source, target, synchronization, extraction): Deep merge -- fields within are merged individually.
• Collections (columnTypeMappings, columnNameMappings, etc.): Table value replaces default entirely.
• Scalars (whereClauseCriteria): Table value overrides default.

SourceTargetIdentifier

Used by source and target in TableConfiguration to identify a database object.

Property	Type	Required	Description
databaseName	String	Yes	Name of the database.
schemaName	String	Yes	Name of the schema.
tableName	String	Yes	Name of the table.

The target object accepts two additional optional fields: tableType ("native" or "iceberg") and icebergConfig (required when using Iceberg). If tableType is omitted or null, it defaults to "native" (standard Snowflake table). See IcebergConfig and the Redshift UNLOAD with Iceberg Tables example.

IcebergConfig (target.icebergConfig)

Used when target.tableType is "iceberg". Fields are merged with defaultTableConfiguration.target.icebergConfig; table-level keys override defaults.

Property	Type	Required	Description
catalog	String	No	Default SNOWFLAKE for Snowflake-managed Iceberg. Use a catalog integration name for externally cataloged tables (for example AWS Glue).
externalVolume	String	For catalog SNOWFLAKE	Snowflake external volume for Iceberg data and metadata.
baseLocationPrefix	String	No	Optional path prefix for BASE_LOCATION when using Snowflake-managed Iceberg (catalog SNOWFLAKE).
catalogTableName	String	For external catalog	Fully qualified name of the table in the external catalog (for example glue_db.my_table).
catalogSync	String	No	Optional catalog integration used to sync Snowflake-managed metadata back to an external catalog.
sourceDataStage	String	No	Stage path starting with @ pointing at existing Parquet files; used for copy_files-style loads with Snowflake-managed Iceberg.
migrationStrategy	String	No	One of catalog_link, convert_to_managed, copy_files. When omitted, the orchestrator infers a strategy from catalog and sourceDataStage.

Snowflake account setup for Iceberg (external volumes, catalog integrations, stages, and privileges) follows Snowflake’s Iceberg documentation; use the examples above as a template for JSON fields.

ColumnTypeMapping

Property	Type	Required	Description
sourceType	String	Yes	Type name in the source system.
targetType	String	Yes	Target type in Snowflake.

ColumnNameMapping

Property	Type	Required	Description
sourceName	String	Yes	Column name in the source system.
targetName	String	Yes	Target column name in Snowflake.

ExtractionStrategy

Field	Type	Required	Description
strategy	"regular", "unload", "write_nos", "dbms_cloud", or "cet_as"	Yes	Extraction method. "regular" is the default. "unload" is for Redshift. "write_nos" is for Teradata direct-to-cloud. "dbms_cloud" is for Oracle direct-to-object-storage. "cet_as" is for Azure Synapse CETAS to Azure Blob. Legacy "tpt" is accepted as an alias for "regular" (TPT is selected on the worker when tpt_* is set in the DEA config).
externalStage	String	UNLOAD / WRITE_NOS / DBMS_CLOUD / CETAS	Fully-qualified Snowflake external stage name (e.g., "MY_DB.MY_SCHEMA.S3_STAGE").

regular (default) -- Data is queried and downloaded through the Worker:

"extraction": { "strategy": "regular" }

unload (Redshift only) -- Data is written to S3 via Redshift UNLOAD and loaded from an external stage:

"extraction": { "strategy": "unload", "externalStage": "MY_DB.MY_SCHEMA.S3_EXTERNAL_STAGE" }

write_nos (Teradata only) -- Data is written directly to cloud object storage (S3, Azure Blob, or GCS) via the Teradata WRITE_NOS table function, then loaded from an external stage that points at the same location. Requires the worker to have write_nos_* settings under [connections.source.teradata]:

"extraction": { "strategy": "write_nos", "externalStage": "MY_DB.MY_SCHEMA.TD_NOS_STAGE" }

cet_as (Azure Synapse only) -- Data is written directly to Azure Blob via Synapse CREATE EXTERNAL TABLE AS SELECT (CETAS), then loaded from an external stage that points at the same container prefix. Requires pre-created Synapse EXTERNAL DATA SOURCE / EXTERNAL FILE FORMAT objects and cet_as_* settings under [connections.source.azure_synapse] on the worker. Create the workflow with --source-platform azure_synapse or azure_synapse_serverless:

"extraction": { "strategy": "cet_as", "externalStage": "@TARGET_DB.SYNAPSE_MIG.SYNAPSE_BLOB_STAGE" }

For Teradata bulk jobs with "strategy": "regular", the worker uses Teradata Parallel Transporter (tbuild) when tpt_output_directory (and related tpt_* keys) are set under [connections.source.teradata] in the DEA config—same idea as SQL Server BCP, which needs no orchestrator extraction setting. Otherwise the worker uses ODBC/teradatasql. The path convention to the Snowflake internal stage matches plain "regular" extraction.

Partition Size

Controls how large each partition should be during extraction. Configured at the TableConfiguration level via two flat, mutually exclusive fields: targetPartitionSizeMb or targetPartitionSizeRows. When both are omitted, the system uses auto sizing.

Form	Description
Both omitted (default)	Auto. The system picks optimal partition sizes based on the source platform, extraction strategy, and table size.
"targetPartitionSizeMb": N	Each partition targets approximately N megabytes of data. Must be greater than 0.
"targetPartitionSizeRows": N	Each partition targets N rows, regardless of data size. Must be greater than 0.

Only one of targetPartitionSizeMb or targetPartitionSizeRows may be specified for a given table (setting both is a configuration error).

Auto (default) -- Omit both fields. The system selects partition sizes tuned for the platform and extraction strategy. Auto mode uses larger partitions for Redshift UNLOAD (where S3 handles large files well) and smaller partitions for ODBC-based extraction (SQL Server, Redshift REGULAR) where data flows through the Worker's memory.

Fixed size in MB -- Specify a target size per partition:

"targetPartitionSizeMb": 2048

Fixed row count -- Specify a target number of rows per partition:

"targetPartitionSizeRows": 500000

LoadSegmentation

Controls post-upload load segmentation. When a large number of files are staged for a single partition (common with Redshift UNLOAD), the orchestrator can split the COPY INTO into multiple parallel statements, each targeting a subset of files. All resulting COPY INTO tasks fan-in to the same successor task.

Property	Type	Required	Description
targetSegmentSizeMb	Integer	Yes	Target total file size (in MB) per COPY INTO segment.

When loadSegmentation is omitted, a single COPY INTO is used for all files in the partition (the default behavior).

"loadSegmentation": { "targetSegmentSizeMb": 5000 }

Each segment will contain files whose total size does not exceed the target. Files larger than the target are placed in their own segment. The Snowflake FILES parameter limit (1,000 files) is also enforced per statement.

SynchronizationStrategy

Controls whether subsequent workflow runs perform a full re-extraction or only sync changed data.

Field	Type	Required	Description
strategy	"none", "checksum", or "watermark"	Yes	The synchronization method.
checksumExpression	String	No	Custom SQL aggregate expression for the checksum strategy. See below.
watermarkColumn	String	Watermark only	Column name to track (must be monotonically increasing).
trackModifications	Boolean	No	If true, uses the primary key to deduplicate modified rows after COPY INTO. Requires primaryKeyColumns in TableConfiguration.
trackDeletions	Boolean	No	If true, extracts a full PK snapshot of the source partition on every incremental run and deletes target rows absent from the snapshot before loading new rows. Requires primaryKeyColumns. Independent of trackModifications; both can be combined.

none (default) -- Full extraction on every run. No synchronization metadata is stored.

"synchronization": { "strategy": "none" }

• Use when: Data is small, changes are unpredictable, or guaranteed consistency is needed.

checksum -- Computes a hash of all column values per partition. Only changed partitions are cleared and re-extracted.

"synchronization": { "strategy": "checksum" }

• Use when: You need to detect any change but lack a reliable monotonic column (e.g., dimension tables).
• Trade-offs: Requires a checksum computation on the source for every partition on every run.

checksum with a custom expression -- Instead of the default per-column MD5 hashing, you can supply any SQL aggregate expression that returns a single value representing the state of the partition. Only partitions where the value has changed since the last run are cleared and re-extracted.

"synchronization": {
  "strategy": "checksum",
  "checksumExpression": "MAX(ORA_ROWSCN)"
}

The expression is evaluated as SELECT <checksumExpression> AS checksum FROM <table> WHERE <partition_condition>. It must be a valid aggregate SQL expression for the source platform that returns a scalar value.

• Use when: The source platform exposes a built-in change-detection mechanism (e.g., Oracle ORA_ROWSCN, a row-version column, or a MAX(last_modified) timestamp) that is cheaper to compute than a full per-column hash.
• Trust model: The expression is inlined verbatim into the source query. Semicolons (;) are rejected as a basic guard, but the workflow configuration is a trusted input — treat it accordingly.
• Empty partitions: When the expression evaluates to NULL (e.g., MAX over zero rows), the orchestrator treats the partition as unchanged on subsequent runs. On the first run it will always extract (no stored value to compare against), which is the correct behavior.

watermark -- Tracks a monotonic column (timestamp, ID, version) to sync only rows newer than the last observed maximum.

Basic (inserts only):

"synchronization": { "strategy": "watermark", "watermarkColumn": "UPDATED_AT" }

Only rows whose UPDATED_AT is strictly greater than the maximum value seen in the previous run are extracted. Rows that already exist in the target are not touched.

• Use when: Your table has a reliable monotonic column that increases on every insert (e.g., event logs, append-only fact tables).

With trackModifications (inserts + updates):

When a row is updated at the source, its watermark advances above the stored maximum, so it is re-extracted and appended to the target. The old version remains until trackModifications removes it: after COPY INTO, the orchestrator deletes duplicate rows, keeping only the latest version per primary key.

{
  "primaryKeyColumns": ["id"],
  "synchronization": {
    "strategy": "watermark",
    "watermarkColumn": "updated_at",
    "trackModifications": true
  }
}

Requires primaryKeyColumns (configured or inferred automatically).

With trackDeletions (inserts + deletions):

Deletions are invisible to watermark filtering because deleted rows have no new watermark value. When trackDeletions is enabled, the orchestrator extracts the full set of primary keys currently in the source partition on every incremental run. Before loading new rows, it deletes target rows whose primary key is absent from that snapshot — i.e. rows that were deleted at the source since the last sync.

{
  "primaryKeyColumns": ["id"],
  "synchronization": {
    "strategy": "watermark",
    "watermarkColumn": "updated_at",
    "trackDeletions": true
  }
}

• The primary-key snapshot is extracted in parallel with the changed-rows extraction, so it adds minimal latency.
• Deletion tracking begins on the first incremental run (after the initial load); the initial load itself is unaffected.
• Requires primaryKeyColumns (configured or inferred automatically).

With both trackModifications and trackDeletions (inserts + updates + deletions):

{
  "primaryKeyColumns": ["id"],
  "synchronization": {
    "strategy": "watermark",
    "watermarkColumn": "updated_at",
    "trackModifications": true,
    "trackDeletions": true
  }
}

The two flags are independent and can be freely combined. The orchestrator applies deletions first (removes absent-PK rows), then upserts the new/changed rows.

• Use when: Your table has a reliable monotonic column and you need full CDC-like sync (inserts, updates, and deletes) without a dedicated change-tracking table.
• Trade-offs: trackDeletions scans the full PK set of each source partition on every run. For very large partitions this adds source load; consider tighter partition sizing or the checksum strategy if the source has no reliable watermark column.

Quoting Identifiers

Names that need quoting (or brackets) must be manually quoted as they would normally be in JSON. For example: "tableName": "\"MyCaseSensitiveTable\"".

Workflow Configuration Examples

Basic Migration (SQL Server)

Migrates two tables with shared source/target schemas, type mappings, column renaming, watermark sync, and row filtering:

{
  "defaultTableConfiguration": {
    "source": {
      "schemaName": "data_migration_cloud_test",
      "databaseName": "SampleStoreDB"
    },
    "target": {
      "schemaName": "data_migration_cloud_test",
      "databaseName": "samplestoredb"
    }
  },
  "tables": [
    {
      "source": { "tableName": "store_employee" },
      "target": { "tableName": "target_employee" },
      "columnNamesToPartitionBy": ["ID"]
    },
    {
      "source": { "tableName": "Sales_Simple" },
      "target": { "tableName": "Sales_Simple" },
      "columnNamesToPartitionBy": ["ID"],
      "columnTypeMappings": [
        { "sourceType": "MONEY", "targetType": "DECIMAL(19,4)" }
      ],
      "columnNameMappings": [
        { "sourceName": "id", "targetName": "old_id" },
        { "sourceName": "name", "targetName": "full_name" }
      ],
      "synchronization": {
        "strategy": "watermark",
        "watermarkColumn": "UPDATED_AT"
      },
      "targetPartitionSizeMb": 2048,
      "whereClauseCriteria": "is_deleted = 0"
    }
  ]
}

Redshift UNLOAD

Uses the UNLOAD extraction strategy with an external stage for S3-based data transfer:

{
  "defaultTableConfiguration": {
    "source": {
      "schemaName": "ecommerce_raw",
      "databaseName": "snowconvert_demo"
    },
    "target": {
      "schemaName": "ecommerce_raw",
      "databaseName": "TARGET_DB"
    },
    "extraction": {
      "strategy": "unload",
      "externalStage": "MY_DB.MY_SCHEMA.S3_EXTERNAL_STAGE"
    },
    "loadSegmentation": { "targetSegmentSizeMb": 5000 }
  },
  "tables": [
    {
      "source": { "tableName": "customers" },
      "target": { "tableName": "customers" },
      "columnNamesToPartitionBy": ["customer_id"]
    },
    {
      "source": { "tableName": "orders" },
      "target": { "tableName": "orders" },
      "columnNamesToPartitionBy": ["order_id"],
      "columnTypeMappings": [
        { "sourceType": "NUMERIC(10,2)", "targetType": "DECIMAL(10,2)" }
      ]
    }
  ]
}

Redshift UNLOAD with Iceberg Tables

Combines Redshift UNLOAD with Iceberg table targets, including Snowflake-managed and Glue catalog configurations:

{
  "defaultTableConfiguration": {
    "source": {
      "schemaName": "public",
      "databaseName": "analytics_db"
    },
    "target": {
      "schemaName": "public",
      "databaseName": "TARGET_DB",
      "tableType": "iceberg",
      "icebergConfig": {
        "catalog": "SNOWFLAKE",
        "externalVolume": "my_iceberg_ext_vol",
        "baseLocationPrefix": "migrations/redshift",
        "sourceDataStage": "@TARGET_DB.PUBLIC.ICEBERG_SOURCE_STAGE"
      }
    },
    "extraction": {
      "strategy": "unload",
      "externalStage": "TARGET_DB.PUBLIC.S3_EXTERNAL_STAGE"
    }
  },
  "tables": [
    {
      "source": { "tableName": "customers" },
      "target": { "tableName": "customers" },
      "columnNamesToPartitionBy": ["customer_id"]
    },
    {
      "source": { "tableName": "events" },
      "target": {
        "tableName": "events",
        "tableType": "iceberg",
        "icebergConfig": {
          "catalog": "my_glue_catalog_integration",
          "externalVolume": "my_iceberg_ext_vol",
          "catalogTableName": "glue_db.events"
        }
      },
      "columnNamesToPartitionBy": ["event_id"]
    },
    {
      "source": { "tableName": "orders" },
      "target": {
        "tableName": "orders",
        "tableType": "iceberg",
        "icebergConfig": {
          "catalog": "my_glue_catalog_integration",
          "externalVolume": "my_iceberg_ext_vol",
          "catalogTableName": "glue_db.orders",
          "migrationStrategy": "convert_to_managed"
        }
      },
      "columnNamesToPartitionBy": ["order_id"]
    }
  ]
}

Incremental Sync with Watermark

Uses watermark-based synchronization with modification tracking for incremental data migration:

{
  "defaultTableConfiguration": {
    "source": { "databaseName": "SRC", "schemaName": "dbo" },
    "target": { "databaseName": "TGT", "schemaName": "public" },
    "synchronization": {
      "strategy": "watermark",
      "watermarkColumn": "updated_at"
    }
  },
  "tables": [
    {
      "source": { "tableName": "orders" },
      "target": { "tableName": "orders" },
      "columnNamesToPartitionBy": ["order_id"],
      "primaryKeyColumns": ["order_id"],
      "synchronization": {
        "trackModifications": true
      }
    }
  ]
}

Incremental Sync with Watermark and Deletion Tracking

Syncs only changed rows (watermark) and removes rows deleted at the source since the last run (trackDeletions). Both trackModifications and trackDeletions can be combined:

{
  "defaultTableConfiguration": {
    "source": { "databaseName": "SRC", "schemaName": "dbo" },
    "target": { "databaseName": "TGT", "schemaName": "public" }
  },
  "tables": [
    {
      "source": { "tableName": "orders" },
      "target": { "tableName": "orders" },
      "columnNamesToPartitionBy": ["order_id"],
      "primaryKeyColumns": ["order_id"],
      "synchronization": {
        "strategy": "watermark",
        "watermarkColumn": "updated_at",
        "trackModifications": true,
        "trackDeletions": true
      }
    }
  ]
}

Early stopping (Data Validation)

Enables L3 early stopping globally, with a per-table override using a lower threshold for a known-problematic table. The orchestrator will stop processing remaining partitions for a table once the configured number of mismatches is detected:

{
  "source_platform": "Teradata",
  "target_platform": "Snowflake",
  "validation_configuration": {
    "schema_validation": true,
    "metrics_validation": true,
    "row_validation": true,
    "continue_on_failure": true,
    "max_failed_rows_number": 1000,
    "early_stopping_for_row_hashing": true,
    "early_stopping_for_cell_by_cell_comparison": true,
    "early_stop_check_interval_minutes": 5
  },
  "tables": [
    {
      "fully_qualified_name": "my_database.large_fact_table",
      "target_database": "MY_DATABASE",
      "target_schema": "PUBLIC",
      "target_name": "LARGE_FACT_TABLE",
      "column_names_to_partition_by": ["ID"],
      "target_partition_size_mb": 200
    },
    {
      "fully_qualified_name": "my_database.known_problematic_table",
      "target_database": "MY_DATABASE",
      "target_schema": "PUBLIC",
      "target_name": "KNOWN_PROBLEMATIC_TABLE",
      "column_names_to_partition_by": ["ORDER_ID"],
      "validation_configuration": {
        "max_failed_rows_number": 100,
        "early_stop_check_interval_minutes": 1
      }
    }
  ]
}

Hybrid validation with early stopping (Data Validation)

Hybrid mode requires early stopping. The row-hash phase detects mismatched partitions quickly, and once enough are found the remaining partitions are skipped. Only mismatched partitions proceed to cell-level drill-down:

{
  "source_platform": "Teradata",
  "target_platform": "Snowflake",
  "validation_configuration": {
    "schema_validation": true,
    "metrics_validation": true,
    "row_validation": true,
    "row_validation_mode": "hybrid",
    "continue_on_failure": true,
    "max_failed_rows_number": 500,
    "early_stopping_for_row_hashing": true,
    "early_stopping_for_cell_by_cell_comparison": true,
    "early_stop_check_interval_minutes": 2
  },
  "tables": [
    {
      "fully_qualified_name": "my_database.sales_transactions",
      "target_database": "MY_DATABASE",
      "target_schema": "PUBLIC",
      "target_name": "SALES_TRANSACTIONS",
      "column_names_to_partition_by": ["TRANSACTION_ID"],
      "target_partition_size_mb": 200
    }
  ]
}

View validation (Teradata and Redshift)

Validates a source view against its Snowflake counterpart using the top-level views array (or object_type: "VIEW" on a table entry). Teradata uses basic L1 schema (existence + datatype) via HELP COLUMN. Redshift uses full L1 schema via SVV_COLUMNS, the same path as tables. PostgreSQL uses full L1 schema via information_schema / pg_catalog, the same path as tables. Cloud DV never CTAS-materializes views on Teradata, Redshift, or PostgreSQL.

For the local snowflake-data-validation CLI, Teradata, Redshift, and PostgreSQL also skip view materialization so runs match cloud behavior; SQL Server and other sources still materialize views to temporary tables before validation.

The example below is Teradata-specific (source_platform, base table + view names). For Redshift, set source_platform to "Redshift", use Redshift-style three-part names, and configure the worker’s [connections.source.redshift] block as in Worker Configuration. For PostgreSQL, set source_platform to "postgresql" (or "postgres"), use PostgreSQL-style identifiers, and configure [connections.source.postgresql].

Teradata example

Validates a Teradata view against its Snowflake counterpart. The views array tags entries as object_type = "VIEW" automatically. Use a workflow with only views (no tables) to validate views by themselves; per-table overrides cannot disable all three validation levels at once (the parser rejects configurations with no work to do):

{
  "source_platform": "Teradata",
  "target_platform": "Snowflake",
  "validation_configuration": {
    "schema_validation": true,
    "metrics_validation": true,
    "row_validation": true,
    "continue_on_failure": true,
    "max_failed_rows_number": 100
  },
  "comparison_configuration": {
    "tolerance": 0.001
  },
  "views": [
    {
      "fully_qualified_name": "my_database.sales_summary_view",
      "target_database": "MY_DATABASE",
      "target_schema": "PUBLIC",
      "target_name": "SALES_SUMMARY_VIEW",
      "index_column_list": ["ID"],
      "target_index_column_list": ["ID"],
      "column_names_to_partition_by": ["ID"],
      "target_partition_size_rows": 50000
    }
  ]
}

Worker Configuration

This section documents the configuration for the Worker (snowflake-data-exchange-agent package). The Worker configuration file uses TOML format.

Section	Property	Type	Description
Top Level	selected_task_source	String	Currently should always be set to "snowflake_stored_procedure".
[application]	max_parallel_tasks	Integer	Maximum number of tasks the worker will process in parallel (using threads).
[application]	task_fetch_interval	Integer	Interval (in seconds) between attempts to fetch new tasks from the Orchestrator.
[application]	snowflake_database_for_metadata	String	Optional. Database where the orchestrator deployed the task queue (default SNOWCONVERT_AI). Must match the orchestrator's CUSTOM_SNOWFLAKE_DATABASE_FOR_METADATA if you override it there.
[application]	snowflake_schema_for_data_migration_metadata	String	Optional. Schema for PULL_TASKS / COMPLETE_TASK / FAIL_TASK (default DATA_MIGRATION). Must match the orchestrator's CUSTOM_SNOWFLAKE_SCHEMA_FOR_DATA_MIGRATION_METADATA if overridden.
[application]	local_results_directory	String	Optional. Base directory where exported Parquet/CSV files are written before upload. Defaults to ~/.data_exchange_agent/result_data.
[connections.source.*]		Object	Configuration for source system connections. The Worker typically requires an ODBC driver. See examples below.
[connections.target.snowflake_connection_name]	connection_name	String	The name of the connection entry in the ~/.snowflake/config.toml file to use.

When selected_task_source is snowflake_stored_procedure, the worker calls the task-queue procedures using application.snowflake_database_for_metadata and application.snowflake_schema_for_data_migration_metadata. These values are independent of Snowflake session defaults (SNOWFLAKE_DATABASE, SNOWFLAKE_SCHEMA) in the connection profile.

Example: SQL Server (Standard Authentication)

[connections.source.sqlserver]
username = "username"
password = "password"
database = "database_name"
host = "127.0.0.1"
port = 1433

Example: Azure Synapse Analytics (Dedicated SQL pool, SQL authentication)

Use extraction.strategy = "regular" (or omit extraction) so the worker extracts via ODBC and uploads to the Snowflake internal stage. Requires ODBC Driver 18 for SQL Server on the worker host. Create the workflow with --source-platform azure_synapse.

[connections.source.azure_synapse]
host = "myworkspace.sql.azuresynapse.net"
database = "MyDedicatedPool"
port = 1433
username = "sql_user"
password = "sql_password"
mode = "sql_auth"
pool_type = "dedicated"

Example: Azure Synapse Analytics (serverless SQL pool, SQL authentication)

Use the same ODBC worker configuration; set database to master (or your preferred session database). Table endpoints in the workflow JSON must use the database that owns the table (e.g. source.databaseName = MyDb). Create the workflow with --source-platform azure_synapse_serverless.

[connections.source.azure_synapse]
host = "myworkspace.sql.azuresynapse.net"
database = "master"
port = 1433
username = "sql_user"
password = "sql_password"
mode = "sql_auth"
pool_type = "serverless"

Example: Azure Synapse CETAS (Dedicated or serverless SQL pool)

Use extraction.strategy = "cet_as" and set externalStage to a Snowflake external stage that points at the same Azure Blob container prefix where Synapse CETAS will write Parquet files. On Synapse, create DATABASE SCOPED CREDENTIAL, EXTERNAL DATA SOURCE, and EXTERNAL FILE FORMAT ahead of time. On the worker, set cet_as_external_data_source and cet_as_file_format to match those object names:

"extraction": {
  "strategy": "cet_as",
  "externalStage": "@TARGET_DB.SYNAPSE_MIG.SYNAPSE_BLOB_STAGE"
}

[connections.source.azure_synapse]
host = "myworkspace.sql.azuresynapse.net"
database = "MyDedicatedPool"
port = 1433
username = "sql_user"
password = "sql_password"
mode = "sql_auth"
pool_type = "dedicated"

cet_as_external_data_source = "MyBlobDataSource"
cet_as_file_format = "MyParquetFormat"
# cet_as_path_prefix = "migration/prod"
# cet_as_table_schema = "dbo"

Example: Amazon Redshift (IAM Authentication)

[connections.source.redshift]
username = "demo-user"
database = "demo_db"
auth_method = "iam-provisioned-cluster"
cluster_id = "my-aws-cluster"
region = "us-west-2"
access_key_id = "your-access-key-id"
secret_access_key = "your-secret-access-key"

Example: Amazon Redshift (Standard Authentication)

[connections.source.redshift]
username = "myuser"
password = "mypassword"
database = "mydatabase"
host = "my-cluster.abcdef123456.us-west-2.redshift.amazonaws.com"
port = 5439
auth_method = "standard"

Example: Teradata

The Worker supports two Teradata drivers and automatically selects the best one available. The pure Python teradatasql driver is preferred; install it with pip install snowflake-data-exchange-agent[teradata]. If it is not installed, the Worker falls back to pyodbc with the Teradata ODBC driver.

[connections.source.teradata]
host = "your-teradata-host.example.com"
port = 1025
database = "tpcds"
username = "your_username"
password = "your_password"
# odbc_driver = "Teradata Database ODBC Driver 17.20"  # only needed when teradatasql is not installed (ODBC fallback)
# dbc_name = "TDPID_ALIAS"  # optional; defaults to host
# authentication = "LDAP"  # optional ODBC AuthMech when using pyodbc (e.g. TD2, LDAP, KRB5)

Note: Only one source connection is needed. The Snowflake target connection should point to a valid entry in your ~/.snowflake/config.toml.

Changelog

v1.26.0

New features

• Added key inference for Azure Synapse, BigQuery, and SQLite.
• Added support for the INTERVAL data type.
• Added support for custom COMMON and TEMP schema names via environment variable.
• Added support for overriding the orchestrator logs directory via environment variable.

Improvements

• Retired standalone preflight script; wired preflight as a task-graph configuration flag.
• Improved support for data validation of the INTERVAL data type.

Bug fixes

• Fixed BigQuery BIGNUMERIC casting to STRING for Parquet INFER_SCHEMA compatibility.

v1.25.1

New features

• Added data re-validation mechanism: targeted retry of failed tables and partitions from a finished parent workflow.
• Added key inference support for PostgreSQL and Oracle sources.

Bug fixes

• Fixed missing snowflake-code-unit-registry runtime dependency causing ModuleNotFoundError on fresh installs.

v1.25.0

New features

• Added automatic primary key inference for SQL Server, Teradata, and Redshift.
• Added BigQuery support for cloud Data Validation (L1/L2/L3).
• Added Redshift data type coverage for data migration and validation.
• Added support for system columns and pseudo-columns as watermark columns for incremental synchronization.
• Added row-validation KPI tiles with mismatch category breakdown.

Improvements

• Improved numeric canonical rendering with scale-aware precision for Oracle, Teradata, Redshift, and PostgreSQL.

Bug fixes

• Fixed row-validation KPI lookup failing with KeyError on missing categories.
• Fixed NULL-affinity consumer not picking up affinity-tagged tasks in the task queue.

v1.24.2

New features

• Added wall-clock execution timeout for boundary analysis tasks.
• Added support for custom checksum expressions in incremental synchronization.
• Added support for additional Data Migration target platforms.

Improvements

• Improved float canonical rendering to full precision for L2/L3 validation.
• Improved numeric canonical rendering with scale-aware handling for T-SQL and Snowflake.
• Improved Orchestrator table task queue reliability.
• Added query modifier support for SQL Server and Synapse.

Bug fixes

• Fixed Teradata L3 row-hash computation.
• Fixed appuser writable HOME directory so the logger can start on SPCS.

v1.24.1

New features

• Added wall-clock execution timeout for boundary analysis tasks.
• Added scale-aware numeric canonical rendering for T-SQL and Snowflake.
• Added query modifier support for SQL Server and Synapse.
• Added custom checksum expression support for incremental synchronization.

Improvements

• Improved float canonical rendering to full precision for L2/L3 validation.
• Improved orchestrator table task queue scheduling.

Bug fixes

• Fixed Teradata L3 row-hash computation.
• Fixed container startup failure when appuser HOME directory was not writable.

v1.24.0

New features

• Added validate-workflow-config orchestrator CLI command.

v1.23.1

New features

• Added unified objects configuration section with runtime type detection for cloud data validation.
• Added query modifiers foundation with resolver, DEA config, and scan registry.

Improvements

• Introduced dependency container for service lifecycle management in the orchestrator.

Bug fixes

• Fixed BigQuery cloud-extraction type mapping and timestamp correctness.
• Fixed Oracle NUMBER, BLOB, LONG, LONG RAW, and XMLTYPE types that were landing zero rows during migration.
• Fixed publishing issue where target directories were not included correctly in the package.

v1.23.0

New features

• Added identifier case-sensitivity policy with support for quoted, case-sensitive Snowflake targets.
• Added configurable text comparison mode (logical vs raw) for column-level validation.
• Added SQLite as a supported source platform for data validation.

Improvements

• Implemented Oracle ISO 8601 timestamp normalization for Snowflake migration.
• Enhanced timestamp normalization for Teradata with fractional-second precision.
• Implemented timestamp timezone preservation for Azure Synapse migration.
• Decoupled target platform for data validation, enabling non-Snowflake target comparisons.
• Migrated schema migrations to the Cloud State library with backwards compatibility.

Bug fixes

• Fixed Load partition tasks firing before Setup Snowpipe completion under high concurrency.
• Fixed table preprocessing failing on collapsed partition slices caused by skewed partition keys.
• Fixed SQL Server data migration with BCP and Snowpipe by setting up both CSV and Parquet pipes.
• Fixed row-hash early-stop so all failed rows are reported when early stopping is disabled.
• Fixed empty-source handling to record an explicit validation failure instead of silently passing.

v1.22.1

New features

• Added BigQuery as a source platform for migration orchestration.
• Added support for specifying projectName and datasetName on BigQuery sources.

v1.22.0

Improvements

• Preserved ISO 8601 timestamp timezone for SQL Server migration.
• Preserved ISO 8601 timestamp timezone for Teradata migration.
• Used ColumnUDTName for Teradata spatial types when the column type is a UDT.
• Improved the L2/L3 partition mechanism.

Bug fixes

• Fixed Oracle type mapping and COPY INTO timestamp loading.

v1.21.0

New features

• Added pre-flight validation that refuses workflow creation when target tables are missing in Snowflake; scai data doctor and scai data migrate start now report the missing tables and exit non-zero.

Improvements

• Made hybrid validation mode the default: row-level comparison runs first; cell-level drilldown applied to any mismatches, replacing explicit row or cell mode selection.
• Removed deprecated CHUNK_HASHES table and associated definitions; a schema migration is applied automatically on upgrade.

Bug fixes

• Fixed the partition calculator to use a row-based fallback for targets with NULL or zero BYTES (e.g. Snowflake views), preventing OOM errors on large views.
• Fixed partition row size estimation for views to use sampled rows instead of catalog size.
• Fixed data validation to fail when the Snowflake target table is empty.
• Fixed TABLE_PROGRESS.ROWS_PASSED to aggregate row validation results across all partition summaries using BOOLAND_AGG, preventing false L3-valid reports when only the latest partition passed.

v1.20.5

New features

• Added EXPORT_DATA migration strategy with GCS_EXPORT target wiring.

Improvements

• Improved Azure Synapse data validation support.
• Made the COPY extraction strategy the default for PostgreSQL.
• Hardened non-hybrid metadata handling.

Bug fixes

• Fixed the doctor command to run schema validation when checking configuration structure.
• Prevented orchestrator idle shutdown while migrations are still in progress.

v1.20.4

New features

• Added preflight subcommand skeleton to the orchestrator CLI.
• Added CETAS-based data migration support for Azure Synapse.
• Added non-hybrid metadata storage mode.
• Added edge partitions to the Data Validation GenerateValidationQueriesHandler.

Improvements

• Improved doctor subcommand with friendlier wording, native-driver detection, and target-objects existence checks.
• Expanded PostgreSQL column metrics templates.
• Switched Snowpipe REFRESH to use PATTERN-scoped pipes instead of PREFIX.

Bug fixes

• Fixed target column identifier case folding for PostgreSQL.

v1.20.3

New features

• Added external secret-managers framework.

Improvements

• Refactored Streamlit dashboard to show Data Validation run progress and metadata.

Bug fixes

• Fixed orchestrator doctor subcommand to drop unsupported connection_name kwarg.
• Corrected data exchange worker import paths, ConfigManager API, and label clarity.
• Fixed datatype reconciliation by coercing values to schema datatypes.
• Fixed DV mapper to coerce custom_overrides dict back to dataclass after JSON round-trip.
• Fixed casing issues in Data Validation runs.
• Fixed hybrid handler to forward custom_overrides to L3 anchor evaluate task.

v1.20.2

New features

• Added ODBC data migration support for Azure Synapse serverless SQL pool.

Improvements

• Improved L3 row validation with a safeguard for mismatched target columns.

Bug fixes

• Fixed identifier quoting: uppercase identifiers remain unquoted while lowercase or mixed-case identifiers are auto-quoted.

v1.20.1

New features

• Added ODBC-based data migration support for Azure Synapse Analytics (Dedicated SQL pool), including type-aware SELECT, checksum logic, smart partition sizing, and stage utilities aligned with the SQL Server T-SQL path.

Bug fixes

• Fixed intermediate partition threshold calculation to use the next partition's initial index as the exclusive boundary, preventing unmoved rows on partition edges.

v1.20.0

New features

• Added doctor subcommand for the data exchange agent to support scai data doctor.
• Added doctor subcommand for the data migration orchestrator to support scai data doctor.
• Added support for Oracle quoted and mixed-case identifiers in catalog and object-type queries.
• Added unified Teradata Docker support with teradatasql, tbuild fallback, and a configurable image registry database.

Improvements

• Improved L3 row validation by skipping per-partition information_schema queries.

Bug fixes

• Fixed PostgreSQL special datatype extraction and loading in Snowflake.
• Fixed GET_PENDING_WORKFLOWS so an orchestrator without affinity picks up any workflow.
• Fixed PUSH_TASK overload to accept a PAYLOAD TEXT argument.

v0.11.2

Improvements

• Improved task retry safety by requiring tasks to explicitly declare idempotency, preventing duplicate child-task chains when fan-out handlers retry.

Bug fixes

• Capped abandoned task retries via MAX_RETRIES_IF_ABANDONED so leases that expire stop retrying after the configured limit.
• Fixed L3 row validation payload to forward column-selection and column-mapping config so tables with mismatched column counts no longer crash.

v0.11.1

New features

• Added DBMS_CLOUD extraction strategy and task wiring for Oracle migrations.
• Added external-stage JSON load and stage file detection for DBMS_CLOUD.

Improvements

• Improved Snowpipe robustness on Snowflake accounts without database-scoped event tables and on roles that cannot set LOG_EVENT_LEVEL.

Bug fixes

• Fixed Oracle metadata queries on adb-free databases by avoiding ORA-00937, using USER_SEGMENTS for table size, and falling back to COUNT(*) when NUM_ROWS is missing.
• Failed warehouse and Snowpipe tasks immediately on first failure (previous retry attempts had no effect).

v0.11.0

New features

• Added Oracle ODBC extraction with staged Parquet metadata and improved agent ODBC robustness.
• Added Azure Synapse connection support (Phase 1).
• Enabled PostgreSQL data migration with updated metadata queries.
• Added PostgreSQL bulk extract via psql \copy with PG_CSV_FILE_FORMAT support.
• Added support for composite partition keys.

Improvements

• Enhanced task telemetry by integrating MigrationTracker and current-step tracking.
• Refactored Teradata extraction strategies to unify REGULAR and TPT handling.
• Made write_nos_location_container optional in WRITE_NOS configuration and removed the external stage URL and storage integration requirement.
• Improved Oracle fully-qualified-name and view validation handling.
• Updated VARCHAR limits in validation templates and constants.
• Improved row-hashing performance.
• Improved PULL_SINGLE_TASK performance.

Bug fixes

• Fixed partition metadata batch insertion to include row_count.
• Fixed workers without AFFINITY claiming affined tasks and workflows.
• Fixed partition metadata generation for tables with very large partition counts.

v0.10.0

New features

• Added view validation support to the Cloud Data Validation pipeline.
• Added DATA_MIGRATION_WORKFLOW and DATA_VALIDATION_WORKFLOW views with workflow-type filtering and per-workflow L1/L2/L3 progress rollups.
• Added Oracle ODBC source support.
• Added Teradata type mappings and Snowflake target fully-qualified-name helpers.
• Added a Teradata orchestrator platform module.
• Added Teradata object_type query in the shared dispatcher.
• Added Teradata to the data-migration-orchestrator workflow.
• Added Teradata ODBC support to the data-exchange-agent.
• Added Oracle source platform with ALL_ schema discovery, type mappings, and preprocessing hooks.
• Added early-stopping support for L3 row-hashing validation.
• Added support for custom metrics and templates in Cloud Data Validation.
• Added Oracle Data Validation foundation with L1 schema validation.
• Added Oracle catalog type mappings and strict require_type_mapping enforcement.
• Added L3 row and cell MD5 validation for Oracle.
• Added the teradata optional install extra (pip install snowflake-data-exchange-agent[teradata]).
• Added support for early stopping, hybrid L3, and Snowpipe (breaking change).
• Added per-table progress percentage, elapsed time, and ETA to the Data Validation dashboard.
• Added Snowflake schema utilities and type-mapping updates for the orchestrator.
• Added TPT and WRITE_NOS data sources for Teradata extraction.
• Added extraction-stage validation and workflow configuration.
• Added TPT and WRITE_NOS integration in Teradata workflow tasks.
• Added a metrics skill and PostgreSQL metrics templates.
• Added PostgreSQL connector with L0 and L1 validation.
• Added Redshift view validation.
• Added Oracle as a supported Data Validation source across the orchestrator and agent.
• Added L2 and L3 row and cell validation for PostgreSQL.
• Cloud Data Validation: PostgreSQL is now a supported source platform (postgresql, postgres, pg). The orchestrator and DEA resolve SDV Platform.POSTGRESQL and templates under postgresql/; hybrid and evaluate handlers fold schema column names to lowercase like Redshift. Local SDV skips CTAS view materialization for PostgreSQL (with Teradata, Redshift, and Oracle) so CLI view runs align with cloud DV. View entries still use the top-level views array or object_type: "VIEW" (see View validation); Teradata views remain basic L1 via HELP COLUMN.

Improvements

• Optimized L3 row-hashing queries.
• Extended Teradata ODBC connection configuration.

Bug fixes

• Deduplicated Redshift catalog rows in the planner for DISTSTYLE=ALL and AUTO(SORTKEY) cases.
• Prevented out-of-memory errors in cell-by-cell and row-hashing comparisons in workers.
• Fixed L3 row-hashing producing false positives.
• Prevented unnecessary shared-cache eviction when the loaded copy already matches the workspace.
• Fixed row-hashing algorithm errors and now surfaces duplicates and missing rows distinctly (breaking change).

v0.9.1

Improvements

• Improved Cloud Data Validation column metrics performance by consolidating per-column CTEs into a single wide-row query.

Bug fixes

• Fixed aggregate overflow on STDDEV and VARIANCE in Cloud Data Validation by casting SUM/AVG/STDDEV inputs to FLOAT; removed the VARIANCE metric.
• Fixed migration 0015 failing on Snowflake accounts that do not support ALTER DATABASE ... SET EVENT_TABLE; the event table is now set only when a Snowpipe-based data migration starts.

v0.9.0

Improvements

• Vertical partitioning for cell validation on wide tables.
• Granular per-table L1/L2/L3 progress in the validation dashboard.
• Validated table-level ROW_COUNT during schema validation.
• Created Data Validation Snowpipes synchronously and restored the task_queue argument in the hybrid validation handler.
• Table-centric Streamlit dashboards with CSV export.

Bug fixes

• Fixed timestamp copy handling for SQL Server BCP loads.
• Fixed duplicate tasks created when evaluating L1 results under race conditions.
• Fixed decimal partition coercion and parallelized L3 validation fixes.
• Fixed orchestrator logging to stderr instead of stdout.

v0.8.0

New features

• Added hybrid row validation mode — two-phase MD5 + cell drilldown.
• Added support for smart partitioning in Cloud Data Validation.
• Added load segmentation for multi-file COPY INTO.
• Added DEFAULT normalization templates for various data types.

Improvements

• Improved result set snapshots validation.
• Added a Table Progress tab to the Streamlit dashboard for Data Validation.
• Improved Data Validation performance.
• Included thread name and ID in log output for easier troubleshooting.
• Improved the task queue to support a higher number of parallel workers.

Bug fixes

• Cloud Data Validation now defaults to False when null in the workflow config.
• Fixed SQL compilation memory exhaustion by batching L2 metrics queries for wide tables.
• Fixed cell validation key mismatch, partition WHERE clause handling, and metrics payload cleanup.
• Fixed duplicate rows caused by an orchestrator crash during COPY INTO.
• Fixed an issue with the incremental sync watermark on Redshift.
• Fixed usage of the vectorized scanner.

v0.7.2

Improvements

• Log package and Python versions when the orchestrator starts.

Bug fixes

• Snowflake: recover cleanly when a session expires instead of staying stuck in a bad state.
• Task queue: avoid unblocking tasks before predecessor rows exist (stored procedures and schema migration).
• Improve generation of data validation queries for Teradata: handle very large counts and optional WHERE filters correctly.

v0.7.1

Improvements

• Cloud Data Validation dashboard: paginate large tab views for easier browsing.
• Cloud validation: load large query results in smaller batches for the dashboard and query helpers.

Bug fixes

• Data migration partition planning no longer caps the number of partitions with a fixed maximum.
• Snowflake connectivity: more resilient sessions and automatic retry for transient failures.