Snowflake Data Exchange Agent

The Data Exchange Agent is the Worker component of the Cloud Data Migration solution. It connects to source databases (SQL Server, Amazon Redshift, Teradata, Oracle), extracts data, and uploads it to Snowflake stages for ingestion by the Data Migration Orchestrator (snowflake-data-migration-orchestrator).

The same worker process also executes Cloud Data Validation tasks (data_validation) when the orchestrator schedules them. That path relies on the optional snowflake-data-validation package being installed in the worker environment (see the orchestrator documentation for creating validation workflows and JSON configuration).

Installation

pip install snowflake-data-exchange-agent

For Teradata sources using the native teradatasql driver (recommended over ODBC when you can use it), install the optional extra:

pip install snowflake-data-exchange-agent[teradata]

For Oracle sources using python-oracledb in thin mode (no Instant Client or ODBC required for typical EZ Connect), install the optional extra:

pip install snowflake-data-exchange-agent[oracle]

If oracledb is not installed, the agent falls back to Oracle ODBC + pyodbc (odbc_driver must match pyodbc.drivers()). Connections always try oracledb first when that package is present (same pattern as Teradata / teradatasql).

Legacy accounts (thick mode): thin mode cannot authenticate accounts whose password is stored only with the old 10G verifier — it raises DPY-3015: password verifier type 0x939 is not supported by python-oracledb in thin mode. For those, enable thick mode so oracledb uses the Oracle Client (OCI) libraries, which support the legacy verifier. This needs only the Oracle Instant Client "Basic" libraries on the host (no unixODBC or registered ODBC driver):

[connections.source.oracle]
oracle_connection_mode = "basic"
host = "oracle-host.example.com"
port = 1521
database = "FREEPDB1"
username = "sdv_user"
password = "secret"
oracle_thick_mode = true
# Optional: omit to rely on LD_LIBRARY_PATH / the system library search path.
oracle_client_lib_dir = "/home/user/oracle/instantclient_21_1"

The Instant Client is just an unzipped archive (no sudo/install step). Thick mode is process-global, so once enabled every Oracle connection in the agent process uses it.

For local development against the SnowConvert CLI worker venv, scripts/install-into-cli-venv.sh installs the [oracle] extra. SnowConvert Desktop must install snowflake-data-exchange-agent[oracle] for Oracle migrations until the desktop installer is updated to match.

Python Version: 3.11 or higher

Usage

The agent provides two subcommands: run (default) and test.

# Start with a configuration file
data-exchange-agent run -c <configuration-file-path>

# Start with default configuration.toml in current directory
data-exchange-agent run

# Omitting the subcommand defaults to 'run' (backward compatible)
data-exchange-agent -c <configuration-file-path>

# Custom parallelism and port
data-exchange-agent run --max-parallel-tasks 8 --port 8080

# Task-handling only, without the HTTP server (for multi-worker setups)
data-exchange-agent run --no-server

# Custom base directory for exported files (overrides config)
data-exchange-agent run --local-results-directory /mnt/dea-exports

# Debug mode
data-exchange-agent run --debug --port 5001

# Test all configured connections (executes SELECT 1)
data-exchange-agent test -c <configuration-file-path>

Run Command Options

Worker Configuration

The Worker configuration file uses TOML format.

When selected_task_source is snowflake_stored_procedure, the worker issues CALL statements against the task-queue using application.snowflake_database_for_metadata and application.snowflake_schema_for_data_migration_metadata. For data-validation tasks it also resolves the results stage and CSV_FILE_FORMAT object using the validation schema resolution order above. These settings are independent of Snowflake connection session defaults (SNOWFLAKE_DATABASE, SNOWFLAKE_SCHEMA in the connection profile).

All [application] keys follow CLI → TOML → default precedence via ConfigManager, except validation schema which also falls back to CUSTOM_SNOWFLAKE_SCHEMA_FOR_DATA_VALIDATION_METADATA when the TOML key is unset.

Unrecognized top-level keys in configuration.toml emit a warning at load time (they are ignored). Use only the keys documented in this reference and in configuration_example.toml.

Environment variables

The worker reads a small set of environment variables in addition to TOML/CLI config. In SPCS, docker entrypoints often copy orchestrator CUSTOM_SNOWFLAKE_* values into TOML at startup so runtime resolution stays TOML-first.

Snowflake SPCS injects SNOWFLAKE_ACCOUNT, SNOWFLAKE_HOST, SNOWFLAKE_WAREHOUSE, SNOWFLAKE_DATABASE, SNOWFLAKE_SCHEMA, and related service identity variables when the worker runs as a container service.

Example: SQL Server (Standard Authentication)

[connections.source.sqlserver]
username = "username"
password = "password"
database = "database_name"
host = "127.0.0.1"
port = 1433

For SQL Server CETAS extraction, use Azure SQL Managed Instance or SQL Server 2022+ with PolyBase. Azure SQL Database does not support CETAS; use cloud_direct there. Create the external data source and Parquet file format on the source, then add their names to the same connection:

[connections.source.sqlserver]
# Standard SQL Server connection fields above
cet_as_external_data_source = "MyBlobDataSource"
cet_as_file_format = "MyParquetFormat"
# cet_as_path_prefix = "migration/prod"
# cet_as_table_schema = "dbo"

SQL Server Iceberg extracts must use cet_as or cloud_direct. For Azure Blob cloud_direct, configure one supported authentication method:

[connections.target.blob]
container_name = "iceberg-landing"
account_name = "mystorageaccount"
use_default_credential = true
# Alternatively, set account_key or sas_token with account_name,
# or replace those fields with connection_string.

Example: Amazon Redshift (IAM Authentication)

[connections.source.redshift]
username = "demo-user"
database = "demo_db"
auth_method = "iam-provisioned-cluster"
cluster_id = "my-aws-cluster"
region = "us-west-2"
access_key_id = "your-access-key-id"
secret_access_key = "your-secret-access-key"

Example: Amazon Redshift (Standard Authentication)

Redshift connections use the native redshift_connector driver (not ODBC). IAM and standard username/password modes are supported.

[connections.source.redshift]
username = "myuser"
password = "mypassword"
database = "mydatabase"
host = "my-cluster.abcdef123456.us-west-2.redshift.amazonaws.com"
port = 5439
auth_method = "standard"

Example: Azure Synapse Analytics

Requires ODBC Driver 18 for SQL Server on the worker host. Use pool_type = "serverless" and database = "master" when the orchestrator runs with --source-platform azure_synapse_serverless; the DEA still uses [connections.source.azure_synapse].

REGULAR extraction (default) — ODBC fetch, Parquet on the worker, PUT to the Snowflake internal stage. Orchestrator: omit extraction or "strategy": "regular".

[connections.source.azure_synapse]
host = "myworkspace.sql.azuresynapse.net"
database = "MyDedicatedPool"
port = 1433
username = "sql_user"
password = "your_password"
mode = "sql_auth"
pool_type = "dedicated"
# encrypt = true
# trustServerCertificate = false

CETAS extraction — Synapse writes Parquet to Azure Blob; the orchestrator loads via an external stage. Orchestrator: "strategy": "cet_as" plus externalStage in workflow JSON. Create on Synapse ahead of time: scoped credential, external data source, and Parquet file format.

Cloud-direct extraction — The worker streams ODBC cursor results directly to S3 or Azure Blob via PyArrow (no local disk, no upload step). Orchestrator: "strategy": "cloud_direct" plus externalStage in workflow JSON. Configure [connections.target.s3] and/or [connections.target.blob] on the worker; the external stage must point at the same bucket/container prefix. See docs/data-exchange-agent/cloud-direct-classic.md in the repository.

[connections.source.azure_synapse]
host = "myworkspace.sql.azuresynapse.net"
database = "MyDedicatedPool"
port = 1433
username = "sql_user"
password = "your_password"
mode = "sql_auth"
pool_type = "dedicated"

cet_as_external_data_source = "MyBlobDataSource"
cet_as_file_format = "MyParquetFormat"
# cet_as_path_prefix = "migration/prod"
# cet_as_table_schema = "dbo"

Example: PostgreSQL (ODBC)

Use this block when the orchestrator schedules Cloud Data Validation (or migration) against a PostgreSQL source. Install a PostgreSQL ODBC driver on the worker host; see pyodbc.drivers() for the exact odbc_driver string if you need to pin one.

[connections.source.postgresql]
username = "my_user"
password = "my_password"
database = "my_database"
host = "postgres.example.com"
port = 5432
# odbc_driver = "PostgreSQL Unicode"  # optional

Example: Teradata

The agent supports two Teradata drivers and automatically selects the best one available:

1. teradatasql (preferred) -- Pure Python driver. No OS-level ODBC installation required. Install with pip install teradatasql.
2. ODBC fallback -- If teradatasql is not installed, the agent falls back to pyodbc with the Teradata ODBC driver. Set odbc_driver to the exact name returned by pyodbc.drivers().

When teradatasql is available, odbc_driver is ignored and no ODBC driver needs to be installed on the host. Use dbc_name when your Teradata COP / TDPID alias differs from host.

Older configs used driver_name for the ODBC driver label; that key still works but is deprecated in favor of odbc_driver.

[connections.source.teradata]
host = "your-teradata-host.example.com"
port = 1025
database = "tpcds"
username = "your_username"
password = "your_password"
# odbc_driver = "Teradata Database ODBC Driver 17.20"  # only needed for ODBC fallback
# dbc_name = "TDPID_ALIAS"  # optional; defaults to host
# authentication = "LDAP"  # optional logon mechanism: TD2 (default), TDNEGO, LDAP, KRB5, JWT, BEARER
#                          # applies to native teradatasql (logmech) and ODBC fallback (AuthMech)
# jwt_token = "..."        # JWT token (logdata); used only on the native teradatasql path when authentication = "JWT"
# bearer_token = "..."     # OAuth bearer token (logdata); used on the native path when authentication = "BEARER"
#
# L3 data validation: install HASH_MD5 UDF in the same database as `database` above.
# See docs/data-migration-orchestrator/teradata-dv-prerequisites.md

# Optional: WRITE_NOS configuration for direct export to cloud object storage
# (S3, Azure Blob, or GCS). The orchestrator's `extraction.strategy` must be
# set to `"write_nos"` for the worker to use these settings.
#
# `write_nos_location_scheme` is one of `/s3/`, `/az/`, or `/gs/`.
# Provide credentials in exactly one of the three modes below.
#
# write_nos_location_scheme    = "/az/"
# write_nos_location_host      = "myaccount.blob.core.windows.net"
# Optional path segment after host (omit when Snowflake stage is bucket root):
# write_nos_location_container = "td-nos-exports"
#
# Mode 1 - Function mapping (recommended; credentials live on Teradata):
# write_nos_function_mapping = "nos_util.WRITE_NOS_FM"
#
# Mode 2 - Named AUTHORIZATION object (credentials live on Teradata):
# write_nos_authorization_name = "nos_util.DefAuth_Write"
#
# Mode 3 - Inline credentials (embedded in SQL; redacted in logs):
# write_nos_access_id  = "your_account_or_access_key_id"
# write_nos_access_key = "your_secret"
#
# Optional output overrides (defaults shown):
# write_nos_stored_as       = "PARQUET"
# write_nos_compression     = "SNAPPY"
# write_nos_max_object_size = "16MB"
# write_nos_overwrite       = "TRUE"
#
# Optional: TPT (Teradata Parallel Transporter). Use when the orchestrator sets
# `extraction.strategy` to `"tpt"`. Requires TTU (`tbuild`) on PATH or set
# `TPT_TBUILD_EXECUTABLE` to its absolute path. Intermediate delimited files and
# job scripts use `tpt_output_directory` / `tpt_log_directory` on the worker.
#
# tpt_output_directory = "/var/dea/tpt"
# tpt_log_directory    = "/var/log/dea/tpt"
# tpt_delimiter        = "|"
# tpt_max_sessions     = 4
# tpt_charset          = "UTF8"

Example: PostgreSQL

The agent supports PostgreSQL via ODBC. For bulk data_movement tasks, psql \copy is used by default (requires psql installed on the worker host and on PATH). If psql is not available, the agent falls back to ODBC parquet automatically. Set use_copy = false to always use ODBC.

[connections.source.postgresql]
username = "my_user"
password = "my_password"
database = "my_database"
host = "postgres.example.com"
port = 5432
# use_copy = false   # uncomment to always use ODBC parquet instead of psql \copy

Example: Oracle (ODBC)

Install an Oracle Instant Client and the matching ODBC driver on the worker host. Use pyodbc.drivers() to find the exact odbc_driver string if the default label below does not match your installation.

Basic mode uses EZ Connect: service name in database, host/port for the listener.

[connections.source.oracle]
oracle_connection_mode = "basic"
username = "my_user"
password = "my_password"
database = "ORCLPDB1"
host = "oracle.example.com"
port = 1521
odbc_driver = "Oracle in instantclient_21_1"
auto_detect_driver = false

TNS alias mode sets DBQ to a name from tnsnames.ora. When tns_admin is set, TNS_ADMIN is applied only for the duration of pyodbc.connect so the driver can resolve the alias.

[connections.source.oracle]
oracle_connection_mode = "tns_alias"
tns_name = "MY_ORACLE_SERVICE"
username = "my_user"
password = "my_password"
odbc_driver = "Oracle in instantclient_21_1"
auto_detect_driver = false
tns_admin = "/path/to/tns"

Connect descriptor mode passes a full net descriptor as DBQ (for advanced routing, LDAP alternatives, or paste-in from tnsnames.ora).

[connections.source.oracle]
oracle_connection_mode = "connect_descriptor"
connect_descriptor = "(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle.example.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=ORCLPDB1)))"
username = "my_user"
password = "my_password"
odbc_driver = "Oracle in instantclient_21_1"
auto_detect_driver = false

Optional wallet_directory / wallet_password append common Oracle ODBC wallet attributes; TLS naming and LDAP directory options vary by driver — add driver-specific keys via extra options or extend configuration as needed.

DBMS_CLOUD (S3 export inside Oracle): set dbms_cloud_credential_name and dbms_cloud_file_uri_prefix on the Oracle source connection. For local development, use the ghcr.io/oracle/adb-free image (not Oracle Database 23ai Free XE), wallet + DataGrip steps, S3 credential SQL, and troubleshooting — see data-exchange-agent/docs/oracle-dbms-cloud-local-setup.md in the repository. Validate IAM first with scripts/test_s3_bucket_access.py.

Note: Only one source connection is needed. The Snowflake target connection should point to a valid entry in your ~/.snowflake/config.toml.

Custom Extraction Plugins

For Data Migration workflows, you can override how a source is extracted with your own Python class instead of the built-in extractors. A plugin receives the source connection fields and a query, runs the query against that source, and lands the result files where the migration expects them. This is useful for sources or drivers the agent does not support natively.

Steps to add a custom extractor:

1. Write a class that subclasses FileSystemExtractionPlugin (recommended) or ExtractionPlugin (see below) and produces output in the canonical file format.
2. Make the class importable by the worker — install it as a package into the same environment as the agent, or put it on the worker's PYTHONPATH.
3. Add plugin_class (the fully-qualified dotted path) to the source connection in your worker config.
4. Start the worker as usual. When a task for that source is processed, the plugin owns the entire extract-and-upload flow; the built-in extractor is not used.

Step 1 — write your plugin class. There are two ways to implement a plugin.

Option A — extend FileSystemExtractionPlugin (recommended) and implement only write_dataset. The base class validates the output, uploads it to the workflow target, and cleans up the local directory for you. A complete, runnable example that writes conforming Parquet with PyArrow:

import pyarrow as pa
import pyarrow.parquet as pq

from data_exchange_agent.plugins import FileSystemExtractionPlugin

class MyExtractor(FileSystemExtractionPlugin):
    def write_dataset(self, statement: str, output_dir: str, base_file_name: str) -> None:
        # self.connection holds the fields you configured for this source
        # (host, port, username, password, database, ...). For ODBC-based
        # sources a ready-to-use "connection_string" is also included.
        connection = my_driver.connect(self.connection["connection_string"])
        cursor = connection.cursor()
        cursor.execute(statement)

        column_names = [desc[0] for desc in cursor.description]
        rows = cursor.fetchall()
        # Preserve the source column names — the loader reads columns by name.
        table = pa.table(
            {name: [row[i] for row in rows] for i, name in enumerate(column_names)}
        )

        out_path = f"{output_dir}/{base_file_name}_001.parquet"
        pq.write_table(table, out_path, compression="snappy")

Replace my_driver with whatever client connects to your source. Everything else (uploading the Parquet, cleanup, and the output guard) is handled by the base class.

Option B — implement ExtractionPlugin directly when the data does not pass through the local filesystem (for example, a source-side EXPORT/UNLOAD that writes straight to cloud storage). Implement extract, which owns the full flow:

from data_exchange_agent.plugins import ExtractionPlugin

class MyDirectExtractor(ExtractionPlugin):
    def extract(self, statement: str) -> bool:
        # self.connection, self.context.destination, self.context.uploader are available.
        my_client.export_to_cloud(self.connection, statement, self.context.destination)
        return True

Constructor contract (called by the worker): __init__(self, connection, context) where connection is a dict of this platform's connection fields and context is an ExtractionContext with results_folder_path, base_file_name, destination (the upload target), uploader, and logger.

Your output must conform to the canonical file format so the migration's COPY INTO can load it:

• Parquet (preferred): PyArrow/snappy defaults; one file named <base_file_name>_001.parquet; column schema derived from the query result. Prefer Parquet — it avoids all delimiter/encoding ambiguity.
• CSV: field delimiter \x1f (unit separator), record delimiter \x1e (record separator), UTF-8, no header row; write NULL as an empty field. File name ends in .csv (or .csv.gz).

Files that do not match are misread or rejected at load time. The base class performs a lightweight guard (it fails the task if write_dataset produces no .parquet/.csv/.csv.gz file), but it cannot validate delimiters or schema — conformance is the plugin's responsibility.

Step 2 — make the class importable. Install it as a package into the same environment as the agent, or put it on the worker's PYTHONPATH, so the worker can import the dotted path you set next.

Step 3 — add plugin_class to the source connection. Your plugin receives this connection's configured fields as a dict (via self.connection). Only fields valid for that platform are allowed — the config rejects unknown keys — so you get the standard fields (host, port, username, password, database, ...), plus a ready-to-use connection_string for ODBC-based sources. For ODBC sources you can also stash extra key/values under [connections.source.<name>.extra_options].

[connections.source.sqlserver]
plugin_class = "acme_ext.my_extractor.MyExtractor"
# Standard fields for this platform — passed to your plugin as a dict.
username = "my_user"
password = "my_password"
database = "SampleStoreDB"
host = "sqlserver.example.com"
port = 1433

Security: plugin_class is read only from this worker configuration, never from a task. Loading it imports and runs your code in-process with the worker's privileges. Only configure classes you trust and control.

Step 4 — run the worker. Start the worker as usual (data-exchange-agent run -c configuration.toml). When a task for that source is processed, the plugin owns the entire extract-and-upload flow; the built-in extractor is not used.

Idle worker shutdown

Task sources with idle shutdown (supports_idle_shutdown on the adapter; today the Snowflake stored-procedure source): configure idle exit so the worker terminates after sustained genuinely idle operation — not merely an empty PULL_TASKS, but confirmed quiet for this worker's context (no active non-terminal leases for this agent_id, no near-term pullable work or recent queue churn for this affinity, healthy Snowflake connectivity). Before exit it runs a final confirmation pass, inserts WORKER_SHUTDOWN into SYSTEM_EVENT (orchestrator migration 0029), emits Snowhouse stop telemetry, then drains workers (lease refresher first, then bounded worker joins), flushes logs, closes Snowflake sessions, and exits via shared.runtime_forced_exit.forced_idle_process_exit (os._exit(0)). Other task sources ignore idle shutdown settings.

Canonical (worker TOML):

[idle_shutdown]
minutes = 60   # always on; use 10080 (one week) for long-lived local dev

Idle shutdown is always enabled (secure-by-default). Resolution: ConfigManager (CLI → TOML → defaults), then optional DM_IDLE_SHUTDOWN_MINUTES env override when present in os.environ.

Invalid or <=0 minutes (env) clamp to 60 with WARNING. TOML minutes must be >=1. INFO is logged when env minutes differ from the TOML/CLI/default base.

Apply migrations 0029 (SYSTEM_EVENT) and 0037 (worker pre-exit idle safety net) before running DEA with idle shutdown, or audit / safety checks fail. With the HTTP server enabled (default), it stops when idle shutdown runs; use --no-server for worker-only smoke. Idle shutdown rows include PAYLOAD.host from shared.runtime_host.system_event_host() (spcs:… when SNOWFLAKE_SERVICE_* is set, otherwise host: + OS hostname). See ExplicitIdleShutdownSafetySpec.md §5.4.

In SPCS, set DM_IDLE_SHUTDOWN_MINUTES on both orchestrator and DEA service specs to the same value when overriding.

Log file location

The worker writes a rotating log file (data_exchange_agent.log, 10 MB × 5 backups) alongside console output. By default the log file is created under ~/.data_exchange_agent/logs. Set the DATA_EXCHANGE_AGENT_LOGS_DIR environment variable to write logs to a different directory — useful in containers/SPCS, when the home directory is read-only, or when you want logs on a mounted volume.

export DATA_EXCHANGE_AGENT_LOGS_DIR=/var/log/dea
data-exchange-agent run -c configuration.toml

The value is resolved at startup: a leading ~ is expanded, the path is resolved to an absolute location, and the directory is created if it does not exist. In SPCS, set DATA_EXCHANGE_AGENT_LOGS_DIR on the DEA service spec.

ODBC Driver Auto-Detection

SQL Server connections prefer Microsoft's native mssql-python driver, installed via the sqlserver extra (pip install "snowflake-data-exchange-agent[sqlserver]"). It bundles its own ODBC binaries, so no unixODBC driver manager or msodbcsql install is needed. When it is not installed — or when extra_options contains a keyword it does not accept — the agent falls back to pyodbc and automatically detects the best available ODBC driver, preferring the newest (ODBC Driver 18 > 17 > 13 > 11). If a specific driver is requested but not found, it falls back to the best available driver with a warning.

Session identification differs by path. On the ODBC path the reserved identifier SNOWFLAKE-AIM-DMV/<version> is written to program_name via the APP keyword. The native driver reserves APP for its own telemetry, so it is written to the session's CONTEXT_INFO instead:

SELECT session_id, login_name, CAST(context_info AS varchar(128)) AS tool_identifier
FROM sys.dm_exec_sessions
WHERE context_info IS NOT NULL;

Set set_context_info = false on the connection if the deployment already uses CONTEXT_INFO for its own purposes.

To manually specify a driver:

[connections.source.sqlserver]
odbc_driver = "ODBC Driver 17 for SQL Server"

Encryption (SQL Server)

The encrypt and trust_server_certificate parameters are optional. When set, they apply to both the native and ODBC paths. When omitted, each driver applies its own default:

• mssql-python (native, preferred): Encryption is mandatory by default.
• ODBC Driver 18 and above: Encryption is mandatory by default.
• ODBC Driver 17 and below: Encryption is disabled by default.

An unset encrypt is never inherited across paths. A host with only ODBC 17 installed still gets an encrypted connection on the native path, and the agent logs a warning so the change is visible. If the server cannot terminate TLS, set encrypt = false explicitly; if it presents a self-signed certificate, set trust_server_certificate = true.

[connections.source.sqlserver]
username = "sa"
password = "mypassword"
database = "mydb"
host = "my-server.example.com"
port = 1433
encrypt = true
trust_server_certificate = false

For development environments or SQL Servers without encryption support, either omit the encryption parameters or set encrypt = false.

Query modifiers (anti-locking hints)

Optional per-connection query modifiers reduce lock contention on the source during extraction and data-validation reads. Configure under the source connection block:

[connections.source.sqlserver.query_modifiers]
object_modifier = " WITH (NOLOCK)"
# select_modifier = " TOP 100 PERCENT"  # optional; after SELECT

Platform behavior:

• Teradata: LOCKING ROW FOR ACCESS is applied automatically on source reads; no user configuration required.
• SQL Server / Azure Synapse: opt-in via object_modifier (e.g. WITH (NOLOCK)). Dirty reads are possible — use only when acceptable for your workload.
• Oracle: when select_modifier is unset, a default PARALLEL hint may be applied on source reads.

Workflow-level queryModifiers in migration or validation JSON merge with connection-level settings (table/workflow overrides win). See the orchestrator Workflow Configuration Reference for JSON shape.

Secrets

Connection passwords and other sensitive fields can reference external secret stores instead of plain text.

Command substitution — embed a shell recipe in a string field:

[connections.source.sqlserver]
password = "$(aws secretsmanager get-secret-value --secret-id my/dea/sql --query SecretString --output text)"

Enable with SECRET_MANAGERS_ALLOW_CMD_SUBSTITUTION=true or a separate secret-managers config file (below).

REST providers — register HTTP secret backends in a TOML file pointed to by SECRET_MANAGERS_CONFIG_FILE:

[secret_managers.providers.vault]
type = "rest"
base_url = "https://vault.example.com/v1/secret/data/dea"

Do not put [secret_managers] inline in configuration.toml — the worker loader ignores unrecognized top-level sections. Use SECRET_MANAGERS_CONFIG_FILE or the env vars below.

Secrets are resolved at config load time via the shared snowflake-secret-managers library (installed with the agent).

Rate limiting

The orchestrator stores rate-limit rules in Snowflake (DATA_MIGRATION.RATE_LIMIT by default). Each rule caps concurrent executing tasks whose scope matches a SQL LIKE pattern, optionally filtered by workflow ID and affinity.

Every worker task pull goes through PULL_TASKS with capacity 1, so rate limiting is always applied to DEA workers regardless of max_parallel_tasks. Rules are a soft cap — under concurrent polling by multiple worker processes, executing tasks can briefly overshoot the target (roughly up to ~2× with a large worker fleet).

max_parallel_tasks sets the number of in-flight worker threads the agent runs concurrently. It does not change the pull batch size — each thread still fetches one task at a time, so every claim is subject to rate-limit rules.

Insert or update rules with SQL against your migration metadata schema (same database/schema as the task queue). The orchestrator README includes example INSERT statements. Orchestrator batch pulls do not enforce rate limits — only worker pulls do.

Query Tagging

The Worker automatically sets Snowflake's QUERY_TAG session parameter on every query it submits. Tags are compact JSON strings containing identifiers such as the workflow ID, task ID, and worker version. You can use these tags to filter and attribute Worker queries in QUERY_HISTORY:

SELECT query_text, query_tag, start_time
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
WHERE TRY_PARSE_JSON(query_tag):DMVF_WORKFLOW_ID IS NOT NULL
ORDER BY start_time DESC;

Changelog

v1.35.0

New features

• Added SQL Server BCP and PostgreSQL COPY signature extraction.
• Added SQL Server CETAS extract and Iceberg cloud_direct packaging.

Improvements

• Improved Teradata view L1 precision via DBC.ColumnsQV.

Bug fixes

• Fixed DEA doctor TCP reachability probe to retry before failing.

v1.34.0

New features

• Added support for arbitrary columnMappings in L3 validation.

Improvements

• Improved BigQuery checksum expression for incremental synchronization with CHECKSUM strategy.

Bug fixes

• Fixed file-path escaping for the Data Validation PUT local source.

v1.33.0

New features

• Added selectModifier support for customizing the SELECT clause in query modifiers.

Improvements

• Switched to Microsoft's native mssql-python driver for SQL Server connections, with ODBC fallback.
• Improved Teradata logical text comparison across different character sets.
• Added prefer_native_driver opt-out to allow ODBC fallback when the native driver is not preferred.
• Reduced task queue contention when many threads across many workers access the queue concurrently.

Bug fixes

• Fixed data exchange worker upload crash and guarded against preprocessing hangs.

v1.32.0

Improvements

• Increased max supported length for CLOB/BLOB migration in Teradata.

Bug fixes

• Fixed DB2 extraction by switching from delimited text to Parquet and correcting the shared array builder for native DB2 types.

v1.31.0

Bug fixes

• Fixed race condition related to Snowpipe ingestion of data validation results.

v1.30.0

New features

• Added Iceberg v2-compatible L1 and L3 profile support.
• Added Cloud-Direct extraction strategy for writing data files directly to cloud storage without writing to disk first.
• Added DB2 as a supported platform.
• Added Iceberg-aware runtime writers for SPCS environments.

Improvements

• Improved Iceberg Parquet packaging with proper codec selection, typed timestamps, and UTC homogenization.
• Improved configuration validation by warning on unrecognized configuration.toml keys.
• Switched to native redshift_connector for improved Redshift connectivity.

Bug fixes

• Fixed backward compatibility for IPv4 network configurations.

v1.29.0

New features

• Added rate limiting features for controlling request throughput.

Improvements

• Improved text codec column matching to be case-insensitive across source and target.

v1.28.2

New features

• Added Azure AD authentication modes for the SQL Server connector.
• Added L3 signature extract handler for ODBC and native connectors.

Improvements

• Improved Redshift, Teradata, and Oracle signature extract charset parity across connectors.

Bug fixes

• Fixed DATA_SOURCE_PORT being incorrectly required in the SQL Server image configuration.
• Fixed Redshift normalization issues affecting data exchange accuracy.

v1.28.1

No user-facing changes in this release.

v1.28.0

New features

• Added L3 signature comparison for Parquet and External Table pushdown.
• Added L3 control plane with configuration, registry, task types, and gating.
• Added L3 signature partitioning and unordered MD5 SELECT.
• Added trusted connection and Azure AD authentication modes for Synapse.
• Added port forwarding support for BCP.

v1.27.4

New features

• Added BCP (Bulk Copy Program) to the Docker image for SQL Server data exchange workflows.

v1.27.3

New features

• Added base VECTOR type support for data exchange.
• Added runtime teradatasql driver installation in the unified SPCS image.
• Added support for additional Teradata authentication options.

Bug fixes

• Fixed Data Validation workflows failing when a custom DV metadata schema is configured.

v1.27.2

No user-facing changes in this release.

v1.27.1

No user-facing changes in this release.

v1.27.0

New features

• Added support for Oracle's BOOLEAN data type in data exchange.
• Added INTERVAL data type support for platforms with mixed-precision intervals (Postgres, BigQuery).
• Added session tagging for SQL Server and Azure Synapse connections.
• Added session tagging for PostgreSQL and Redshift connections.
• Added support for custom extraction plugins.
• Added a watchdog to detect and handle parent process termination.
• Added support for customizable normalization expressions in row-hashing validation and cell-by-cell comparison.

Improvements

• Simplified Snowpipe drain logic for improved reliability.
• Added staged file readability verification before task completion.
• Renamed product session tag to SNOWFLAKE-AIM-DMV.

Bug fixes

• Fixed Oracle CLOB/NCLOB/BLOB truncation at 4000 bytes during data exchange.
• Fixed pre-flight disk check to honor local_results_directory setting.

v1.26.0

New features

• Added charset-aware normalization to query generators.
• Added Teradata non-Latin column UTF-8 hex hashing for server-side row validation.
• Added key inference for Azure Synapse, BigQuery, and SQLite.
• Added support for the INTERVAL data type.
• Added support for overriding the Data Exchange Agent logs directory.
• Added BigQuery native driver and connection checks to the Data Exchange Agent doctor.
• Added Teradata query band session tagging.
• Added BigQuery query job labels session tagging.
• Added --local-results-directory flag to the data exchange worker CLI.
• Added Oracle DBMS_APPLICATION_INFO session tagging.

Improvements

• Retired standalone preflight script; wired preflight as a task-graph configuration flag.

v1.25.1

New features

• Added Oracle thick-mode support for authenticating against accounts with legacy password hashes.
• Added readiness checks for PULL_TASKS, database, and schema before starting validation.

v1.25.0

New features

• Added Azure Synapse support to the unified worker Docker entrypoint.
• Added BigQuery support for cloud Data Validation (L1/L2/L3).

Improvements

• Improved doctor command performance with connection reuse, ODBC diagnostics, compact panel layout, and flow-scoped grants.

v1.24.2

New features

• Added wall-clock execution timeout for boundary analysis tasks.
• Added Teradata LOCKING ... FOR ACCESS modifier stripping before SQL command classification.
• Added BigQuery Docker entrypoint support with credential handling and config template.

Improvements

• Improved float canonical rendering to full precision for L2/L3 validation.
• Improved numeric canonical rendering with scale-aware handling for T-SQL and Snowflake.
• Added query modifier support for L3 DEA operations.

Bug fixes

• Fixed Teradata L3 row-hash computation.
• Fixed DEA config file writing under $HOME so appuser can start on SPCS.

v1.24.1

New features

• Added wall-clock execution timeout for boundary analysis tasks.
• Added scale-aware numeric canonical rendering for T-SQL and Snowflake.
• Added query modifier support for SQL Server and Synapse.
• Added BigQuery Docker entrypoint support with credential handling and config template.

Improvements

• Improved float canonical rendering to full precision for L2/L3 validation.

Bug fixes

• Fixed Teradata LOCKING ... FOR ACCESS modifier stripping before SQL command classification.
• Fixed Teradata L3 row-hash computation.

v1.24.0

No user-facing changes in this release.

v1.23.1

New features

• Added query modifiers foundation with resolver, DEA config, and scan registry.

Improvements

• Introduced dependency container for service lifecycle management.

Bug fixes

• Fixed BigQuery cloud-extraction type mapping and timestamp correctness.
• Fixed Oracle NUMBER, BLOB, LONG, LONG RAW, and XMLTYPE types that were landing zero rows during extraction.
• Removed unnecessary stage refresh after upload for data extraction tasks, resolving concurrent ALTER STAGE REFRESH failures.

v1.23.0

New features

• Added configurable text comparison mode (logical vs raw) for column-level validation.
• Added SQLite as a supported source platform for data validation.
• Added DATA_SOURCE_ENCRYPT and DATA_SOURCE_TRUST_SERVER_CERTIFICATE optional connection lines for SQL Server.

Improvements

• Implemented Oracle ISO 8601 timestamp normalization for Snowflake migration.
• Enhanced timestamp normalization for Teradata with fractional-second precision.
• Decoupled target platform for data validation, enabling non-Snowflake target comparisons.
• Migrated schema migrations to the Cloud State library with backwards compatibility.

Bug fixes

• Fixed handling of missing database attribute on connection config to fail gracefully instead of crashing.
• Fixed row-hash early-stop so all failed rows are reported when early stopping is disabled.
• Fixed empty-source handling to record an explicit validation failure instead of silently passing.

v1.22.1

Bug fixes

• Fixed validation result uploads failing on Windows by normalizing local file paths for the Snowflake PUT command.

v1.22.0

Improvements

• Hardened worker idle shutdown with a pre-exit safety gate.

Bug fixes

• Added fail-fast when the worker database does not match the task database on Redshift.

v1.21.0

Improvements

• Made hybrid validation mode the default: row-level comparison runs first; cell-level drilldown applied to any mismatches, replacing explicit row or cell mode selection.
• Removed deprecated CHUNK_HASHES table and associated COPY INTO and pipe definitions; a schema migration is applied automatically on upgrade.

Bug fixes

• Fixed partition row size estimation for views to use sampled rows instead of catalog size.
• Fixed data validation to fail when the Snowflake target table is empty.

v1.20.5

Improvements

• Made the COPY extraction strategy the default for PostgreSQL.

v1.20.4

New features

• Added CETAS-based data migration support for Azure Synapse.
• Added non-hybrid metadata storage mode.
• Added edge partitions to validation query generation.

Improvements

• Improved doctor subcommand with friendlier wording, native-driver detection, and target-objects existence checks.
• Expanded PostgreSQL column metrics templates.
• Switched Snowpipe REFRESH to use PATTERN-scoped pipes instead of PREFIX.

Bug fixes

• Fixed NoneType comparison crash in row-hash when index keys are NULL.

v1.20.3

New features

• Added external secret-managers framework.

Improvements

• Deprecated oracledb_preferred and set oracledb by default.
• Refactored Streamlit dashboard to show Data Validation run progress and metadata.

Bug fixes

• Corrected import paths, ConfigManager API, and label clarity.

v1.20.2

New features

• Added ODBC data migration support for Azure Synapse serverless SQL pool.

Improvements

• Improved L3 row validation with a safeguard for mismatched target columns.

Bug fixes

• Fixed identifier quoting: uppercase identifiers remain unquoted while lowercase or mixed-case identifiers are auto-quoted.

v1.20.1

New features

• Added Azure Synapse Analytics connection configuration and ODBC output converters for Synapse-specific types.

Bug fixes

• Fixed intermediate partition threshold calculation to use the next partition's initial index as the exclusive boundary, preventing unmoved rows on partition edges.

v1.20.0

New features

• Added doctor subcommand for the data exchange agent to support scai data doctor.
• Added doctor subcommand for the data migration orchestrator to support scai data doctor.
• Added support for Oracle quoted and mixed-case identifiers in catalog and object-type queries.
• Added unified Teradata Docker support with teradatasql, tbuild fallback, and a configurable image registry database.

Improvements

• Improved L3 row validation by skipping per-partition information_schema queries.
• Updated WRITE_NOS execution to use teradatasql as the primary driver, with ODBC as a fallback.

Bug fixes

• Fixed TeradataConnectionConfig ODBC driver check when teradatasql is installed.

v1.13.2

Bug fixes

• Fixed L3 row validation to honor column-selection and column-mapping config so source and target tables with different column counts no longer crash.

v1.13.1

New features

• Added DBMS_CLOUD extraction so Oracle can unload directly to an external S3 stage without an agent upload step.
• Added external-stage JSON load and stage file detection for DBMS_CLOUD.

Improvements

• Improved Oracle wallet handling: honored tns_admin when wallet_directory is unset, and set both config_dir and wallet_location for oracledb ATP connections.

v1.13.0

New features

• Added Oracle ODBC extraction with staged Parquet metadata and improved ODBC robustness.
• Replaced standalone Oracle connector with oracledb-backed connection and added an Oracle Docker image.
• Added Azure Synapse connection support (Phase 1).
• Added Azure Synapse CLI and agent integration (Phase 5).
• Added PostgreSQL bulk extract via psql \copy with PG_CSV_FILE_FORMAT support.
• Added PostgreSQL support to the unified data exchange agent.
• Added Teradata use_tpt_for_bulk flag and TPT typed Parquet coercion.

Improvements

• Enhanced task telemetry by integrating MigrationTracker and current-step tracking.
• Refactored Teradata extraction strategies to unify REGULAR and TPT handling.
• Made write_nos_location_container optional in WRITE_NOS configuration and removed the external stage URL and storage integration requirement.
• Improved Oracle fully-qualified-name and view validation handling.
• Updated VARCHAR limits in validation templates and constants.
• Improved row-hashing performance.

Bug fixes

• Fixed custom-template model imports.

v1.12.0

New features

• Added view validation support to the Cloud Data Validation pipeline.
• Added Oracle ODBC source support.
• Added Teradata type mappings and Snowflake target fully-qualified-name helpers.
• Added Teradata object_type query in the shared dispatcher.
• Added Teradata to the data-migration-orchestrator workflow.
• Added Teradata ODBC source support.
• Added early-stopping support for L3 row-hashing validation.
• Added support for custom metrics and templates in Cloud Data Validation.
• Added Oracle Data Validation foundation with L1 schema validation.
• Added L3 row and cell MD5 validation for Oracle.
• Added Oracle wiring and factory registration in the SDV core.
• Added the teradata optional install extra (pip install snowflake-data-exchange-agent[teradata]).
• Added support for early stopping, hybrid L3, and Snowpipe (breaking change).
• Added Snowflake schema utilities and type-mapping updates for the orchestrator.
• Added TPT and WRITE_NOS data sources for Teradata extraction.
• Added TPT and WRITE_NOS integration in Teradata workflow tasks.
• Added a metrics skill and PostgreSQL metrics templates.
• Added PostgreSQL connector with L0 and L1 validation.
• Added Oracle as a supported Data Validation source across the orchestrator and agent.
• Added L2 and L3 row and cell validation for PostgreSQL.
• Added PostgreSQL support to Cloud Data Validation.

Improvements

• Optimized L3 row-hashing queries.
• Treated wrapped (200010) PULL_TASKS lock-wait error as transient so the worker retries instead of failing.
• Extended Teradata ODBC connection configuration.

Bug fixes

• Made BCP stdout/stderr pump threads daemon=True to prevent worker hangs on shutdown.
• Prevented out-of-memory errors in cell-by-cell and row-hashing comparisons in workers.
• Fixed L3 row-hashing producing false positives.
• Prevented unnecessary shared-cache eviction when the loaded copy already matches the workspace.
• Fixed row-hashing algorithm errors and now surfaces duplicates and missing rows distinctly (breaking change).

v1.11.1

Improvements

• Improved column metrics query performance by consolidating per-column CTEs into a single wide-row query.

Bug fixes

• Fixed aggregate overflow on STDDEV and VARIANCE during data validation by casting SUM/AVG/STDDEV inputs to FLOAT; removed the VARIANCE metric.

v1.11.0

Improvements

• Cast value columns to Utf8 before unpivot and corrected IS_VALID evaluation.
• Vertical partitioning for cell validation on wide tables.

Bug fixes

• Fixed timestamp copy handling for SQL Server BCP loads.
• Fixed duplicate tasks created when evaluating L1 results under race conditions.
• Fixed decimal partition coercion and parallelized L3 validation fixes.

v1.10.0

New features

• Added hybrid row validation mode — two-phase MD5 + cell drilldown.
• Added DEFAULT normalization templates for various data types.

Improvements

• Improved result set snapshots validation.
• Improved Data Validation performance.
• Included thread name and ID in log output for easier troubleshooting.
• Improved the task queue to support a higher number of parallel workers.

Bug fixes

• Fixed SQL compilation memory exhaustion by batching L2 metrics queries for wide tables.
• Fixed an issue with the incremental sync watermark on Redshift.
• Fixed usage of the vectorized scanner.

v1.9.2

Improvements

• Log installed dependency versions and the Python runtime version at startup.

v1.9.1

Improvements

• Cloud data validation tasks read query results in batches instead of loading full result sets into memory.