snowflake-data-exchange-agent 1.4.0

Data exchange agent for migrations and validation

Snowflake Data Exchange Agent

The Data Exchange Agent is the Worker component of the Cloud Data Migration solution. It connects to source databases (SQL Server, Amazon Redshift), extracts data, and uploads it to Snowflake stages for ingestion by the Data Migration Orchestrator (snowflake-data-migration-orchestrator).

Installation

pip install snowflake-data-exchange-agent

Python Version: 3.11 or higher

Usage

# Start with a configuration file
data-exchange-agent -c <configuration-file-path>

# Start with default configuration.toml in current directory
data-exchange-agent

# Custom port and parallelism
data-exchange-agent --max-parallel-tasks 8 --port 8080

# Debug mode
data-exchange-agent --debug --port 5001

Worker Configuration

The Worker configuration file uses TOML format.

Section	Property	Type	Description
Top Level	selected_task_source	String	Currently should always be set to "snowflake_stored_procedure".
[application]	max_parallel_tasks	Integer	Maximum number of tasks the worker will process in parallel (using threads).
[application]	task_fetch_interval	Integer	Interval (in seconds) between attempts to fetch new tasks from the Orchestrator.
[connections.source.*]		Object	Configuration for source system connections. The Worker typically requires an ODBC driver. See examples below.
[connections.target.snowflake_connection_name]	connection_name	String	The name of the connection entry in the ~/.snowflake/config.toml file to use.

Example: SQL Server (Standard Authentication)

[connections.source.sqlserver]
username = "username"
password = "password"
database = "database_name"
host = "127.0.0.1"
port = 1433

Example: Amazon Redshift (IAM Authentication)

[connections.source.redshift]
username = "demo-user"
database = "demo_db"
auth_method = "iam-provisioned-cluster"
cluster_id = "my-aws-cluster"
region = "us-west-2"
access_key_id = "your-access-key-id"
secret_access_key = "your-secret-access-key"

Example: Amazon Redshift (Standard Authentication)

[connections.source.redshift]
username = "myuser"
password = "mypassword"
database = "mydatabase"
host = "my-cluster.abcdef123456.us-west-2.redshift.amazonaws.com"
port = 5439
auth_method = "standard"

Note: Only one source connection is needed. The Snowflake target connection should point to a valid entry in your ~/.snowflake/config.toml.

ODBC Driver Auto-Detection

The agent automatically detects the best available ODBC driver for SQL Server connections. If no odbc_driver is specified in the configuration, it will prefer the newest available driver (ODBC Driver 18 > 17 > 13 > 11). If a specific driver is requested but not found, it falls back to the best available driver with a warning.

To manually specify a driver:

[connections.source.sqlserver]
odbc_driver = "ODBC Driver 17 for SQL Server"

ODBC Encryption (SQL Server)

The encrypt and trust_server_certificate parameters are optional. By default, they are omitted from the connection string, allowing the ODBC driver to use its default behavior:

• ODBC Driver 17 and below: Encryption is disabled by default.
• ODBC Driver 18 and above: Encryption is mandatory by default.

[connections.source.sqlserver]
username = "sa"
password = "mypassword"
database = "mydb"
host = "my-server.example.com"
port = 1433
encrypt = true
trust_server_certificate = false

For development environments or SQL Servers without encryption support, either omit the encryption parameters or set encrypt = false.