fivetran-connector-sdk-core

The fivetran-connector-sdk-core is the engine behind fivetran debug and fivetran run in the Fivetran Connector SDK. It runs your connector on your own machine, in the current Python process, and writes what the connector emits to a local DuckDB file (or, with fivetran run, to an Iceberg catalog), applying the rules Fivetran applies in production: types, renaming, primary keys, deletes, truncates, files, checkpoints and state.

You do not install or call it yourself. It is installed with fivetran-connector-sdk and has no command of its own.

Install

Installed automatically with fivetran-connector-sdk. To install it on its own:

pip install fivetran-connector-sdk-core

Requirements

• Python version ≥3.10 and ≤3.14
• Operating system:
  • Windows: 10 or later (64-bit only)
  • macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
  • Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)

Getting started

See Quickstart guide to get started, then run your connector locally:

fivetran debug

The data lands in files/warehouse.db in your project folder, which you can open with duckdb -ui files/warehouse.db.

Writing to an Iceberg catalog

fivetran run runs your connector like fivetran debug, with the same options and the same logs, and writes what it emits to an Apache Iceberg REST catalog instead of the local DuckDB file. It reads the catalog from the file given with --destination-file <path>; when that is not given, from the destination.json in your project folder, beside your connector.py. If neither exists, the command stops and says so. The file looks like this:

{
  "uri": "https://my-catalog.example.com/api/catalog",
  "warehouse": "my_warehouse",
  "namespace": "my_connector",
  "client_id": "${CATALOG_CLIENT_ID}",
  "client_secret": "${CATALOG_CLIENT_SECRET}",
  "header.X-Iceberg-Access-Delegation": "vended-credentials",
  "s3.region": "us-east-1"
}

The names are the ones Iceberg clients (PyIceberg, Spark) give the properties of a REST catalog and of its S3 storage; namespace is the only setting of our own. A catalog that does not vend credentials is given the storage instead:

{
  "uri": "http://localhost:8181/api/catalog",
  "warehouse": "local_catalog",
  "namespace": "tester",
  "client_id": "${CATALOG_CLIENT_ID}",
  "client_secret": "${CATALOG_CLIENT_SECRET}",
  "s3.region": "us-east-1",
  "s3.endpoint": "http://localhost:8333",
  "s3.force-virtual-addressing": false,
  "s3.access-key-id": "${S3_ACCESS_KEY}",
  "s3.secret-access-key": "${S3_SECRET_KEY}"
}

A value may read an environment variable: "${NAME}", in the whole value or in a part of it. A credential written in the file can be committed with it, so fivetran run warns about one unless .gitignore lists the file.

What to know:

• Same as a local run. Tables, types, primary keys, schema changes, deletes and truncates are what fivetran debug does. Each checkpoint is one commit, and the state is saved to files/state.json after it, so run from the same project folder. fivetran reset deletes that folder's files/, not the tables in the catalog.
• Primary keys are recorded in the table: the key columns are required and named as its identifier fields. A key that changes rewrites the table if its rows fit the new key, and the sync fails if they do not.
• Types. Iceberg has no SMALLINT and no JSON: they are stored as integer and string. An integer becomes a bigint, a float a double and a decimal one of more digits in place; any other change of a column's type rewrites the table.
• Naming. The naming strategy of the first run is kept as the property fivetran.naming_strategy of the namespace, so a run from another machine uses the same one.
• A schema change is a commit of its own. A sync that fails part way can leave a table or a column it had already added, never rows.
• Not supported yet: file uploads (op.upsert(..., file=...)); with source naming, table and namespace names that have characters other than ASCII (DuckDB's iceberg extension cannot write them: use Fivetran naming); compaction, so data files and delete files pile up (use your catalog's own maintenance); several syncs writing one table at once.
• Catalogs. It is tested against Apache Polaris (locally, and with vended credentials) and Project Nessie; test-resources/iceberg in the package's source has a stack of each (./setup.sh starts one and writes its destination.json). A catalog that cannot commit several tables at once writes a checkpoint table by table. DuckDB downloads its iceberg extension the first time it is used.

How it fits with the SDK

The core knows nothing of the SDK: the SDK hands it a Source that declares tables and emits operations, and the core gives that source a Sink to emit into. A change that breaks this contract is a new major version of the core, and the SDK asks for the core as >=0.1,<1. fivetran configuration is in the SDK, not here.

License

The core is licensed under the Fivetran Connector SDK Core License Agreement. The core is not open source: its source is proprietary to Fivetran.

Maintenance

This package is actively maintained by Fivetran Developers. Please reach out to our Support team for any inquiries.