datacontract-x 0.1.8

Data Contract eXtended — AI-native, platform-extensible data contracts

Author data contracts with an LLM, sync them with your live platforms.
A lean, no-fork extension of datacontract-cli, built on the Open Data Contract Standard (ODCS).

What is dcx?

dcx (Data Contract eXtended) adds three things to the Open Data Contract Standard workflow that plain datacontract-cli doesn't do:

1. AI authoring — use an LLM to enrich a contract with column descriptions, validation constraints, governance tags from your own catalog, and an executable data-quality suite.
2. Live import — build a contract from a running system (its real columns, keys, comments, tags).
3. Apply — push the contract's governance back to the platform (comments, tags, data-quality, and the table itself).

The pipeline:

import  ──→  enrich  ──┬──→  apply     push governance back to the platform
                       └──→  export    to SQL / docs / schemas

  import   a live schema into an ODCS contract
  enrich   columns · tags · quality

Everything is available both as a CLI and as a REST API (dcx api).

It's platform-extensible by design — each platform is a small importer / exporter / apply module that plugs into datacontract-cli's factories. Snowflake is the first end-to-end platform (import → enrich → apply), with Kafka import today and more platforms built to slot in the same way.

Why dcx?

• 🧠 AI authoring that's safe to ship. Forced tool-calling, temperature=0, and strict server-side validation against the ODCS schema — the model can only produce spec-valid output, never free-form guesses.
• 🏷️ A tag manager, not a tag guesser. You define a controlled tag catalog (names, allowed values, examples); the LLM classifies columns into your vocabulary, with optional defaults.
• ✅ Executable, portable data quality. Quality rules prefer ODCS library metrics (portable, mappable to platform-native checks) and fall back to portable sql checks — across all seven ODCS dimensions.
• 🔌 Any LLM provider. Powered by litellm — Anthropic, OpenAI, Azure, Bedrock, Gemini, Ollama, … behind one --model flag.
• 🧩 Pluggable platforms, no fork. You keep all 30+ upstream importers/exporters and lint / test / changelog, and gain the AI + platform layer on top.
• 🔐 Auth that makes sense per surface. Live platform operations over the API use caller-supplied credentials — OAuth, key-pair, or password, never the server's; on the CLI, secrets are never flags.

Install

pip install datacontract-x

The import package and CLI are both dcx:

dcx --help
dcx info

From source (for development):

git clone https://github.com/MickaelBZH/data-contract-x.git
cd data-contract-x
pip install -e ".[dev]"

Requires Python 3.10–3.12. Installing pulls in datacontract-cli, litellm, FastAPI, and the platform connectors automatically.

Quickstart

The full loop — import a live schema, enrich it with an LLM, sync it back. Snowflake here is the example platform.

# 1. Import an existing schema into a contract (real columns, PKs, comments, tags)
dcx import snowflake --database MY_DB --schema LOAD --authenticator externalbrowser --output contract.yaml

# 2. Enrich with an LLM: descriptions + constraints + tags + data-quality tests
export ANTHROPIC_API_KEY=...           # or OPENAI_API_KEY / AZURE_API_KEY / ...
dcx enrich all contract.yaml --catalog tags_catalog.yaml --output contract.enriched.yaml

# 3. Preview exactly what will run — no connection needed
dcx apply snowflake contract.enriched.yaml --include-quality --dry-run

# 4. Apply it: creates the table if missing, governs it (comments + tags + DQ) if it exists
dcx apply snowflake contract.enriched.yaml --include-quality

Commands

Every command is dcx <command>, and most are mirrored to a REST endpoint when you run dcx api. Each section below lists the sub-commands, a CLI example, and the matching API call. Run dcx <command> --help for the full option list.

import — build a contract from a source

dcx import snowflake --database MY_DB --schema LOAD --authenticator externalbrowser --output contract.yaml
dcx import snowflake --database MY_DB --schema LOAD --quality --output contract.yaml   # + attached DMFs
dcx import kafka --schema-registry https://sr:8081 --topic orders --output contract.yaml
dcx import sql --source schema.sql --dialect snowflake --output contract.yaml

API

• POST /import/snowflake — live import, authenticated by the caller's own credentials: an auth block (oauth · key_pair · password) or an Authorization: Bearer token. See Connecting to Snowflake.
• POST /import/{format} — file-based importers; send the document inline as source_content.
• (Kafka import is CLI-only.)

enrich — AI authoring with an LLM

Each sub-command is independent and idempotent — existing values are preserved unless you pass --overwrite.

The provider key is read from the environment — there is no --api-key flag.

dcx enrich columns contract.yaml --output contract.enriched.yaml
dcx enrich tags    contract.yaml --catalog tags_catalog.yaml --output contract.tagged.yaml
dcx enrich quality contract.yaml --model gpt-4o --output contract.dq.yaml
dcx enrich all     contract.yaml --catalog tags_catalog.yaml --output contract.full.yaml

API (the LLM key comes from the server's environment)

• POST /enrich/columns · POST /enrich/quality
• POST /enrich/tags · POST /enrich/all — take the tag catalog inline in the request body.

export — convert a contract to a target format

snowflake-full

Emits the exact script apply --dry-run would, and shares its SQL-generation knobs:

--ddl-mode · --structured-types · --comments · --include-tags · --include-quality · --create-tags · --tag-namespace · --tag-namespace-filter

See the apply option table below for what each does. Only --strict has no export equivalent — drift detection needs a live connection.

dbt

Unifies upstream's dbt-models / dbt-sources / dbt-staging-sql under one command via --kind (those upstream commands remain available, unchanged), and maps ODCS governance the idiomatic dbt way:

dcx export snowflake-full contract.yaml --include-quality --create-tags --output setup.sql
dcx export snowflake-full contract.yaml --ddl-mode never --output govern.sql   # alter-only
dcx export dbt contract.yaml --kind models --server production --output schema.yml
dcx export html contract.yaml --output contract.html

API

• POST /export/{format} — including POST /export/snowflake-full and POST /export/dbt ({options: {kind: "models"}}). The response media type depends on the format (JSON / YAML / text / binary).

apply — push governance to a live platform

With the default --ddl-mode auto you don't need to know whether the table exists:

• missing → created with CREATE TABLE IF NOT EXISTS
• existing → governed: column/table comments, tags, and (with --include-quality) data-quality metrics

For existing tables dcx also compares the live schema to the contract and reports drift as warnings — or, with --strict, an error that aborts before any change. The check uses DESCRIBE TABLE, so it needs no active warehouse.

dcx apply snowflake contract.yaml --dry-run            # preview
dcx apply snowflake contract.yaml --include-quality    # create-or-govern

Views

Objects with physicalType: view are governed as views — tags, comments and DQ use ALTER VIEW / COMMENT ON VIEW. This holds for both apply snowflake and export snowflake-full.

Column comments are the catch. Snowflake persists them only inside the CREATE VIEW column list — there is no ALTER path (Snowsight uses the same trick). So dcx has to recreate the view, which needs the viewDefinition captured on import:

To update an existing view's column comments, use --ddl-mode always. Every other combination leaves them as they are, and dcx notes each skip.

Materialized and external tables are imported with their real physicalType, but are currently governed as tables.

API

• POST /apply/snowflake — authenticated by the caller's own credentials (see Connecting to Snowflake; dry_run needs none). Supports dry_run, ddl_mode, strict, structured_types, tag_namespace_filter, … (all under options) and returns the executed SQL plus any drift warnings.

target — bind a contract to a platform

dcx target <type> does two things: sets the contract's server block, and resolves each column's physicalType for that platform.

~30 types — snowflake, bigquery, databricks, postgres, redshift, mysql, sqlserver, oracle, s3, kafka, trino, athena, glue, duckdb, local, …

dcx target snowflake contract.yaml --output contract.snowflake.yaml

API

• POST /target/{type} — one route per supported platform type.

From datacontract-cli

These commands work unchanged — dcx <command> behaves exactly like datacontract <command>.

api / info

dcx api --port 4242      # start the REST server (Swagger UI at /docs)
dcx info                 # show dcx + datacontract-cli versions   (API: GET /info)

Connecting to Snowflake

Two rules cover everything below:

1. Credentials differ per surface. The CLI takes secrets from the environment or your Snowflake connection profile — never from a flag. The API takes them from the request, so the server acts as the caller rather than with one shared identity.
2. Where the objects land never depends on the credentials. Every generated statement is qualified DATABASE.SCHEMA.OBJECT from the contract's server block. A connection profile, an env var, or an OAuth token decides who you are, never what you touch.

To apply the same contract to a different database, name a different server block — --server dev on the CLI, "server_name": "dev" in the API — or edit the contract. There is deliberately no --database / --schema on apply: those could not retarget anything, and only pointed the drift check at a different database than the one being written.

CLI

Non-secret connection context resolves CLI flag → env var → contract server block:

Secrets are environment-only — there is no --password flag and there never will be (shell history, ps aux, CI logs):

--authenticator selects the method: snowflake (password, the default), externalbrowser (SSO), oauth, snowflake_jwt (key-pair). The connector auto-detects when you omit it.

--secondary-roles accepts ALL or NONE alone, or one or more comma-separated Snowflake role names (for example, DATA_READER,DATA_STEWARD). It resolves as CLI flag → SNOWFLAKE_SECONDARY_ROLES → unset. ALL and NONE are case-insensitive; named roles must use valid Snowflake identifier syntax. A role name with spaces must preserve its Snowflake identifier quotes in the shell, for example --secondary-roles '"Finance Reader"'. When set, dcx executes USE SECONDARY ROLES ... immediately after connecting, before metadata reads, drift checks, or DDL. When omitted, dcx does not execute a USE SECONDARY ROLES statement. This is session configuration only: it is not stored in the Data Contract server block, and dcx never selects, switches, or retries roles. Snowflake determines whether named roles are granted to the user or permitted by a session policy. This setting applies to dcx's Snowflake import and apply commands; upstream dcx test uses datacontract-cli's connection path and does not honor SNOWFLAKE_SECONDARY_ROLES.

export SNOWFLAKE_ACCOUNT=xy12345.eu-central-1 SNOWFLAKE_USER=me SNOWFLAKE_PASSWORD=...
dcx import snowflake --database MY_DB --schema LOAD --output contract.yaml

dcx apply snowflake contract.yaml --authenticator externalbrowser --role TRANSFORMER --secondary-roles NONE

Connection profiles (config.toml)

A ready-to-edit template with all four methods lives in examples/snowflake_config.example.toml.

--connection-name uses a named profile from the connector's own config file — dcx defines no config format of its own and never parses these files; snowflake-connector-python resolves them:

# config.toml — see "Where the file lives" below
[connections.dev]
account = "xy12345.eu-central-1"
user = "SVC_DCX"
authenticator = "SNOWFLAKE_JWT"
private_key_file = "/home/me/.snowflake/rsa_key.p8"   # absolute: `~` is NOT expanded
role = "TRANSFORMER"
warehouse = "DEV_WH"

dcx import snowflake --connection-name dev --database MY_DB --schema LOAD
dcx apply snowflake contract.yaml --connection-name dev --server dev

The profile supplies the whole connection; only --user, --role, --warehouse, --account and --authenticator layer on top of it — plus --database / --schema on import, which name what to read. Env vars are not consulted at all on this path.

Profiles may equally live in connections.toml alongside it (same tables, without the connections. prefix) — the connector reads both.

Where the file lives

The connector resolves the directory, in this order:

1. $SNOWFLAKE_HOME if that directory exists — defaults to ~/.snowflake/.
2. Otherwise the platform config dir: ~/.config/snowflake/ on Linux, ~/Library/Application Support/snowflake/ on macOS, %LOCALAPPDATA%\snowflake\ on Windows.

Note the first rule tests for existence: with no ~/.snowflake/ directory, ~/.config/snowflake/config.toml is the file in play. To check which one your install uses:

python -c "from snowflake.connector.constants import CONFIG_FILE; print(CONFIG_FILE)"

If you set nothing at all, your default profile is used. When no flag, no SNOWFLAKE_* variable and no contract server block identify a connection, dcx falls back to the profile named by default_connection_name — the same one connect() uses with no arguments — instead of erroring:

# config.toml
default_connection_name = "dev"

dcx apply snowflake contract.yaml        # no credentials anywhere: uses [connections.dev]

The contract's account is still layered on top, so a default profile can never silently move an apply to another account. If no default profile is configured, you get the usual Cannot determine Snowflake account/user error.

Snowflake requires the file to be private: chmod 0600 it, or the connector warns Bad owner or permissions on every connect.

private_key_file must be an absolute path. The connector opens it with a bare open() and does not expand ~, so a tilde path fails at connect time with No such file or directory: '~/...'.

SnowSQL's ~/.snowsql/config is a different file in a different format and is not read. If your profiles live there, port them to config.toml (snow connection add) or use the env vars above.

A profile authenticates; the contract targets. A dev profile applied to a contract whose server block names PROD_DB will authenticate as dev and still write to PROD_DB. Pair a profile with the matching --server block.

API

The live endpoints (POST /import/snowflake, POST /apply/snowflake) take an auth object in the request body — the server holds no credentials of its own:

POST /import/snowflake
{
  "account": "xy12345.eu-central-1", "database": "MY_DB", "schema": "LOAD",
  "secondary_roles": "NONE",
  "auth": {
    "type": "key_pair",
    "user": "SVC_DCX",
    "private_key": "-----BEGIN ENCRYPTED PRIVATE KEY-----\n...",
    "private_key_passphrase": "..."
  }
}

With config the server's profile carries the account, user and credentials, so the body needs only what to read:

POST /import/snowflake
{
  "database": "MY_DB", "schema": "LOAD",
  "auth": {"type": "config", "connection_name": "dev"}
}

Omit connection_name to use the server's default_connection_name; if it has none configured, that is a 400.

private_key is PEM text or base64-encoded PKCS#8 DER. There is no private_key_file: a path would make the server read its own filesystem on a caller's behalf.

Authorization: Bearer <token> is shorthand for {"type": "oauth"} and still works on its own; when both are sent, the body wins. dry_run on /apply/snowflake needs no credentials at all.

Both endpoints accept optional secondary_roles: use it at the top level for /import/snowflake, or in the options object for /apply/snowflake. Values may be "ALL", "NONE", a role name, or a comma-separated list of role names. API requests use only this explicit request value; they never inherit SNOWFLAKE_SECONDARY_ROLES from the API host.

Errors: 401 no credentials · 400 unusable key material · 403 method disabled on this server · 502 Snowflake refused.

Server-side profiles — dcx api --allow-local-credentials

auth.type: config is the one method that reads the API host's own connection config instead of the caller's credentials, so it is disabled unless you start the server with it:

dcx api --allow-local-credentials                              # only when the server is yours alone
dcx api --allow-local-credentials --snowflake-config /etc/dcx  # profiles from a specific location

--snowflake-config takes the directory holding config.toml / connections.toml (or the file itself) and points the connector at it via SNOWFLAKE_HOME, so a service account's profiles need not live in the server user's home. It is validated at startup — a missing path, an unreadable filename, or a directory with no config in it fails immediately rather than silently falling back to the default location. It requires --allow-local-credentials, since without that the server never reads a Snowflake config at all.

Whenever server-side profiles are enabled, startup reports which config is in play and what it contains (names only, never values):

Snowflake config: /etc/dcx/config.toml (4 profiles: dev, sso, pw, oauth; default: dev)
Server-side profiles enabled: any caller may authenticate as this host via `auth.type: config`.

A config that cannot be read, or that defines no profiles, is reported as a warning at startup rather than surfacing later as a failed request.

dcx api has no authentication of its own. On a shared instance a server-side profile would let anyone who can reach the port connect as whoever runs the server — so it stays off by default, and requests using it get a 403. On a personal localhost server it is the safest option available, since no secret crosses the wire. Only a profile name is accepted, never a path.

The tag catalog

dcx enrich tags does controlled-vocabulary tagging. Instead of letting the model invent tags, you give it a catalog of allowed names and values, and it classifies each column into that vocabulary.

The catalog is a small YAML (or JSON) file — the only extra input auto-tagging needs.

# tags_catalog.yaml
tags:
  - name: DATA_CLASSIFICATION          # the tag name (becomes the platform TAG name)
    description: >                      # tells the model what this tag is for
      Data sensitivity level. Assign exactly one — the highest level that applies.
    multiple: false                    # false = at most one value per column; true = many
    values:
      - value: PUBLIC                   # the model may only pick from these values
        description: Non-sensitive data that can be shared freely.
        examples: [country_code, currency, language, product_category]   # guide classification
      - value: INTERNAL
        description: Internal business data, not for public release. The default.
        default: true                  # assigned when the model picks nothing else
        examples: [order_id, status, created_at, loyalty_points]
      - value: CONFIDENTIAL
        description: Personal data or sensitive business data; need-to-know access.
        examples: [full_name, email, phone, home_address, date_of_birth]
      - value: RESTRICTED
        description: Highly sensitive data under legal/regulatory controls (financial, health, credentials, IDs).
        examples: [national_id, passport_number, iban, credit_card_number, health_status]

  - name: DATA_DOMAIN                   # you can define several tags
    description: The business domain that owns the column.
    multiple: false
    values:
      - value: CUSTOMER
        examples: [customer_id, email, loyalty_points]
      - value: FINANCE
        examples: [amount, currency, invoice_id, iban]

Assigned tags are written on each column as NAME=VALUE (e.g. DATA_CLASSIFICATION=CONFIDENTIAL) — the convention export snowflake-full and apply snowflake consume. A worked catalog and example contracts live in examples/.

REST API

dcx api --port 4242      # Swagger UI at http://127.0.0.1:4242/docs

Every command above is mirrored to an endpoint, with request and response schemas in the OpenAPI spec. Auth model:

• Live platform operations (/import/snowflake, /apply/snowflake) act as the caller — credentials ride on the request (an auth block or a bearer token), so the server never uses ambient credentials for someone else's data. The one exception, server-side connection profiles, is off unless you start the server with --allow-local-credentials. Details: Connecting to Snowflake.
• Enrichment (/enrich/*) uses the server's LLM key (from the environment). Put service-level auth/quota in front of it before exposing it publicly.
• The CLI never takes secrets as flags — platform secrets come from env vars or the connector's own config.toml; LLM keys from the provider's standard env var. (Over HTTP a caller does send credentials in the request body — that is the point: they are the caller's, not the server's. Terminate TLS in front of it.)

How it fits with datacontract-cli

dcx is a separate package that depends on datacontract-cli as a library — no fork. It plugs into upstream's own extension points:

So you keep all of upstream's importers, exporters, lint, test and changelog, and gain the AI + platform layer on top.

Development

pip install -e ".[dev]"
pytest          # 375 tests
ruff check dcx  # lint

Tests never hit live services or real LLMs — platform connections, the Schema Registry, and every LLM call are mocked, so the suite stays fast and offline. See RELEASING.md for the PyPI release process.

Contributing

Issues and PRs welcome. Please run pytest and ruff check dcx before opening a PR, and add tests for new behavior.

License

MIT © MickaelBZH.

_{Built on datacontract-cli · Open Data Contract Standard · litellm}