prizm-dbt

Prizm dbt artifact utility CLI for collecting dbt artifacts, packaging them into a compressed artifact bundle, and pushing it to Prizm.

Location: prizm-cli/prizm-dbt-cli (under Server root)

Key Design Principles:

• ✅ Does NOT execute dbt
• ✅ Operates purely on filesystem artifacts and sends gzip-compressed artifact bundles
• ✅ Safe to use in CI/CD, Airflow, GitHub Actions, Jenkins, etc.
• ✅ Explicit flags, no magic, deterministic behavior

Installation

From Wheel File

pip install dist/prizm_dbt_cli-*.whl

The wheel is self-contained and has no dependency on prizm-dbt; only click, requests, and PyYAML are required.

From Source (Development)

pip install -e .

For development, you can install the package in editable mode. See the Development section for more details.

Quick Start

1. Set Environment Variables

Set your Prizm API credentials (see Configuration section for detailed setup instructions):

export PRIZM_API_TOKEN=prizm_xxx
export PRIZM_API_ENDPOINT=https://api.prizm.ai

2. Verify Configuration (Optional)

Check that your environment is configured correctly:

prizm-dbt doctor

3. Run dbt (using your existing orchestration)

dbt run --target prod

4. Push Artifacts to Prizm

prizm-dbt push-artifacts \
  --prizm-connection "My Prizm Source" \
  --project-dir . \
  --target-path target \
  --env prod \
  --dbt-target prod \
  --adapter snowflake

Commands

prizm-dbt push-artifacts

Collect dbt artifacts, build a gzip-compressed artifact bundle, and push it to Prizm.

Required Flags:

• --prizm-connection: Prizm connection/source name (must match an existing Source in Prizm)
• --project-dir: dbt project root (where dbt_project.yml exists)
• --target-path: Directory containing dbt artifacts
• --env: Logical environment (dev, staging, prod)
• --dbt-target: dbt target name used during execution
• --adapter: Warehouse adapter (snowflake, bigquery, databricks, etc.)

Optional Flags:

• --execution-status: success | failed | partial (default: success)
• --invocation-id: CI/CD run ID (GitHub Actions, Airflow DAG run, etc.)
• --timeout: API response timeout in seconds (default: 1800 / 30 minutes)
• --retries: Retry attempts for transient upload failures (default: 3)
• --dry-run: Validate artifacts without pushing

Example:

prizm-dbt push-artifacts \
  --prizm-connection "My Prizm Source" \
  --project-dir /workspace/dbt \
  --target-path /workspace/dbt/target \
  --env production \
  --dbt-target prod \
  --adapter snowflake \
  --execution-status success \
  --invocation-id "$GITHUB_RUN_ID"

Artifacts Collected:

• manifest.json (required when --artifacts is omitted)
• run_results.json (optional)
• semantic_manifest.json (optional)
• catalog.json (optional)

Selective push (--artifacts): omit for the normal full discovery flow. Pass a comma-separated list to package only those files, e.g. --artifacts run_results,catalog or --artifacts manifest,run_results. Names may be bare (run_results) or filenames (run_results.json).

prizm-dbt push-artifacts \
  --prizm-connection "My Warehouse" \
  --project-dir . \
  --target-path target \
  --env prod \
  --dbt-target prod \
  --adapter snowflake \
  --artifacts run_results,catalog

The CLI sends these artifacts as a gzip-compressed tar bundle, not as multipart file uploads. The bundle contains metadata.json plus the original dbt artifact JSON files under artifacts/, with per-file SHA-256 checksums and a whole-bundle checksum header. Dry runs show artifact names, paths, sizes, checksums, and compressed bundle size without printing artifact contents.

For large dbt projects, schedule-side parsing and persistence can take several minutes after the upload finishes. The default response timeout is 30 minutes. If the CLI still reports a response timeout, check schedule logs for the printed upload_id before retrying, or rerun with a larger --timeout.

prizm-dbt validate

Preflight validation to ensure the environment is correctly configured.

Required Flags:

• --project-dir: dbt project root
• --target-path: Directory containing dbt artifacts

Example:

prizm-dbt validate \
  --project-dir . \
  --target-path target

Validates:

• ✔ Prizm auth token
• ✔ Endpoint connectivity
• ✔ Read permissions
• ✔ Artifact presence
• ✔ Project structure

prizm-dbt doctor

Diagnostics and troubleshooting for enterprise support and debugging.

No flags required - reads config + environment

Example:

prizm-dbt doctor

Outputs:

• Resolved project_dir and target_path
• Detected dbt artifacts and their sizes
• Environment context (env, target, adapter)
• Connectivity check to Prizm
• Result of the most recent artifact push (if available)

CI/CD Integration

GitHub Actions

- name: Run dbt
  run: dbt run --target prod

- name: Push dbt metadata to Prizm
  env:
    PRIZM_API_TOKEN: ${{ secrets.PRIZM_API_TOKEN }}
    PRIZM_API_ENDPOINT: ${{ secrets.PRIZM_API_ENDPOINT }}
  run: |
    prizm-dbt push-artifacts \
      --prizm-connection "My Prizm Source" \
      --project-dir . \
      --target-path target \
      --env prod \
      --dbt-target prod \
      --adapter snowflake \
      --invocation-id "${{ github.run_id }}"

Airflow

from airflow.operators.bash import BashOperator

push_prizm_metadata = BashOperator(
    task_id="push_prizm_metadata",
    bash_command="""
    prizm-dbt push-artifacts \
      --prizm-connection "My Prizm Source" \
      --project-dir /usr/local/airflow/dbt \
      --target-path /usr/local/airflow/dbt/target \
      --env prod \
      --dbt-target prod \
      --adapter snowflake
    """,
    env={
        "PRIZM_API_TOKEN": "{{ var.value.PRIZM_API_TOKEN }}",
        "PRIZM_API_ENDPOINT": "{{ var.value.PRIZM_API_ENDPOINT }}",
    },
)

Jenkins

stage('Push to Prizm') {
    steps {
        sh '''
            export PRIZM_API_TOKEN="${PRIZM_API_TOKEN}"
            export PRIZM_API_ENDPOINT="${PRIZM_API_ENDPOINT}"
            prizm-dbt push-artifacts \
              --prizm-connection "My Prizm Source" \
              --project-dir . \
              --target-path target \
              --env prod \
              --dbt-target prod \
              --adapter snowflake \
              --invocation-id "${BUILD_NUMBER}"
        '''
    }
}

Development

Development Setup

For local development and testing:

# Install the CLI package in editable mode
pip install -e .

# Install connector for testing (optional, only needed for running tests)
# The connector is built from prizm-dbt-mcp/dbt, not prizm-common
make install-connector

# Install test dependencies
pip install -e ".[test]"

Connector source: The dbt connector used by this CLI is built from prizm-dbt-mcp/dbt. To build the connector wheel separately: cd prizm-dbt-mcp/dbt && python -m build --wheel -o ../dist/

Running Tests

# Run tests (requires connector to be installed)
make test

# Run tests with coverage
make test-cov

Note: The install-connector step installs the connector from prizm-dbt-mcp/dbt and is only needed for development/testing when running tests. For production builds, the connector is automatically bundled from prizm-dbt-mcp/dbt into the wheel during make build.

Development Workflow

1. Make changes to CLI code or connector library
2. Run tests using make test (requires install-connector for now)
3. Build wheel using make build (automatically bundles connector)
4. Test wheel by installing it in a clean environment

Building Wheels

The CLI package uses a bundling approach where the dbt connector from prizm-dbt-mcp/dbt is automatically included in the wheel during build. This creates a single, self-contained wheel file that includes everything needed. The connector is not sourced from prizm-common.

Build Wheel Package

From the Server root:

cd prizm-cli/prizm-dbt-cli
make build

This command builds a wheel file containing the CLI and its dependencies (click, requests, PyYAML) and outputs it to the dist/ directory. The CLI is standalone and does not require the prizm-dbt package.

Clean Build Artifacts

make clean

This removes build and dist directories, egg-info, and Python cache files.

Configuration

The prizm-dbt CLI requires authentication credentials to communicate with the Prizm API. These are configured via environment variables.

Environment Variables

Variable	Description	Required	Default
PRIZM_API_TOKEN	Authentication token for Prizm API	Yes	-
PRIZM_API_ENDPOINT	Prizm API endpoint URL	No	https://api.prizm.ai

Setting Environment Variables

There are several ways to set these environment variables depending on your use case:

Method 1: Temporary (Current Shell Session)

Linux/macOS (bash/zsh):

export PRIZM_API_TOKEN=prizm_xxx
export PRIZM_API_ENDPOINT=https://api.prizm.ai

Windows (PowerShell):

$env:PRIZM_API_TOKEN="prizm_xxx"
$env:PRIZM_API_ENDPOINT="https://api.prizm.ai"

Windows (CMD):

set PRIZM_API_TOKEN=prizm_xxx
set PRIZM_API_ENDPOINT=https://api.prizm.ai

These settings only last for the current terminal session and are lost when you close the terminal.

Method 2: Permanent Setup

Linux/macOS - Add to Shell Profile:

For bash (~/.bashrc or ~/.bash_profile):

echo 'export PRIZM_API_TOKEN=prizm_xxx' >> ~/.bashrc
echo 'export PRIZM_API_ENDPOINT=https://api.prizm.ai' >> ~/.bashrc
source ~/.bashrc

For zsh (~/.zshrc):

echo 'export PRIZM_API_TOKEN=prizm_xxx' >> ~/.zshrc
echo 'export PRIZM_API_ENDPOINT=https://api.prizm.ai' >> ~/.zshrc
source ~/.zshrc

For all shells (~/.profile):

echo 'export PRIZM_API_TOKEN=prizm_xxx' >> ~/.profile
echo 'export PRIZM_API_ENDPOINT=https://api.prizm.ai' >> ~/.profile
source ~/.profile

Windows - System Environment Variables:

1. Open Settings → System → About → Advanced system settings
2. Click "Environment Variables"
3. Under "User variables" or "System variables", click "New"
4. Add PRIZM_API_TOKEN with your token value
5. Add PRIZM_API_ENDPOINT with your endpoint URL (optional, defaults to https://api.prizm.ai)
6. Restart your terminal/command prompt

Method 3: Using .env Files (Development)

For local development, you can use a .env file in your project directory:

Create .env file:

# .env
PRIZM_API_TOKEN=prizm_xxx
PRIZM_API_ENDPOINT=https://api.prizm.ai

Load before running commands:

Linux/macOS:

export $(cat .env | xargs)
prizm-dbt push-artifacts ...

Or use tools like direnv or python-dotenv to automatically load .env files.

Important: Never commit .env files to version control. Add .env to your .gitignore.

Verifying Configuration

After setting environment variables, verify your configuration:

prizm-dbt doctor

This command will show:

• Whether PRIZM_API_TOKEN is set
• The configured PRIZM_API_ENDPOINT
• Connectivity status to Prizm

Security Best Practices

• Never commit tokens to version control: Always use .gitignore for .env files and never hardcode tokens in scripts
• Use secrets management in CI/CD: Store tokens as encrypted secrets in your CI/CD platform (GitHub Secrets, GitLab Variables, etc.)
• Rotate tokens regularly: Update your API tokens periodically for better security
• Use different tokens per environment: Use separate tokens for dev, staging, and production environments
• Limit token permissions: Create tokens with only the minimum required permissions
• Monitor token usage: Regularly review token access logs in your Prizm dashboard

Supported Adapters

• snowflake
• bigquery
• databricks
• postgres
• redshift
• spark
• athena
• trino
• duckdb

Error Handling

The CLI provides clear, actionable error messages:

• Missing artifacts: Lists checked paths and next steps
• Authentication failures: Clear token validation errors
• Network errors: Connection and timeout handling
• Permission errors: File access validation

Security

• Token-based authentication via environment variables
• Tokens never logged or printed
• TLS enforced for all API calls
• No secrets in CLI arguments

License

MIT License

Support

For issues and questions, please visit: https://github.com/DQLabs-Inc/prizm-dbt-mcp/issues