prizm-airflow

Prizm Airflow metadata extraction CLI for reading Airflow REST API metadata, packaging it into a JSON bundle, and pushing it to Prizm.

Location: prizm-cli/prizm-airflow-cli (under Server root)

Key Design Principles:

• ✅ Does NOT execute Airflow DAGs or tasks
• ✅ Does NOT process metadata locally — reads Airflow's REST API and forwards the bundle as-is
• ✅ Safe to use in CI/CD, cron, Airflow itself, GitHub Actions, Jenkins, etc.
• ✅ Explicit flags, no magic, deterministic behavior

Prizm stages the uploaded bundle and runs TECHNICAL followed by OPERATIONAL processing server-side.

Installation

From PyPI

pip install prizm-airflow-cli

From Wheel File

pip install dist/prizm_airflow_cli-*.whl

The wheel is self-contained; only requests is required at runtime.

From Source (Development)

pip install -e .

For development, you can install the package in editable mode. See the Development section for more details.

Quick Start

1. Set Environment Variables (Optional)

You can pass credentials as flags, or export them and reference via your shell:

export PRIZM_API_TOKEN=prizm_xxx

2. Extract and Push Airflow Metadata to Prizm

prizm-airflow extract \
  --prizm-url https://<prizm-host> \
  --api-token "$PRIZM_API_TOKEN" \
  --connection-name "airflow_cli" \
  --hostname http://localhost:8080 \
  --username admin \
  --password admin \
  --runs 10 \
  --extract-source-code true

Commands

prizm-airflow extract

Read DAGs, runs, tasks, and task instances from the Airflow REST API, build a JSON bundle, and upload it to Prizm.

Required Flags:

• --prizm-url: Prizm base URL
• --connection-name: Prizm connection/source name (must match an existing Source in Prizm)

Authentication Flags (one of):

• --api-token: Prizm API access token
• --client-secret: Alias for --api-token

Airflow Connection Flags:

• --hostname: Airflow base URL (default: http://localhost:8080)
• --username / --password: Airflow basic-auth credentials
• --basic-auth: Base64-encoded username:password, alternative to --username/--password

Optional Flags:

• --endpoint-url: Full ingest URL; overrides --prizm-url + default path
• --runs: Number of recent DAG runs to fetch per DAG (default: 30)
• --extract-source-code: Fetch DAG file source (default: true)
• --include-full-log: Fetch and include the full task log for failed task instances (default: false)
• --dry-run: Build the bundle and write it to --out locally without uploading
• --out: Output path for --dry-run (default: airflow_bundle.json)

Example:

prizm-airflow extract \
  --prizm-url https://api.prizm.ai \
  --api-token "$PRIZM_API_TOKEN" \
  --connection-name "My Airflow Source" \
  --hostname https://airflow.internal.example.com \
  --basic-auth "$AIRFLOW_BASIC_AUTH_B64" \
  --runs 50 \
  --extract-source-code true \
  --include-full-log false

Bundle Contents:

• DAGs (id, metadata)
• DAG runs (up to --runs most recent per DAG)
• Tasks per DAG
• Task instances per DAG run
• DAG source code (when --extract-source-code true)
• Full failed-task logs (when --include-full-log true)

Dry run (--dry-run): builds the bundle and writes it to --out instead of uploading — useful for inspecting bundle contents or debugging Airflow connectivity before sending anything to Prizm.

prizm-airflow extract \
  --prizm-url https://api.prizm.ai \
  --connection-name "My Airflow Source" \
  --hostname http://localhost:8080 \
  --username admin \
  --password admin \
  --dry-run \
  --out /tmp/airflow_bundle.json

The CLI authenticates to Prizm's ingest endpoint with Authorization: Bearer <token> and uploads the bundle as a multipart file (airflow_bundle.json) alongside the connection_name field.

CI/CD Integration

GitHub Actions

- name: Push Airflow metadata to Prizm
  env:
    PRIZM_API_TOKEN: ${{ secrets.PRIZM_API_TOKEN }}
  run: |
    prizm-airflow extract \
      --prizm-url https://api.prizm.ai \
      --api-token "$PRIZM_API_TOKEN" \
      --connection-name "My Airflow Source" \
      --hostname https://airflow.internal.example.com \
      --basic-auth "${{ secrets.AIRFLOW_BASIC_AUTH_B64 }}" \
      --runs 30

Airflow (self-monitoring)

from airflow.operators.bash import BashOperator

push_prizm_metadata = BashOperator(
    task_id="push_prizm_airflow_metadata",
    bash_command="""
    prizm-airflow extract \
      --prizm-url https://api.prizm.ai \
      --connection-name "My Airflow Source" \
      --hostname http://localhost:8080 \
      --username "{{ var.value.AIRFLOW_ADMIN_USER }}" \
      --password "{{ var.value.AIRFLOW_ADMIN_PASSWORD }}" \
      --runs 30
    """,
    env={
        "PRIZM_API_TOKEN": "{{ var.value.PRIZM_API_TOKEN }}",
    },
)

Jenkins

stage('Push Airflow metadata to Prizm') {
    steps {
        sh '''
            prizm-airflow extract \
              --prizm-url https://api.prizm.ai \
              --api-token "${PRIZM_API_TOKEN}" \
              --connection-name "My Airflow Source" \
              --hostname https://airflow.internal.example.com \
              --basic-auth "${AIRFLOW_BASIC_AUTH_B64}" \
              --runs 30
        '''
    }
}

Development

Development Setup

pip install -e ".[dev,test]"

Running Tests

make test

Build Wheel

make build

For offline/local environments where build dependencies are already installed:

make build-local

This outputs a wheel and sdist to dist/.

Clean Build Artifacts

make clean

This removes build/ and dist/ directories, egg-info, and Python cache files.

Publish

Build both source and wheel distributions:

python -m build --sdist --wheel

Upload to TestPyPI first:

python -m twine upload --repository testpypi dist/*

Upload to PyPI:

python -m twine upload dist/*

Security

• Token-based authentication via --api-token/--client-secret or environment variable
• Tokens never logged or printed
• TLS enforced for all API calls
• No secrets written to the dry-run output bundle

License

MIT License

Support

For issues and questions, please visit: https://github.com/DQLabs-Inc/prizm-cli/issues