explainio-airflow-agent 0.2.4

Explain.io FinOps Airflow Cost Agent

💸 Explain.io — Airflow Cost Agent

Explain.io is a FinOps tool for Data Engineers. This lightweight Airflow plugin automatically attributes Google Cloud BigQuery compute costs to the exact DAG and Task that triggered them.

No more guessing which pipeline caused the bill spike. No code changes required.

✨ Features

• Zero Code Changes — Uses Airflow's Listener interface to automatically intercept all BigQuery task completions. You don't need to modify any DAG or add any callbacks.
• Zero Blast Radius — All network requests run asynchronously in a separate thread with strict timeouts. If the Explain.io API is unreachable, your DAG will still succeed. Exceptions are silently caught and logged.
• Instant Dashboard — View your pipeline costs, projected monthly spend, and heaviest DAGs at explain.io.

🔍 How It Works

┌──────────────┐     ┌─────────────────────┐     ┌──────────────────┐
│  Airflow      │     │  Explain.io Agent    │     │  Explain.io API  │
│  (Scheduler)  │────▶│  (Listener Plugin)   │────▶│  (Backend)       │
│               │     │                     │     │                  │
│  Task succeeds│     │  1. Intercepts event │     │  Fetches BQ cost │
│               │     │  2. Extracts job ID  │     │  from GCP API    │
│               │     │  3. POSTs to API     │     │  and stores it   │
└──────────────┘     └─────────────────────┘     └──────────────────┘

The agent registers as an Airflow Listener plugin. On every task success, it checks if the task was a BigQuery operator, extracts the job ID via XCom, and sends it to the Explain.io API — all in a background thread.

---

🚀 Installation

Quick Start (Any Environment)

pip install explainio-airflow-agent

Then configure your API Key in Airflow. You can use either Airflow Variables (recommended, easiest) or Environment Variables:

Variable	Required	Description
EXPLAIN_IO_API_KEY	✅	Your API key. Set via Airflow UI (Admin → Variables) OR Environment Variable.
EXPLAIN_IO_API_URL	❌	API endpoint. Defaults to https://api.explain.io/api/v1/ingest

Verify it's working:

airflow plugins

You should see cost_agent_plugin in the output.

---

📦 Installation by Platform

Self-Hosted: Docker (Recommended)

Don't use _PIP_ADDITIONAL_REQUIREMENTS — it reinstalls packages on every container start and is only meant for quick testing.

Extend the official Airflow image with a Dockerfile:

FROM apache/airflow:3.1.6

USER airflow
RUN pip install --no-cache-dir explainio-airflow-agent

Then in your docker-compose.yaml, replace image: with build::

x-airflow-common:
  &airflow-common
  build: ./airflow          # instead of: image: apache/airflow:3.1.6
  environment:
    EXPLAIN_IO_API_KEY: ${EXPLAIN_IO_API_KEY}
    EXPLAIN_IO_API_URL: ${EXPLAIN_IO_API_URL}
    # ... other env vars

Build and start:

docker compose build
docker compose up -d

Important: All Airflow components (scheduler, webserver, workers, triggerer) must use the same image. The shared x-airflow-common anchor ensures this automatically.

---

Self-Hosted: Bare Metal / virtualenv

Install directly into the Python environment where Airflow runs:

pip install explainio-airflow-agent

Set the environment variables:

export EXPLAIN_IO_API_KEY="your-api-key"
export EXPLAIN_IO_API_URL="https://api.explain.io/api/v1/ingest"  # optional

If you run a multi-node setup (e.g., separate scheduler and workers), the package must be installed on every node.

---

AWS MWAA (Managed Workflows for Apache Airflow)

1. Create or update your requirements.txt with the Airflow constraints file:

   --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.10.1/constraints-3.11.txt"
   explainio-airflow-agent==0.1.2
   

   Adjust the constraints URL to match your MWAA Airflow version and Python version. See MWAA docs.

2. Upload requirements.txt to your MWAA environment's S3 bucket.

3. In the AWS Console → MWAA → your environment → Edit → point to the new requirements.txt path → Save.

4. Set EXPLAIN_IO_API_KEY and (optionally) EXPLAIN_IO_API_URL under Environment variables → Airflow configuration options, using the prefix AIRFLOW__:

   • Key: EXPLAIN_IO_API_KEY, Value: your-api-key

5. Wait for the environment update to complete (can take 20–40 minutes).

Troubleshooting: Check CloudWatch Logs → Log group: airflow-{env-name}-requirements_install for pip errors.

---

Google Cloud Composer

Option A — Console UI

1. Go to Environments → click your environment → PyPI Packages tab.
2. Click Edit → Add Package.
3. Enter:
   • Package name: explainio-airflow-agent
   • Version: >=0.1.2
4. Save. Composer will restart workers and the scheduler.

Option B — gcloud CLI

gcloud composer environments update YOUR_ENV_NAME \
    --location YOUR_LOCATION \
    --update-pypi-package "explainio-airflow-agent>=0.1.2"

Environment variables

Set via the Console UI (Environment Variables tab) or CLI:

gcloud composer environments update YOUR_ENV_NAME \
    --location YOUR_LOCATION \
    --update-env-variables EXPLAIN_IO_API_KEY=your-api-key

---

Astronomer (Astro CLI)

1. Add the package to requirements.txt in your Astro project root:

   explainio-airflow-agent==0.1.2
   

2. Set environment variables in your .env file:

   EXPLAIN_IO_API_KEY=your-api-key
   

3. Restart:

   astro dev restart
   

4. Verify:

   astro dev bash --scheduler "airflow plugins"
   

For Astro Cloud deployments, add the environment variables via the Astro UI under Deployments → Environment Variables.

---

⚙️ Configuration Reference

Variable	Required	Default	Description
EXPLAIN_IO_API_KEY	✅	—	Your Explain.io API key. If not set, the plugin logs an info message and becomes a no-op.
EXPLAIN_IO_API_URL	❌	https://api.explain.io/api/v1/ingest	Override the API endpoint (e.g., for self-hosted backends).

---

🔧 Troubleshooting

Plugin not showing in airflow plugins

• Make sure the package is installed in the same Python environment as Airflow.
• Run pip show explainio-airflow-agent to confirm it's installed.
• Check Airflow scheduler logs for import errors.

Events not appearing in the dashboard

• Confirm EXPLAIN_IO_API_KEY is set. Without it, the agent silently disables itself.
• Check that EXPLAIN_IO_API_URL points to a reachable endpoint.
• Look for [ExplainIO] log lines in the Airflow task logs.

AWS MWAA: Package fails to install

• Ensure the --constraint line in requirements.txt matches your MWAA Airflow + Python version.
• Check the requirements_install log stream in CloudWatch.

---

Supported Workloads

Explain.io perfectly isolates and captures BigQuery costs for the following workloads:

1. Native Airflow Operators (e.g., BigQueryInsertJobOperator): Works instantly out-of-the-box.
2. dbt-core (via BashOperator, KubernetesPodOperator, or Cosmos): Requires a 1-line configuration in your profiles.yml (see below).
3. Custom Scripts (e.g., raw Python/Bash): Supported if you manually inject the airflow_dag_id and airflow_task_id labels into your BigQuery job configuration.

---

dbt Support (Required Configuration)

Because dbt runs externally and does not natively report BigQuery job IDs back to Airflow, Explain.io requires you to configure dbt to pass context labels to BigQuery to guarantee 100% accurate cost attribution.

1. Configure dbt Labels

To prevent cost collisions when multiple DAGs run concurrently, inject the Airflow context variables into your BigQuery jobs by adding this to your profiles.yml:

# profiles.yml
my_bigquery_profile:
  target: dev
  outputs:
    dev:
      type: bigquery
      # ... other config ...
      labels:
        airflow_dag_id: "{{ env_var('AIRFLOW_CTX_DAG_ID', '') }}"
        airflow_task_id: "{{ env_var('AIRFLOW_CTX_TASK_ID', '') }}"

Explain.io perfectly isolates the costs to the correct DAG using these labels. If you skip this step, Explain.io will not be able to track your dbt costs.

2. Enable Per-Model Attribution (Optional)

For per-dbt-model cost breakdowns, add job labeling to your dbt_project.yml:

# dbt_project.yml
query-comment:
  comment: "{{ query_comment(node) }}"
  job-label: true

This tells dbt to label every BigQuery job with the model name (node_id). Explain.io reads these labels to show you exactly which dbt models are costing the most.

GCP Permissions

Your dedicated Explain.io service account needs the following roles:

1. BigQuery Resource Viewer (roles/bigquery.resourceViewer): To read BigQuery job metadata and labels (bigquery.jobs.listAll).

Security Note: Explain.io only requires metadata permissions. We cannot read your actual table data.

Grant it via:

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:YOUR_SA@YOUR_PROJECT.iam.gserviceaccount.com" \
  --role="roles/bigquery.resourceViewer"

---

📄 License

MIT