dbt-spark-livy-iceberg

The dbt-spark-livy-iceberg adapter lets you use dbt with Apache Spark on Cloudera Data Platform through the Livy interface. It is based on the dbt-spark project and the Cloudera dbt-spark-livy adapter, extended with:

• Full Apache Iceberg support — file_format='iceberg' for tables, incremental models (append, merge, insert_overwrite), seeds, and snapshots.
• incremental_predicates for merges — pruning predicates are now passed into MERGE INTO ... ON, which the base adapter dropped.
• Iceberg table properties — a tblproperties config for setting Iceberg options like format-version and write.merge.mode.
• Dynamic Spark session configuration — a per-model spark_session_config config that issues SET key=value statements before the model runs.

The internal adapter type is still spark_livy, so existing profiles (method: livy) keep working. Only the pip distribution name changed.

Getting started

• Install dbt
• Read the introduction and viewpoint

Requirements

• Python >= 3.8
• dbt-core ~= 1.3.0
• pyspark, sqlparams, requests_kerberos, requests-toolbelt, python-decouple

Installing

pip install dbt-spark-livy-iceberg

Profile setup

demo_project:
  target: dev
  outputs:
    dev:
     type: spark_livy
     method: livy
     schema: my_db
     host: https://spark-livy-gateway.my.org.com/dbt-spark/cdp-proxy-api/livy_for_spark3/
     user: my_user
     password: my_pass
     # Optional: Spark conf applied once at Livy session creation.
     # Use this for catalog wiring (e.g. Iceberg catalog + extensions).
     livy_session_parameters:
       spark.sql.extensions: "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
       spark.sql.catalog.spark_catalog: "org.apache.iceberg.spark.SparkSessionCatalog"
       spark.sql.catalog.spark_catalog.type: "hive"

Iceberg models

Incremental merge

{{ config(
    materialized='incremental',
    file_format='iceberg',
    incremental_strategy='merge',
    unique_key='id',
    incremental_predicates=[
      "DBT_INTERNAL_DEST.event_date >= current_date - interval 7 days"
    ],
    tblproperties={
      'format-version': '2',
      'write.merge.mode': 'merge-on-read'
    }
) }}

select id, event_date, payload
from {{ ref('source_events') }}
{% if is_incremental() %}
where event_date >= current_date - interval 7 days
{% endif %}

• incremental_strategy supports append, merge, and insert_overwrite for Iceberg.
• incremental_predicates are appended to the merge ON clause to enable partition pruning.
• on_schema_change: sync_all_columns is supported for Iceberg (including dropping columns).

Dynamic Spark session configuration

Set Spark conf per model with spark_session_config. This issues SET key=value before the model builds.

{{ config(
    materialized='incremental',
    file_format='iceberg',
    incremental_strategy='insert_overwrite',
    partition_by=['event_date'],
    spark_session_config={
      'spark.sql.shuffle.partitions': '400',
      'spark.sql.sources.partitionOverwriteMode': 'dynamic'
    }
) }}
select ...

You can also set project-wide defaults in dbt_project.yml:

models:
  my_project:
    +spark_session_config:
      spark.sql.shuffle.partitions: '200'
    +tblproperties:
      write.merge.mode: 'merge-on-read'

Note: The Livy connection reuses a single shared SQL session for the whole dbt run, so SET values persist across models. Set the properties each model needs (or a project-level default) rather than assuming they reset between models.

Caveats

• While using livy, in the Livy UI if you notice sessions change state to dead from starting instead of idle, make sure there is a proper mapping for the user in the IDBroker mapping section.
• Actions > Manage Access > IDBroker Mappings. Reference
• Also make sure the workload password is set either through UI or CLI. Reference
• Iceberg catalog/extension wiring must be provided via Spark conf (livy_session_parameters in the profile, or spark_session_config per model).
• Python models currently write with the Delta writer; Iceberg support here targets SQL models.

Supported features

See the original adapter documentation: https://github.com/dbt-labs/dbt-spark and https://docs.getdbt.com/reference/warehouse-profiles/spark-profile