Skip to main content

The Apache Spark adapter plugin for dbt

Project description

dbt logo

dbt

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.

dbt-spark

dbt-spark enables dbt to work with Apache Spark. For more information on using dbt with Spark, consult the docs.

Getting started

Review the repository README.md as most of that information pertains to dbt-spark.

Running locally

A docker-compose environment starts a Spark Thrift server and a Postgres database as a Hive Metastore backend. Note: dbt-spark now supports Spark 3.3.2.

The following command starts two docker containers:

docker-compose up -d

It will take a bit of time for the instance to start, you can check the logs of the two containers. If the instance doesn't start correctly, try the complete reset command listed below and then try start again.

Create a profile like this one:

spark_testing:
  target: local
  outputs:
    local:
      type: spark
      method: thrift
      host: 127.0.0.1
      port: 10000
      user: dbt
      schema: analytics
      connect_retries: 5
      connect_timeout: 60
      retry_all: true

Connecting to the local spark instance:

  • The Spark UI should be available at http://localhost:4040/sqlserver/
  • The endpoint for SQL-based testing is at http://localhost:10000 and can be referenced with the Hive or Spark JDBC drivers using connection string jdbc:hive2://localhost:10000 and default credentials dbt:dbt

Note that the Hive metastore data is persisted under ./.hive-metastore/, and the Spark-produced data under ./.spark-warehouse/. To completely reset you environment run the following:

docker-compose down
rm -rf ./.hive-metastore/
rm -rf ./.spark-warehouse/

Additional Configuration for MacOS

If installing on MacOS, use homebrew to install required dependencies.

brew install unixodbc

Contribute

Project details


Release history Release notifications | RSS feed

This version

1.9.2

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dbt_spark-1.9.2.tar.gz (38.7 kB view details)

Uploaded Source

Built Distribution

dbt_spark-1.9.2-py3-none-any.whl (50.2 kB view details)

Uploaded Python 3

File details

Details for the file dbt_spark-1.9.2.tar.gz.

File metadata

  • Download URL: dbt_spark-1.9.2.tar.gz
  • Upload date:
  • Size: 38.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for dbt_spark-1.9.2.tar.gz
Algorithm Hash digest
SHA256 1fd6763f2220213515ccb55ae172a335edaf93541e7c8d351ea33529278867e7
MD5 35ec0f675ff9ca77a4f5169837950285
BLAKE2b-256 d7f0ae9dcacd84e98b9d09656ae2defa51f1ae74b0fc9251d304dbeb0d7c2619

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_spark-1.9.2.tar.gz:

Publisher: publish-oss.yml on dbt-labs/dbt-adapters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dbt_spark-1.9.2-py3-none-any.whl.

File metadata

  • Download URL: dbt_spark-1.9.2-py3-none-any.whl
  • Upload date:
  • Size: 50.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for dbt_spark-1.9.2-py3-none-any.whl
Algorithm Hash digest
SHA256 840b943136a7ab756bfac0f5e4ef2c4913c667a4601f68da563fe53814196aa2
MD5 a83469054aaa96c2c52f6acb9704d7f4
BLAKE2b-256 1bc526cbce88eb7ae39b30002cacab7f392a8ece9715184ce96e1f6ec5bcfe73

See more details on using hashes here.

Provenance

The following attestation bundles were made for dbt_spark-1.9.2-py3-none-any.whl:

Publisher: publish-oss.yml on dbt-labs/dbt-adapters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page