dbt
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.
dbt-singlestore
dbt-singlestore enables dbt to work with SingleStore.
For more information on using dbt with SingleStore, consult the docs.
Installation
dbt-singlestore is available on PyPI. To install the latest version via pip, run:
pip install dbt-singlestore
Configuring your profile
A sample profile can be found in
sample_profiles.yml. It is a standard profile for SQL data sources in dbt.
In SingleStore, like in MySQL, database and schema denote the same concept. In dbt, and also in a number of DBMS (e.g. Postgres) schema has a different meaning (a namespace within a database). Therefore, schema must be specified in dbt profile, but it won't affect database objects by default. However, it can be used as a prefix for table names, as outlined in dbt docs for SingleStore profile.
Supported Features
| Category | Feature | Supported? | Tested? |
|---|---|---|---|
| Materializations | Table | Yes | Yes |
| View | Yes | Yes | |
| Incremental | Yes | Yes | |
| Ephemeral | Yes | Yes | |
| Resources | Seeds | Yes | Yes |
| Sources | Yes | Yes | |
| Testing - Bespoke | Yes | Yes | |
| Testing - Generic | Yes | Yes | |
| dbt Documentation | Yes | Yes | |
| Snapshots | Yes | Yes | |
| Administration | Hooks | Yes | Yes |
| Models | Custom schema | Limited* | Yes |
-
Custom schemas can be defined in a project models, but they can only serve as a modification to the prefix of the corresponding table names.
-
SSL connection would be automatically established by dbt-singlestore for SingleStore users created with 'Require SSL' flag.
SingleStore-specific table options
Reference tables
SingleStore supports REFERENCE tables, which are replicated across the cluster and are useful for small/dimension tables that are frequently joined.
To create a REFERENCE table from a dbt model, set reference=true on a table materialization:
{{
config(
materialized='table',
reference=true,
)
}}
select ...
When reference=true (default false), the adapter generates CREATE REFERENCE TABLE ... rather than a regular CREATE TABLE ...
Rowstore reference tables
If you want a rowstore reference table, set storage_type='rowstore':
{{
config(
materialized='table',
reference=true,
storage_type='rowstore',
)
}}
select ...
This maps to CREATE ROWSTORE REFERENCE TABLE ...
Restrictions / validation
To match SingleStore semantics, dbt-singlestore enforces:
-
No
shard_keywithreference=true— reference tables don’t use sharding; compilation fails if both are set. -
No
temporary=truewithreference=true— SingleStore does not support temporary reference tables; compilation fails if both are set.
Testing and supported versions
Default dbt tests and jaffle_shop project are used to check the adapter functionality. Development overview has a section "Run tests" which contains instructions on running the tests. Currently, the tests have been successfully run for the following product versions:
| SingleStore | dbt-core | dbt-tests-adapter |
|---|---|---|
| 9.0.9 | 1.10.13 | 1.19.1 |
| 8.9.10 | 1.10.13 | 1.19.1 |
| 8.7.43 | 1.9.9 | 1.17.0 |
| 8.7.43 | 1.8.13 | 1.11.0 |
| 8.5.16 | 1.7.15 | 1.7.15 |
| 8.5.16 | 1.6.13 | 1.6.13 |
| 8.1.1 | 1.6.0 | 1.6.0 |
| 8.1.1 | 1.5.0 | 1.5.0 |
| 8.1.1 | 1.4.0 | 1.4.0 |
| 8.1.1 | 1.3.0 | 1.3.0 |
| 7.8.29 | 1.2.2 | 1.2.2 |
| SingleStore | dbt-core | pytest-dbt-adapter |
|---|---|---|
| 7.8.3 | 1.1.1 | 0.6.0 |
| 7.6.6 | 1.0.1 | 0.6.0 |
| 7.5.12 | 1.0.1 | 0.6.0 |
To use this adapter, SingleStore must be upgraded to the version 8.5 or newer.
Contributors
We thank Doug Beatty who build an adapter for mysql which has been used to build an initial version of this adapter.
Contribute
- Want to help us build
dbt-singlestore? Check out the Contributing Guide.
Metadata
Release files for dbt-singlestore 1.10.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dbt_singlestore-1.10.1.tar.gz | 29.3 kB | Without provenance | Details for dbt_singlestore-1.10.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dbt_singlestore-1.10.1-py3-none-any.whl | Python 3 | none | any | Details for dbt_singlestore-1.10.1-py3-none-any.whl |
Total release size: 74.4 kB
Release files / dbt_singlestore-1.10.1.tar.gz
| Download URL | dbt_singlestore-1.10.1.tar.gz |
|---|---|
| Size | 29.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
25c4b50b9caef04482eac58cb87aac6d4389a63176d69facecd9c9c5696914e6
|
|
BLAKE2b-256 checksum How to use checksums |
e6826dea1883415efa2058b1bb1db943c2b6973c51dd2ff48b6036919da01fe4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.15
|
Release files / dbt_singlestore-1.10.1-py3-none-any.whl
| Download URL | dbt_singlestore-1.10.1-py3-none-any.whl |
|---|---|
| Size | 45.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
591ee003681aa945c4d672c60a7f996a65e9ef1d0f1e2367f0a72febb6f90d5c
|
|
BLAKE2b-256 checksum How to use checksums |
dd207fb04ed621191b172b9d2f1ebc36e1450eb7feedf7459903370ba6c5bcdc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.15
|
Security for dbt-singlestore 1.10.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.