Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

SQLBucket

https://travis-ci.org/socialpoint-labs/sqlbucket.svg?branch=master

SQLBucket was built to help write, orchestrate and validate SQL ETL. It gives the possibility to set variables and introduces some control flow like if/else and for loops, to make your queries dynamic when writing them. It also implements a very simplistic integration testing framework to validate the results of your ETL pipelines in the form of SQL checks.

Lightweight, it can work as a stand alone service, or be part of your workflow manager environment (Airflow, Luigi, ..).

Installing

Install and update using pip:

pip install -U sqlbucket

SQLBucket works only for Python 3.6 and 3.7, and probably 3.8 although not tested yet.

A Simple Example

To start working with SQLBucket, you need to have a ‘projects’ folder that will contain all your SQL ETL. SQLBucket, essentially works with the following folder structure as a root folder, where you can have as many projects as you want.

projects/
    |-- project1/
    |-- project2/
    |-- project3/
        ...

Inside a project, it must contain a folder called queries that will contains your SQL files to be ran, and a config.yaml that will let you set the order in which those queries must be processed. SQLBucket leverages on the amazing Jinja2 templating library to give you the possibility to set variables in your SQL as well as giving you pure execution flows like for loops. To see in more depth what can be done, see the documentation on how to write SQL in SQLBucket.

In practice, this is how a project folder would look.

projects/
    |-- my_super_project/
        |-- config.yaml
        |-- queries/
            |-- my_super_insert_query.sql
            |-- some_other_query.sql
        |-- integrity/
            |-- test1.sql
            |-- test2.sql

The integrity folder gives you the possibility to write some checks in SQL, that will ran at the end of your ETL to validate your data. Check documentation on integrity for a more detailed explanation on testing the integrity of your ETL.

This is how you would launch a project:

from sqlbucket import SQLBucket

connections = {
'db_test': 'postgresql://user:password@host:5439/database'
}

bucket = SQLBucket(connections=connections)
project = bucket.load_project(
    project_name='fat_etl',
    connection_name='db_test',
    variables={'foo': 1}
)

project.run()
project.run_integrity()

This would trigger logs as below.

documentation/images/terminal.gif

You can also launch a project using the sqlbucket command line interface.

Contributing

For guidance on how to make a contribution to SQLBucket, see the contributing guidelines.

Metadata

Release files for sqlbucket 0.2.10.dev3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sqlbucket 0.2.10.dev3
File Size Uploaded Provenance Actions
sqlbucket-0.2.10.dev3.tar.gz 11.8 kB Without provenance Details for sqlbucket-0.2.10.dev3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sqlbucket 0.2.10.dev3
File Interpreter ABI Platform Actions
sqlbucket-0.2.10.dev3-py3-none-any.whl Python 3 none any Details for sqlbucket-0.2.10.dev3-py3-none-any.whl

Total release size: 25.2 kB

Release files / sqlbucket-0.2.10.dev3.tar.gz

Download URL sqlbucket-0.2.10.dev3.tar.gz
Size 11.8 kB
Tags Source
SHA-256 checksum
How to use checksums
a062811bdc04e070344ae3a5196754ebe7d656db6cb86c3e005af62a38109519
BLAKE2b-256 checksum
How to use checksums
9681a536d808e50cc401f40d369822b6a4e053d48c29292b3167f55354fc3e23
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.13.0 pkginfo/1.5.0.1 requests/2.9.1 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.5.3

Release files / sqlbucket-0.2.10.dev3-py3-none-any.whl

Download URL sqlbucket-0.2.10.dev3-py3-none-any.whl
Size 13.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
58d3932b431d7c2d37557ba07fcd8be6bd15600f71c16db6bb4f4bbd3c99be65
BLAKE2b-256 checksum
How to use checksums
791dd6c6c9ca1e1346774edeefe9d83de13d71d6d6fd68b4a236937176a67ffb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.13.0 pkginfo/1.5.0.1 requests/2.9.1 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.5.3

Security for sqlbucket 0.2.10.dev3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page