gusty
Gusty is an opinionated framework for data ETL built on top of Airflow, where every task is represented by one YAML file, and each task creates a view in a database.
Structure
The .yml approach to generating jobs within Airflow DAGs is not a new idea, but it is useful and there are a few built in benefits to it here.
-
Dependencies - Dependencies can quickly be set in
.ymlfiles through one of three means:- Using the
dependenciesspecification, you can set dependencies between jobs in the same DAG. - Using the
external_dependenciesspecification, you can set dependencies between jobs in different DAGs. - For the
MaterializedPostgresOperator, dependencies in the same DAG that are a part of theviewsschema are automatically registered.
- Using the
-
Operator configuration - After you build an operator, you can pass parameters to it in each
.ymljob definition file. This means that, for example, if you have to call different API endpoints, you may only need to build one operator to ingest data from this API, and then can specify the endpoint to call in the.ymljob definition file. -
Support for popular notebook formats - There are currently two notebook operators,
RmdOperatorandJupyterOperator, which enable you to simply write RMarkdown or Jupyter Notebook files and deploy them as jobs in your data pipeline. More importantly,RmdOperatorandJupyterOperatorare actually executed on separate dedicated docker containers, and interact with the Airflow container via SSH, which is useful if you want to deploy these services separately in the cloud!
Metadata
Release files for gusty 0.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| gusty-0.0.2.tar.gz | 8.5 kB | Without provenance | Details for gusty-0.0.2.tar.gz |
Release files / gusty-0.0.2.tar.gz
| Download URL | gusty-0.0.2.tar.gz |
|---|---|
| Size | 8.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d54dbfead71f43124406d62487a33692242a02ca5ec2df43e456f467c92452b0
|
|
BLAKE2b-256 checksum How to use checksums |
47b535dbc9881bf9c3fbd08b0876ecb9d3c093dd5839e1a9aae1e54c1ae502bf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.11.0 pkginfo/1.4.2 requests/2.22.0 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.42.0 CPython/3.6.5
|
Security for gusty 0.0.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.