Entry point for the Nessie Airflow provider.
Project description
Nessie Airflow Provider
Documentation: https://projectnessie.github.io/nessie_provider
Source Code: https://github.com/projectnessie/nessie_provider
Usage
To use in airflow install via pip pip install airflow-provider-nessie
. See Nessie Documentation for
instructions on starting and using a Nessie server.
Operators and Hooks
To interact with Nessie from an airflow DAG you have the following options:
- Nessie Hook: register as a connection w/ Airflow and store your Nessie url and credentials
- Create reference operator: Create a Branch or Tag as part of an airflow DAG
- Delete reference operator: Delete a Branch or Tag as part of an airflow DAG
- Commit operator: commit objects to the Nessie database on a given branch
- Merge operator: merge one branch into another
These can be seen in action by looking at the Example DAGs.
The basic_nessie.py
DAG shows each operator in action and the spark_nessie_iceberg.py
DAG shows a more complicated example of performing an iceberg
transaction in Nessie from the Spark operator.
Development
Setup environement
You should have Pipenv installed. Then, you can install the dependencies with:
pipenv install --dev
After that, activate the virtual environment:
pipenv shell
Run unit tests
You can run all the tests with:
make test
Alternatively, you can run pytest
yourself:
pytest
Format the code
Execute the following command to apply isort
and black
formatting:
make format
License
This project is licensed under the terms of the Apache Software License 2.0.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file airflow-provider-nessie-0.1.2.tar.gz
.
File metadata
- Download URL: airflow-provider-nessie-0.1.2.tar.gz
- Upload date:
- Size: 52.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: python-requests/2.26.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 05636b5fcd484f9d0eda7c68f4eb010d0c41cffa270f2974d59b40b948045834 |
|
MD5 | 1a233bec3e0e174897b477ebb565e90a |
|
BLAKE2b-256 | 68a87ba727ee5d31446f56bd5b4cdb110d00432556fc897d9db637d984f7b24b |
File details
Details for the file airflow_provider_nessie-0.1.2-py3-none-any.whl
.
File metadata
- Download URL: airflow_provider_nessie-0.1.2-py3-none-any.whl
- Upload date:
- Size: 11.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: python-requests/2.26.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | bd004f20b58b3d393d5035803fa1ead0ef9ad4ea16193f0a7c20e9fcaf29d7a1 |
|
MD5 | a11750028668a799a741dc2064582f9c |
|
BLAKE2b-256 | 350f82ef17ea4aed05c2a8fae9aebbb19c7e2815b5c5b7cedb8f01c8592fa127 |