Skip to main content

Entry point for the Nessie Airflow provider.

Project description

Nessie Airflow Provider

build codecov PyPI version


Documentation: https://projectnessie.github.io/nessie_provider

Source Code: https://github.com/projectnessie/nessie_provider


Usage

To use in airflow install via pip pip install airflow-provider-nessie. See Nessie Documentation for instructions on starting and using a Nessie server.

Operators and Hooks

To interact with Nessie from an airflow DAG you have the following options:

  • Nessie Hook: register as a connection w/ Airflow and store your Nessie url and credentials
  • Create reference operator: Create a Branch or Tag as part of an airflow DAG
  • Delete reference operator: Delete a Branch or Tag as part of an airflow DAG
  • Commit operator: commit objects to the Nessie database on a given branch
  • Merge operator: merge one branch into another

These can be seen in action by looking at the Example DAGs. The basic_nessie.py DAG shows each operator in action and the spark_nessie_iceberg.py DAG shows a more complicated example of performing an iceberg transaction in Nessie from the Spark operator.

Development

Setup environement

You should have Pipenv installed. Then, you can install the dependencies with:

pipenv install --dev

After that, activate the virtual environment:

pipenv shell

Run unit tests

You can run all the tests with:

make test

Alternatively, you can run pytest yourself:

pytest

Format the code

Execute the following command to apply isort and black formatting:

make format

License

This project is licensed under the terms of the Apache Software License 2.0.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-provider-nessie-0.1.2.tar.gz (52.3 kB view details)

Uploaded Source

Built Distribution

airflow_provider_nessie-0.1.2-py3-none-any.whl (11.8 kB view details)

Uploaded Python 3

File details

Details for the file airflow-provider-nessie-0.1.2.tar.gz.

File metadata

File hashes

Hashes for airflow-provider-nessie-0.1.2.tar.gz
Algorithm Hash digest
SHA256 05636b5fcd484f9d0eda7c68f4eb010d0c41cffa270f2974d59b40b948045834
MD5 1a233bec3e0e174897b477ebb565e90a
BLAKE2b-256 68a87ba727ee5d31446f56bd5b4cdb110d00432556fc897d9db637d984f7b24b

See more details on using hashes here.

File details

Details for the file airflow_provider_nessie-0.1.2-py3-none-any.whl.

File metadata

File hashes

Hashes for airflow_provider_nessie-0.1.2-py3-none-any.whl
Algorithm Hash digest
SHA256 bd004f20b58b3d393d5035803fa1ead0ef9ad4ea16193f0a7c20e9fcaf29d7a1
MD5 a11750028668a799a741dc2064582f9c
BLAKE2b-256 350f82ef17ea4aed05c2a8fae9aebbb19c7e2815b5c5b7cedb8f01c8592fa127

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page