Skip to main content

Composable filesystem hooks and operators for Airflow.

Project description

airflow-fs

https://img.shields.io/circleci/project/github/jrderuiter/airflow-fs/master.svg https://img.shields.io/pypi/v/airflow_fs.svg

airflow-fs is Python package that provides hooks and operators for manipulating files across a variety of file systems using Apache Airflow.

Why airflow-fs?

Airflow-fs implements a single interface for different file system hooks, in contrast to Airflows builtin file system hooks/operators. This approach allows us to interact with files independently of the underlying file system, using a common set of operators for performing general operations such as copying and deleting files.

Currently, airflow-fs supports the following file systems: local, FTP, HDFS, S3 and SFTP. Support for additional file systems can be added by implementing an additional file system hook that adheres to the required hook interface. See the documentation for more details.

Documentation

Detailed documentation is available at: https://jrderuiter.github.io/airflow-fs.

License

This software is freely available under the MIT license.

History

Version 0.1.0

Initial release supporting local, FTP, HDFS, S3 and SFTP file systems.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow_fs-0.1.0.tar.gz (27.5 kB view details)

Uploaded Source

Built Distribution

airflow_fs-0.1.0-py2.py3-none-any.whl (14.2 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file airflow_fs-0.1.0.tar.gz.

File metadata

  • Download URL: airflow_fs-0.1.0.tar.gz
  • Upload date:
  • Size: 27.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.2

File hashes

Hashes for airflow_fs-0.1.0.tar.gz
Algorithm Hash digest
SHA256 2e388d5cedc23cd32b382f4ba221e0eb106f1205c90034b49210016783478c98
MD5 59928ecc3a7f882afd458532febda6dc
BLAKE2b-256 ee939dd4d7aae61aef9079f1d9f720ded7f643047ac6d52675ecd0fa4cb4de2d

See more details on using hashes here.

File details

Details for the file airflow_fs-0.1.0-py2.py3-none-any.whl.

File metadata

  • Download URL: airflow_fs-0.1.0-py2.py3-none-any.whl
  • Upload date:
  • Size: 14.2 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.2

File hashes

Hashes for airflow_fs-0.1.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 16a34250539bb37e412baf90bdd854d3a2dff47ada179dfd5b1eb267ba7f2124
MD5 178814b060e288c26b4441ab0b248ff7
BLAKE2b-256 feade91244b4fb359ed2d27f8d56db166d4af34303e7440652db6f64d1d4d50c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page