Skip to main content

Python commandline tool to make deployment of Airflow projects easier.

Project description

afctl

The proposed CLI tool is being authored to make creating and deployment of airflow projects faster and smoother. As of now, there is no tool out there that can empower the user to create a boilerplate code structure for airflow projects and make development + deployment of projects seamless.

Installation

Clone the repository.
Create a new python virtualenv. You can use the following command.

python3 -m venv <name>

Activate your virtualenv

source /path_to_venv/bin/activate

Step into the afctl directory and run

pip3 install .

Requirements

Python 3.5+ Docker

Usage

Commands right now supported are

  • init
  • config
  • deploy
  • list
  • generate

To learn more, run

afctl <command> -h

Initialize a new afctl project.

The project is created in your present working directory. Along with this a configuration file with the same name is generated in /home/.afctl_configs directory.

afctl init “name of the project”
  • Creates a new project directory.
  • Creates a config file in the home directory
afctl init .
  • Initialize the current directory as a project
  • If the directory is already a git repository then the global configs will get automatically set.

Manage configurations

The configuration file is used for deployment.

global:
-git:
--origin:
deployment:
-qubole:
--docker-local:
---compose:

TYPES:
   add - add a config for your deployment.
   update - update an existing config for your deployment.
       Arguments:
           -d : Deployment Type
           -p : Project
            [ Qubole ]
               -n : name of deployment
               -e : name of environment
               -c : cluster label
               -t : auth token
            [ docker-local ]
               Cannot add/update configs.
   global
       Arguments:
           -p : Project
           -o : Set git origin for deployment
           -t : Set personal access token
   show -  Show the config file on console
       No arguments.

positional arguments:
  {add,update,show,global}

optional arguments:
  -h, --help            show this help message and exit
  -d {qubole}
  -o O
  -p P
  -n N
  -e E
  -c C
  -t T
afctl config global -o {origin}
  • Will set the git origin for your project.
  • Supports both inline arguments as well as promoting for input.
afctl configs add -d {deployment}
  • Prompts the user to input connector related values.
  • Can also provide values as inline arguments.
afctl configs update -d {deployment}
  • Prompts the user to update the values
  • Can also provide inline arguments.

Deploy projects

TYPES:
   [qubole] - Deploy your project to QDS.
       Arguments:
           -n : Name of the deployment
   [docker-local] - Deploy your project to local docker.
       Arguments:
           -d : To run in daemon mode

positional arguments:
  {qubole,docker-local}

optional arguments:
  -h, --help            show this help message and exit
  -d
  -n N
afctl deploy {deployment} -n {name of deployment}
  • Fetches the latest commit in remote origin.
  • Deploys the repository with the last commit as shown.
afctl config show
  • Prints the configuration file on the console.

Caution

Not yet ported for Windows.

Credits

Docker-compose file : https://github.com/puckel/docker-airflow

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

afctl-0.1.5.tar.gz (29.1 kB view details)

Uploaded Source

File details

Details for the file afctl-0.1.5.tar.gz.

File metadata

  • Download URL: afctl-0.1.5.tar.gz
  • Upload date:
  • Size: 29.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.7.5

File hashes

Hashes for afctl-0.1.5.tar.gz
Algorithm Hash digest
SHA256 f1182cf33051dd1e78c301c8b895a2a38bcd245796b0f28a7f00e396e5790636
MD5 a16e9de0b11388aa30ed24c2e386caee
BLAKE2b-256 ccebedde99420b798924715c7509fe1e4b03a6de8b3c0b209cde215726a86083

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page