azkaban 0.5.6

AzkabanCLI

AzkabanCLI

A lightweight Azkaban client providing:

• A command line interface to run workflows, upload projects, etc.

• A convenient and extensible way for building projects.

Sample

Below is a simple configuration file for a project containing a workflow with four pig scripts.

from azkaban import PigJob, Project
from getpass import getuser

project = Project('sample', root=__file__)

# default options for all scripts
default_options = {
  'user.to.proxy': getuser(),
  'mapred': {
    'max.split.size': 2684354560,
    'min.split.size': 2684354560,
  },
}

# dictionary of pig script options, keyed on the pig script path
pig_job_options = {
  'first.pig': {},
  'second.pig': {'dependencies': 'first.pig'},
  'third.pig': {'param': {'foo': 48, 'bar': 'abc'}},
  'fourth.pig': {'dependencies': 'second.pig,third.pig'},
}

for path, options in pig_job_options.items():
  project.add_job(path, PigJob(path, default_options, options))

More examples are also available.

Documentation

The full documentation can be found here.

Installation

Using pip:

$ pip install azkaban