Skip to main content

Azkaban CLI

Project description

A lightweight Azkaban client providing:

Installation

Using pip:

$ pip install azkaban

Command line interface

Once the package is installed, we have access to the azkaban command. From there, without leaving our terminal, we can:

  • Create and delete projects on an Azkaban server: azkaban create

  • Upload a project archive: azkaban upload

  • Run entire workflows, or individual jobs: azkaban run

Running azkaban --help shows the list of options for each of these commands.

These previous commands all take a --url parameter used to specify the Azkaban server (and user). In order to avoid having to do this every time, we can also define aliases in ~/.azkabanrc:

[foo]
url = http://url.to.foo.server:port
[bar]
url = http://url.to.bar.server
user = baruser

We can now interact directly to each of these URLs with the shorthand:

$ azkaban upload -a foo -z project.zip

This has the added benefit that we won’t have to authenticate on every upload. The session ID is cached and reused for later connections.

Syntax

For medium to large sized projects, it quickly becomes tricky to manage the multitude of files required for each workflow. .properties files are helpful but still do not provide the flexibility to generate jobs programmatically (i.e. using for loops, etc.). This approach also requires us to manually bundle and upload our project to the gateway every time.

We provide here a convenient framework to define jobs from a single python file. This framework is entirely compatible with the command line interface above, and even provides additional functionality (e.g. building and uploading projects in a single command).

Quickstart

We start by creating a configuration file for our project. Let’s call it jobs.py, the default file name the command line tool will look for. Here’s a simple example of how we could define a project with a single job and static file:

from azkaban import Job, Project

project = Project('foo')
project.add_file('/path/to/bar.txt', 'bar.txt')
project.add_job('bar', Job({'type': 'command', 'command': 'cat bar.txt'}))

The add_file method adds a file to the project archive (the second optional argument specifies the destination path inside the zip file). The add_job method will trigger the creation of a .job file. The first argument will be the file’s name, the second is a Job instance (cf. Job options).

Once we’ve saved our jobs file, the following additional commands are available to us:

  • azkaban list, see the list of all jobs in the current project.

  • azkaban view, view the contents of the .job file for a given job.

  • azkaban build, build the project archive and store it locally.

Job options

The Job class is a light wrapper which allows the creation of .job files using python dictionaries.

It also provides a convenient way to handle options shared across multiple jobs: the constructor can take in multiple options dictionaries and the last definition of an option (i.e. later in the arguments) will take precedence over earlier ones.

We can use this to efficiently share default options among jobs, for example:

defaults = {'user.to.proxy': 'boo', 'retries': 0}

jobs = [
  Job({'type': 'noop'}),
  Job(defaults, {'type': 'noop'}),
  Job(defaults, {'type': 'command', 'command': 'ls'}),
  Job(defaults, {'type': 'command', 'command': 'ls -l', 'retries': 1}),
]

All jobs except the first one will have their user.to.proxy property set. Note also that the last job overrides the retries property.

Alternatively, if we really don’t want to pass the defaults dictionary around, we can create a new Job subclass to do it for us:

class BooJob(Job):

  def __init__(self, *options):
    super(BooJob, self).__init__(defaults, *options)

More

Nested options

Nested dictionaries can be used to group options concisely:

# e.g. this job
Job({
  'proxy.user': 'boo',
  'proxy.keytab.location': '/path',
  'param.input': 'foo',
  'param.output': 'bar',
})
# is equivalent to this one
Job({
  'proxy': {'user': 'boo', 'keytab.location': '/path'},
  'param': {'input': 'foo', 'output': 'bar'},
})

Pig jobs

Because pig jobs are so common, a PigJob class is provided which accepts a file path (to the pig script) as first constructor argument, optionally followed by job options. It then automatically sets the job type and adds the corresponding script file to the project.

from azkaban import PigJob

project.add_job('baz', PigJob('/.../baz.pig', {'dependencies': 'bar'}))

Using a custom pig type is as simple as changing the PigJob.type class variable.

Merging projects

If you have multiple projects, you can merge them together to create a single project. The merge is done in place on the project the method is called on. The first project will retain its original name.

from azkaban import Job, Project

project1 = Project('foo')
project1.add_file('/path/to/bar.txt', 'bar.txt')
project1.add_job('bar', Job({'type': 'command', 'command': 'cat bar.txt'}))

project2 = Project('qux')
project2.add_file('/path/to/baz.txt', 'baz.txt')
project2.add_job('baz', Job({'type': 'command', 'command': 'cat baz.txt'}))

# project1 will now contain baz.txt and the baz job from project2
project2.merge_into(project1)

Next steps

Any valid python code can go inside the jobs configuration file. This includes using loops to add jobs, subclassing the base Job class to better suit a project’s needs (e.g. by implementing the on_add and on_build handlers), …

Extensions

Pig

Azkaban comes with a azkabanpig utility which enables us to run pig scripts directly. azkabanpig --help will display the list of available options (using UDFs, substituting parameters, running several scripts in order, etc.).

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

azkaban-0.2.7.tar.gz (20.9 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page