Skip to main content

A framework for rapidly developing pipelines with Luigi

Project description

Build Status

ALPHA BUILD: A framework for rapidly developing pipelines with Luigi

[documentation]

Features:

  • Convention-over-configuration approach to Luigi projects

  • Standard workspace creation (lor new)

  • Configuration loading

  • A workspace-centric command-line (new, explain, ls, run, etc.)

  • Utility tasks (e.g. EnsureExistsOnLocalFilesystemTask)

  • Other utilities for managing long-running subprocesses, building a CLI, etc.

Installation

To install use pip:

$ pip install lor

Or clone the repo:

$ git clone https://github.com/adamkewley/lor.git

$ python setup.py install

Usage

Once installed, lor can be used from the command-line:

$ lor new foo

This creates a lor workspace at foo/. A lor workspace is a standard python3 pip project which depends on lor as a dependency (this can be removed if you don’t need it). The lor command-line command can detect if the terminal is in a lor workspace and provides convenience commands for working in a standard workspace.

Tasks written for lor are just standard Luigi tasks. The lor library provides various utility methods that a typical project might find useful (all opt-in). For example, lor provides a props module that you can use to load configuration properties at runtime:

# foo/tasks/bar

import lor.props
import luigi


class BarTask(luigi.Task):
    output_path = luigi.Parameter()

    def run(self):
        with open(str(self.output_path), "w") as f:
            config_prop = lor.props.get("CONFIG_PROP")
            f.write(config_prop)

    def output(self):
        return luigi.LocalTarget(self.output_path)

The above is a standard Luigi task that uses lor to load a configuration property called CONFIG_PROP from the workspace (held at etc/properties.yml in the workspace). The task is a standard Luigi task, so it can be ran from Luigi directly:

$ luigi –module foo.tasks.bar BarTask –output-path some/path

The lor run command can also be used to run the task:

$ lor run –module foo.tasks.bar BarTask –output-path some/path

The lor run command runs luigi by proxy, so it effectively has the same interface. However, lor run also adds useful functionality, such as the ability to override a variable’s value at runtime:

$ lor run –properties CONFIG_PROP=overridden –module foo.tasks.bar BarTask –output-path some/path

The task will then write “overridden” to the output file instead of whatever was loaded from the workspace’s configuration file. This is because lor run bootstraps the workspace global with the override before running Luigi.

TODO: This documentation is work in progress

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lor-0.0.14.tar.gz (21.8 kB view details)

Uploaded Source

File details

Details for the file lor-0.0.14.tar.gz.

File metadata

  • Download URL: lor-0.0.14.tar.gz
  • Upload date:
  • Size: 21.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for lor-0.0.14.tar.gz
Algorithm Hash digest
SHA256 2902d952c9ea78a4c3d6daac2e8695f498c48958baa2697039dbdf2d7e2d5d31
MD5 d3dd4920cc9da248f88cb483b578c312
BLAKE2b-256 b19f7d458337e09f1b9055739bf0fb3263bba16f3ebe0dab6e8b7745d7606d97

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page