Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Allow Luigi tasks to fail softly, without cancelling downstream tasks.

Project description

Provides a decorator for Luigi tasks that allows them to fail softly. Tasks that fail softly are seen by Luigi as complete, and thus don't prevent dependent tasks from running. The dependent task can check if it's dependencies actually ran successfully or failed softly.

Installation

pip install luigi_soft_failures

Usage

from luigi_soft_failures import softly_failing

Use as class decorator:

@softly_failing(catch_all=True, propagate=True)
class SomeTask(luigi.Task):
    ...

Or on demand in the requiring task:

def requires(self):
    return softly_failing(catch_all=True)(SomeTask)(some_param=42)

The dependent task can check the status of it's dependencies using failed_softly:

def run(self):
    if self.requires().failed_softly():
        ...

For a complete example see as_decorator.py.

API

softly_failing accepts the following parameters:

  • catch_all (bool, default False):

    When True, any exception thrown in the task's run method will lead to a soft failure. Otherwise, soft failures can be generated manually by calling self.fail_softly('Some error message') from the task's run method and exiting the method without exception.

  • propagate (bool, default False):

    When True, the task fails softly if any of it's dependencies failed softly, and run is never executed. Otherwise, run is executed as if the dependencies ran successfully.

  • output_dir (str, default None): Described below.

Storage of soft failure reports

Whan the wrapped task fails softly, it creates a report with the failure message or exception traceback to indicate this. These reports are stored in a directory specified in one of the following ways (in this order of precedence):

  • output_dir parameter passed to softly_failing

  • Specified in luigi.cfg:

    [luigi_soft_failures.Config]
    output_dir=/some/path
    
  • Environment variable LUIGI_SOFT_FAILURES_OUTPUT_DIR

  • Default ./soft_failures/

Limitations

  • Soft failure status is stored using a LocalTarget, so a local storage that all workers have access to is required.
  • In the Luigi visualizer, softly failed tasks are shown as complete.

Pull requests to address these are very welcome!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for luigi-soft-failures, version 0.1.1
Filename, size File type Python version Upload date Hashes
Filename, size luigi_soft_failures-0.1.1-py2.py3-none-any.whl (5.3 kB) File type Wheel Python version py2.py3 Upload date Hashes View hashes
Filename, size luigi_soft_failures-0.1.1.tar.gz (5.4 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page