Send summary messages of your Luigi jobs to Slack.
Project description
## Luigi Monitor
[![Pypi Latest Version](https://img.shields.io/pypi/v/luigi-monitor.svg)](https://img.shields.io/pypi/v/luigi-monitor.svg)
[![License](https://img.shields.io/pypi/l/luigi-monitor.svg)](https://img.shields.io/pypi/l/luigi-monitor.svg)
[![Python Versions](https://img.shields.io/pypi/pyversions/luigi-monitor.svg)](https://img.shields.io/pypi/pyversions/luigi-monitor.svg)
[![Pypi Format](https://img.shields.io/pypi/format/luigi-monitor.svg)](https://img.shields.io/pypi/format/luigi-monitor.svg)
[![Build Status](https://travis-ci.org/hudl/luigi-monitor.svg?branch=master)](https://travis-ci.org/hudl/luigi-monitor)
![message](https://raw.github.com/hudl/luigi-monitor/master/message.png)
Send summary messages of your Luigi jobs to Slack.
### Overview
[Luigi](https://github.com/spotify/luigi) is a great tool for making
job pipelines, but it's hard to know the status of a run. Luigi has
support for error emails, but this requires configuring your machine
to send email, which is a hassle for short-lived EMR clusters. Further,
it sends an email for every failure, which can quickly swamp your inbox.
By contrast, this tool gathers all your failures and missing dependencies
and sends a summary Slack message when the job is finished.
### Usage
With default app username:
```python
import luigi
from luigi_monitor import monitor
...
if __name__ == "__main__":
with monitor(slack_url=<your_slack_url>, max_print=10):
luigi.run(main_task_cls=MainClass)
```
With dynamic app username:
```python
import luigi
from luigi_monitor import monitor
...
if __name__ == "__main__":
with monitor(slack_url=<your_slack_url>, max_print=10, username="FooBar Monitor"):
luigi.run(main_task_cls=MainClass)
```
Monitoring and notifying on various events:
Currently supports: `SUCCESS`, `DEPENDENCY_MISSING`, and `FAILURE`
By default, all three of the above are monitored and notified on. If, `SUCCESS` event is monitored and
all tasks succeed then the notification text is "Job ran successfully" instead of listing _all_
successful tasks.
```python
import luigi
from luigi_monitor import monitor
...
if __name__ == "__main__":
with monitor(slack_url=<your_slack_url>, events=['DEPENDENCY_MISSING', 'FAILURE']):
luigi.run(main_task_cls=MainClass)
```
Alternatively:
`luigi-monitor --module path.to.module TaskName`
NB: if you plan to use luigi-monitor from the command line, set options using `luigi.cfg`:
```
[luigi-monitor]
slack_url=<slack_hook>
max_print=<int>
username=<string>
```
This is a work in progress. Particularly, note that:
* It only sends notifications via Slack
[![Pypi Latest Version](https://img.shields.io/pypi/v/luigi-monitor.svg)](https://img.shields.io/pypi/v/luigi-monitor.svg)
[![License](https://img.shields.io/pypi/l/luigi-monitor.svg)](https://img.shields.io/pypi/l/luigi-monitor.svg)
[![Python Versions](https://img.shields.io/pypi/pyversions/luigi-monitor.svg)](https://img.shields.io/pypi/pyversions/luigi-monitor.svg)
[![Pypi Format](https://img.shields.io/pypi/format/luigi-monitor.svg)](https://img.shields.io/pypi/format/luigi-monitor.svg)
[![Build Status](https://travis-ci.org/hudl/luigi-monitor.svg?branch=master)](https://travis-ci.org/hudl/luigi-monitor)
![message](https://raw.github.com/hudl/luigi-monitor/master/message.png)
Send summary messages of your Luigi jobs to Slack.
### Overview
[Luigi](https://github.com/spotify/luigi) is a great tool for making
job pipelines, but it's hard to know the status of a run. Luigi has
support for error emails, but this requires configuring your machine
to send email, which is a hassle for short-lived EMR clusters. Further,
it sends an email for every failure, which can quickly swamp your inbox.
By contrast, this tool gathers all your failures and missing dependencies
and sends a summary Slack message when the job is finished.
### Usage
With default app username:
```python
import luigi
from luigi_monitor import monitor
...
if __name__ == "__main__":
with monitor(slack_url=<your_slack_url>, max_print=10):
luigi.run(main_task_cls=MainClass)
```
With dynamic app username:
```python
import luigi
from luigi_monitor import monitor
...
if __name__ == "__main__":
with monitor(slack_url=<your_slack_url>, max_print=10, username="FooBar Monitor"):
luigi.run(main_task_cls=MainClass)
```
Monitoring and notifying on various events:
Currently supports: `SUCCESS`, `DEPENDENCY_MISSING`, and `FAILURE`
By default, all three of the above are monitored and notified on. If, `SUCCESS` event is monitored and
all tasks succeed then the notification text is "Job ran successfully" instead of listing _all_
successful tasks.
```python
import luigi
from luigi_monitor import monitor
...
if __name__ == "__main__":
with monitor(slack_url=<your_slack_url>, events=['DEPENDENCY_MISSING', 'FAILURE']):
luigi.run(main_task_cls=MainClass)
```
Alternatively:
`luigi-monitor --module path.to.module TaskName`
NB: if you plan to use luigi-monitor from the command line, set options using `luigi.cfg`:
```
[luigi-monitor]
slack_url=<slack_hook>
max_print=<int>
username=<string>
```
This is a work in progress. Particularly, note that:
* It only sends notifications via Slack
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
luigi-monitor-1.1.4.tar.gz
(4.3 kB
view details)
Built Distribution
File details
Details for the file luigi-monitor-1.1.4.tar.gz
.
File metadata
- Download URL: luigi-monitor-1.1.4.tar.gz
- Upload date:
- Size: 4.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: Python-urllib/3.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 111661fc9771f7da5404d16d1415f2fe75d9b65ca7c926fe4b718218d35aae77 |
|
MD5 | b86467e678b10706774d4c8c85dfdadd |
|
BLAKE2b-256 | 61a4b170904b6a9a5c49a39231c582bd7bed3692ed4326542f6319437803b427 |
File details
Details for the file luigi_monitor-1.1.4-py2.py3-none-any.whl
.
File metadata
- Download URL: luigi_monitor-1.1.4-py2.py3-none-any.whl
- Upload date:
- Size: 7.0 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: Python-urllib/3.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3dad3b2e9a67c9edffa7234e9d28631d680b009bf4f68e4971737c829477ee53 |
|
MD5 | 9961913090f9438458f71d10c1136e2b |
|
BLAKE2b-256 | c303a4355514559a00a36370dc29358214049daa6cd9dc968e34db8274b227ce |