lemiknow 0.0.1

Let's you know when your function call ends or crashes

Le Mi Know (when it's done)

A small library that extends on knockknock to get a notification when your function call is complete or when it crashes during the process with two additional lines of code.

When training deep learning models, it is common to use early stopping. Apart from a rough estimate, it is difficult to predict when the training will finish. Thus, it can be interesting to set up automatic notifications for your training. It is also interesting to be notified when your training crashes in the middle of the process for unexpected reasons.

Installation

Install with pip or equivalent.

pip install lemiknow

This code has only been tested with Python >= 3.6.

Usage

The library is designed to be used in a seamless way, with minimal code modification: you only need to add a decorator on top your main function call. The return value (if there is one) is also reported in the notification.

There are currently eight ways to setup notifications:

Platform	External Contributors
email	-
Slack	-
Telegram	-
Microsoft Teams	@noklam
Text Message	@abhishekkrthakur
Discord	@watkinsm
Desktop	@atakanyenel
Matrix	@jcklie

Email

The service relies on Yagmail a GMAIL/SMTP client. You'll need a gmail email address to use it (you can setup one here, it's free). I recommend creating a new one (rather than your usual one) since you'll have to modify the account's security settings to allow the Python library to access it by Turning on less secure apps.

Python

from lemiknow import email_sender

@email_sender(recipient_emails=["<[e-mail]>", "<[e-mail]>"], sender_email="<grandma'[e-mail]>")
def train_your_nicest_model(your_nicest_parameters):
    import time
    time.sleep(10000)
    return {'loss': 0.9} # Optional return value

Command-line

lemiknow email \
    --recipient-emails <[e-mail]>,<[e-mail]> \
    --sender-email <grandma'[e-mail]> \
    sleep 10

If sender_email is not specified, then the first email in recipient_emails will be used as the sender's email.

Note that launching this will asks you for the sender's email password. It will be safely stored in the system keyring service through the keyring Python library.

Slack

Similarly, you can also use Slack to get notifications. You'll have to get your Slack room webhook URL and optionally your user id (if you want to tag yourself or someone else).

Python

from lemiknow import slack_sender

webhook_url = "<webhook_url_to_your_slack_room>"
@slack_sender(webhook_url=webhook_url, channel="<your_favorite_slack_channel>")
def train_your_nicest_model(your_nicest_parameters):
    import time
    time.sleep(10000)
    return {'loss': 0.9} # Optional return value

You can also specify an optional argument to tag specific people: user_mentions=[<your_slack_id>, <grandma's_slack_id>].

Command-line

lemiknow slack \
    --webhook-url <webhook_url_to_your_slack_room> \
    --channel <your_favorite_slack_channel> \
    sleep 10

You can also specify an optional argument to tag specific people: --user-mentions <your_slack_id>,<grandma's_slack_id>.

Telegram

You can also use Telegram Messenger to get notifications. You'll first have to create your own notification bot by following the three steps provided by Telegram here and save your API access TOKEN.

Telegram bots are shy and can't send the first message so you'll have to do the first step. By sending the first message, you'll be able to get the chat_id required (identification of your messaging room) by visiting https://api.telegram.org/bot<YourBOTToken>/getUpdates and get the int under the key message['chat']['id'].

Python

from lemiknow import telegram_sender

CHAT_ID: int = <your_messaging_room_id>
@telegram_sender(token="<your_api_token>", chat_id=CHAT_ID)
def train_your_nicest_model(your_nicest_parameters):
    import time
    time.sleep(10000)
    return {'loss': 0.9} # Optional return value

Command-line

lemiknow telegram \
    --token <your_api_token> \
    --chat-id <your_messaging_room_id> \
    sleep 10

Microsoft Teams

Thanks to @noklam, you can also use Microsoft Teams to get notifications. You'll have to get your Team Channel webhook URL.

Python

from lemiknow import teams_sender

@teams_sender(token="<webhook_url_to_your_teams_channel>")
def train_your_nicest_model(your_nicest_parameters):
    import time
    time.sleep(10)
    return {'loss': 0.9} # Optional return value

Command-line

lemiknow teams \
    --webhook-url <webhook_url_to_your_teams_channel> \
    sleep 10

You can also specify an optional argument to tag specific people: user_mentions=[<your_teams_id>, <grandma's_teams_id>].

Text Message (SMS)

Thanks to @abhishekkrthakur, you can use Twilio to send text message notifications. You'll have to setup a Twilio account here, which is paid service with competitive prices: for instance in the US, getting a new number and sending one text message through this service respectively cost $1.00 and $0.0075. You'll need to get (a) a phone number, (b) your account SID and (c) your authentification token. Some detail here.

Python

from lemiknow import sms_sender

ACCOUNT_SID: str = "<your_account_sid>"
AUTH_TOKEN: str = "<your_auth_token>"
@sms_sender(account_sid=ACCOUNT_SID, auth_token=AUTH_TOKEN, recipient_number="<recipient's_number>", sender_number="<sender's_number>")
def train_your_nicest_model(your_nicest_parameters):
    import time
    time.sleep(10)
    return {'loss': 0.9} # Optional return value

Command-line

lemiknow sms \
    --account-sid <your_account_sid> \
    --auth-token <your_account_auth_token> \
    --recipient-number <recipient_number> \
    --sender-number <sender_number>
    sleep 10

Discord

Thanks to @watkinsm, you can also use Discord to get notifications. You'll just have to get your Discord channel's webhook URL.

Python

from lemiknow import discord_sender

webhook_url = "<webhook_url_to_your_discord_channel>"
@discord_sender(webhook_url=webhook_url)
def train_your_nicest_model(your_nicest_parameters):
    import time
    time.sleep(10000)
    return {'loss': 0.9} # Optional return value

Command-line

lemiknow discord \
    --webhook-url <webhook_url_to_your_discord_channel> \
    sleep 10

Desktop Notification

You can also get notified from a desktop notification. It is currently only available for MacOS.

Python

from lemiknow import desktop_sender

@desktop_sender(title="lemiknow Desktop Notifier")
def train_your_nicest_model(your_nicest_parameters):
    import time
    time.sleep(10000)
    return {"loss": 0.9}

Command Line

lemiknow desktop \
    --title 'lemiknow Desktop Notifier' \
    sleep 2

Matrix

Thanks to @jcklie, you can send notifications via Matrix. The homeserver is the server on which your user that will send messages is registered. Do not forget the schema for the URL (http or https). You'll have to get the access token for a bot or your own user. The easiest way to obtain it is to look into Riot looking in the riot settings, Help & About, down the bottom is: Access Token:<click to reveal>. You also need to specify a room alias to which messages are sent. To obtain the alias in Riot, create a room you want to use, then open the room settings under Room Addresses and add an alias.

Python

from lemiknow import matrix_sender

HOMESERVER = "<url_to_your_home_server>" # e.g. https://matrix.org
TOKEN = "<your_auth_token>"              # e.g. WiTyGizlr8ntvBXdFfZLctyY
ROOM = "<room_alias"                     # e.g. #lemiknow:matrix.org

@matrix_sender(homeserver=HOMESERVER, token=TOKEN, room=ROOM)
def train_your_nicest_model(your_nicest_parameters):
    import time
    time.sleep(10000)
    return {'loss': 0.9} # Optional return value

Command-line

lemiknow matrix \
    --homeserver <homeserver> \
    --token <token> \
    --room <room> \
    sleep 10

Note on distributed training

When using distributed training, a GPU is bound to its process using the local rank variable. Since lemiknow works at the process level, if you are using 8 GPUs, you would get 8 notifications at the beginning and 8 notifications at the end... To circumvent that, except for errors, only the master process is allowed to send notifications so that you receive only one notification at the beginning and one notification at the end.

Note: In PyTorch, the launch of torch.distributed.launch sets up a RANK environment variable for each process (see here). This is used to detect the master process, and for now, the only simple way I came up with. Unfortunately, this is not intended to be general for all platforms but I would happily discuss smarter/better ways to handle distributed training in an issue/PR.