A wrapper to run and monitor absl app.
Project description
ABSL-Extra
A collection of utils I commonly use for running my experiments. It will:
- Notify on execution start, finish or failed.
- By default, Notifier will just log those out to
stdout
. - I prefer receiving those in Slack, though (see example below).
- By default, Notifier will just log those out to
- Log parsed CLI flags from
absl.flags.FLAGS
and config values fromconfig_file:get_config()
- Inject
ml_collections.ConfigDict
fromconfig_file
, if kwarg provided. - Inject
pymongo.collection.Collection
ifmongo_config
kwarg provided.
Minimal example
import os
from pymongo.collection import Collection
from ml_collections import ConfigDict
from absl import logging
import tensorflow as tf
from absl_extra import MongoConfig, register_task, run
from absl_extra.notifier import SlackNotifier
from absl_extra.tf_utils import requires_gpu, supports_mixed_precision, make_gpu_strategy
@register_task
@requires_gpu
def main(cmd: str, config: ConfigDict, db: Collection) -> None:
if supports_mixed_precision():
tf.keras.mixed_precision.set_global_policy("mixed_float16")
with make_gpu_strategy().scope():
logging.info("Doing some heavy lifting...")
if __name__ == "__main__":
run(
config_file="config.py",
mongo_config=MongoConfig(uri=os.environ["MONGO_URI"], db_name="my_project", collection="experiment_1"),
notifier=SlackNotifier(slack_token=os.environ["SLACK_BOT_TOKEN"], channel_id=os.environ["CHANNEL_ID"]),
)
Planned for:
- global app state for different tasks
- list of pre/post hooks
- keras callback with notifier
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
absl_extra-0.0.2.dev9.tar.gz
(6.6 kB
view hashes)
Built Distribution
Close
Hashes for absl_extra-0.0.2.dev9-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ef2b1acfb6b80c5ab8f57749ccf002747c5cec3c23be8473ac0a45c5516524d7 |
|
MD5 | ba419d1fb1f0e025bd1d51e5f3255bac |
|
BLAKE2b-256 | 80077f75f7e814fe90a03e47472800d83d1d55d6fa9c4b59cf8148642cd7f13d |