Skip to main content

Monitoring utility for machine learning experiments

Project description

Travis status for master branch codecov.io status for master branch https://img.shields.io/pypi/pyversions/rarog.svg

Rarog is a monitoring utility for machine learning experiments. You may use it as a light-weight alternative for TensorBoard or Visdom. Rarog stores all records in ClickHouse database using ClickHouse Python Driver.

Features

  • log common python data types(bool, int, float, string, iterables)
  • log 1d numpy arrays
  • distributed experiments monitoring

Setup

Install via pip:

pip install rarog

Start ClickHouse database if required:

docker run -d --name clickhouse --ulimit nofile=262144:262144 -p 9000:9000 yandex/clickhouse-server

Important note: the example above is just the easiest way. For production, you should setup database backups or replicated.

Rarog supports Python 3.4 and newer.

Usage

import random

from rarog import Tracker

tracker = Tracker(name='experiment_name')

# trace values one by one
for step in range(10):
    tracker.trace(
        name='int_value',
        value=random.randint(10, 20),
        step=step)
    tracker.trace(
        name='float_value',
        value=random.random(),
        step=step)
    # provide experiment phase as a string
    tracker.trace(
        name='list_value',
        value=[random.random(), random.random()],
        step=step,
        phase='val')

# trace values by dict
for step in range(10, 20):
    tracker.multy_trace({
        'int_value': random.randint(10, 20),
        'float_value': random.random()
    }, step=step)

# get names of traced metrics
tracker.metrics
# Out: ['time', 'step', 'phase', 'int_value', 'float_value', 'list_value']

If you are going to record more than 100 entries per second, it’s better to use sync_step or sync_seconds arguments. Thus writing to the database will be done with some period, which is much faster.

# `exist_ok` flag allow to use the same name for experiment
step_tracker = Tracker(name='experiment_name', sync_step=1000, exist_ok=True)

for step in range(20, 10**4):
    step_tracker.trace(name='bool_value', value=bool(random.randint(0, 1)), step=step)
    step_tracker.multy_trace({
        'int_value': random.randint(10, 20),
        'float_value': random.random()
    }, step=step)

# tracker should be manually synchronized after last entry
step_tracker.sync_accumulated_values()

Experiments can be handled via manager

from rarog import Manager

manager = Manager()
manager.list_experiments()
# Out: ['experiment_name']

manager.remove_experiment('experiment_name')

Retrieving your data

TODO (manually and with visualization)

TODO

  • Pytorch tensors support
  • Support 2d arrays
  • Tensorflow data types support
  • Split Aggregation View for summarization and underlying tables
  • Store experiments metadata(config, author, etc.)
  • Autodocs

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for rarog, version 0.1.dev1
Filename, size File type Python version Upload date Hashes
Filename, size rarog-0.1.dev1.tar.gz (5.9 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page