Skip to main content

tensorflow meta-framework

Project description

IceFlow
=======

_ice floe, nowhere to go_

A lightweight meta-framework for training neural networks with [TensorFlow](https://www.tensorflow.org/).

Installation
------------

pip install iceflow

### Dependencies

- `tensorflow>=1.3.0`
- `dm-sonnet>=1.11`

Quick start
-----------

1. Define [Sonnet modules](https://deepmind.github.io/sonnet/) in `models.py`:

import tensorflow as tf
import sonnet as snt


class MLP(snt.AbstractModule):
def __init__(self, hidden_size, output_size, nonlinearity=tf.tanh):
super(MLP, self).__init__()
self._hidden_size = hidden_size
self._output_size = output_size
self._nonlinearity = nonlinearity

def _build(self, inputs):
lin_x_to_h = snt.Linear(output_size=self._hidden_size, name="x_to_h")
lin_h_to_o = snt.Linear(output_size=self._output_size, name="h_to_o")
return lin_h_to_o(self._nonlinearity(lin_x_to_h(inputs)))


2. Define [Datasets](https://www.tensorflow.org/programmers_guide/datasets)
in `datasets.py`:

from tensorflow.contrib.data import Dataset
from tensorflow.examples.tutorials.mnist import input_data


def mnist():
# load mnist data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)

# make Datasets
train_dataset = Dataset.from_tensor_slices(
(mnist.train._images, mnist.train._labels))
test_dataset = Dataset.from_tensor_slices(
(mnist.test._images, mnist.test._labels))

return train_dataset, test_dataset

3. Describe what you want to do in `test1.cfg`:

[DEFAULT]
model_dir=test1
model=MLP
hidden_size=50
output_size=10

4. Train your model, evaluating every 1000 steps:

$ iceflow train test1.cfg mnist --eval_period 1000

5. Evaluate your model:

$ iceflow eval test1.cfg mnist
{'global_step': 10000, 'loss': 0.13652229, 'accuracy': 0.96079999}

6. Visualize your learning in TensorBoard:

$ tensorboard --logdir=test1

Navigate to <http://localhost:6006> to see the metrics:

![](images/tensorboard.png)

7. Add some new data to `datasets.py`

import numpy as np


def random_image():
return None, Dataset.from_tensors(
np.random.random((784,)).astype(np.float32))


def random_images():
return None, Dataset.from_tensor_slices(
np.random.random((32, 784,)).astype(np.float32))

and make predictions on it

$ iceflow predict test1.cfg random_image
[5]

$ iceflow predict test1.cfg random_images
[5, 5, 5, 5, 3, 5, 5, 5, 5, 5, 3, 5, 5, 5, 5, 5, 3, 5, 5, 5, 5, 5, 3, 3, 5, 5, 5, 5, 5, 5, 3, 5]

Config format reference
-----------------------

The format of the `iceflow` config file is roughly

[DEFAULT]
model_dir=test1
model=MLP
hyperparam_1=50
hyperparam_2=10

[more_hiddens]
model_dir=test2
hyperparam_1=100

To train the model defined in the `[DEFAULT]` section, run

$ iceflow train <config_file> <dataset>

To train the `[more_hiddens]` variant model, which inherits all hyperparameters
from the `[DEFAULT]` section but overrides `model_dir` (to avoid conflicting
with the `[DEFAULT]` model) and `hyperparam_1`, run

$ iceflow train <config_file> <dataset> --config_section more_hiddens

`model` must refer to a Sonnet module defined in `models.py`.

Every key besides `model_dir` and `model` is taken to be a hyperparameter which
will be passed as a kwarg to the constructor of the Sonnet module.

Design philosophy
-----------------

Our typical workload involves training lots of models (usually with complex or
experimental architecture) with different sets of hyperparameters on different
datasets.

Previously, we had been using a hand-built meta-framework around TensorFlow to
organize training, evaluation, and inference.

As of TensorFlow 1.3, the [Dataset API](https://www.tensorflow.org/programmers_guide/datasets),
[Estimator API](https://www.tensorflow.org/programmers_guide/estimators), and
[DeepMind's Sonnet library](https://deepmind.github.io/sonnet/) have arisen as
mature alternatives to our hand-crafted solutions.

IceFlow aims to provide the small bit of code needed to get these three APIs to
work together seamlessly - without sacrificing flexibility - and provide an
efficient "command line and config file"-based interface to the basic train,
eval, predict cycle.

Caveats and future directions
-----------------------------

- Currently, the only supported type of problem is a softmax classification
problem with one-hot labels. We plan to extend this.
- Currently, the only possible output you can obtain from `iceflow predict` is
tensors being printed to the command line. We plan to extend this to allow
specification of an arbitrary Python function that takes the prediction
results (arrays) as input.
- Currently, the optimizer used for training is hard-coded. We plan to expose
this as a parameter either in the config or on the command line. We also plan
to extend this to support learning rate decay and related use cases.
- Currently, there is no easy way to use IceFlow to inject a properly-restored
Estimator into arbitrary Python code. We plan to add this capability.
- Currently, the batch size and shuffle buffer size are not exposed. We plan to
expose this soon.
- Currently, performing validation every so often during training is very
awkward. We are awaiting the return of [`ValidationMonitor`](https://www.tensorflow.org/get_started/monitors#configuring_a_validationmonitor_for_streaming_evaluation)
from its banishment in the desert of deprecation (and following
[this GitHub issue](https://github.com/tensorflow/tensorflow/issues/7669)).


Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

iceflow-0.0.1a2.tar.gz (21.6 kB view hashes)

Uploaded Source

Built Distribution

iceflow-0.0.1a2-py3-none-any.whl (9.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page