polyaxon-helper 0.5.0

A python helper library to report metrics and communicate with Polyaxon.

Polyaxon-helper

Polyaxon helper is a lightweight python library to report metrics and communicate information with Polyaxon.

Installation

$ pip install -U polyaxon-helper

for python3

$ pip3 install -U polyaxon-helper

Install polyaxon

Please check polyaxon installation guide

Install in polyaxonfile

If you want to delegate the installation to polyaxon during the build process, add a new step to the run section in your polyaxonfile:

...
build:
  image: ...
  build_steps:
    - ...
    - pip install -U polyaxon-helper
    - ...

run:
  cmd: ...

Usage

Getting env variables defined by Polyaxon

from polyaxon_helper import (
    get_cluster_def,
    get_declarations,
    get_experiment_info,
    get_task_info,
    get_tf_config,
    get_job_info,
    get_outputs_path,
    get_outputs_refs_paths,
    get_data_paths,
    get_log_level
)

cluster_def = get_cluster_def()
declarations = get_declarations()
experiment_info = get_experiment_info()
task_info = get_task_info()
job_info = get_job_info()
outputs_path = get_outputs_path()
outputs_refs_paths = get_outputs_refs_paths()
data_paths = get_data_paths()
tf_config = get_tf_config()
log_level = get_log_level()

• get_cluster_def: Returns cluster definition created by polyaxon.

  {
      "master": ["plxjob-master0-8eefb7a1146f476ca66e3bee9b88c1de:2000"],
      "worker": ["plxjob-worker1-8eefb7a1146f476ca66e3bee9b88c1de:2000",
                 "plxjob-worker2-8eefb7a1146f476ca66e3bee9b88c1de:2000"],
      "ps": ["plxjob-ps3-8eefb7a1146f476ca66e3bee9b88c1de:2000"],
  }
  

• get_declarations: Returns all the experiment declarations based on both,

  • declarations section
  • matrix section

• get_tf_config: Returns the TF_CONFIG defining the cluster and the current task. if envvar is not null, it will set and env variable with envvar.

• get_experiment_info: Returns information about the experiment.

  • project_name
  • experiment_group_name
  • experiment_name
  • project_uuid
  • experiment_group_uuid
  • experiment_uuid

• get_job_info: Returns information about the job.

  • project_name
  • job_name
  • project_uuid
  • job_uuid

• get_task_info: Returns the task info: {"type": str, "index": int}.

• outputs_path: The outputs path generated by polyaxon based on the hierarchy of the experiment.

   `user/project/group/experiment/files`
  

• data_paths: The data paths mounted for the job/experiment:

   {
     'data1': '/data/1',
     'data-foo': '/data/foo',
   }
  

• outputs_path: The references outputs paths requested by the user, the order follows the order specified by the user:

   {
       'experiments': [
           `user/project/job12/files`,
       ], 'experiments': [
           `user/project/group/experiment1/files`,
           `user/project/experiment100/files`
       ]
   }
  

• get_log_level: If set on the polyaxonfile it will return the log level.

Reporting metrics to Polyaxon

In order to report metrics for an experiment, just add these lines in you program.

from polyaxon_helper import send_metrics

send_metrics(accuracy=0.9, precision=0.95)

Quick start

Please check our quick start guide to start training your first experiment.

License