Skip to main content

Utilities for processing Recorder traces

Project description

recorder-viz

This is a python package which contains tools for processing Recorder traces.

Installation

pip install recorder-viz

Visualization

We relie on a few libraries for visualization. Please install them first.

Dependencies: pandas, bokeh and prettytable.

import recorder_viz
from recorder_viz import RecorderReader

reader = RecorderReader("path/to/Recorder-traces-folder")
recorder_viz.generate_report(reader, "output.html")

The generate_report API will write out a HTML format visualization report.

The visualization process can be slow for large traces. We recommend to use it only for small scale runs, e.g., < 128 ranks runs.

Below are some example graphs generated from the FLASH traces. example graphs

Advanced Usages

The RecorderReader class contains all infomration about the Recorder traces.

class RecorderReader:
    self.GM: instance of GlobalMetadata
    self.LMs: list of LocalMetadata objects, one for each rank
    self.records: self.records[i] is a list of Record objects of rank i.

GlobalMetadta, LocalMetadata and Record are three Python wrappers of C structures.

class LocalMetadata(Structure):
    self.total_records: int
    self.num_files: int
    self.filemap: set()
    self.function_count: []


class RecorderMetadata(Structure):
    _fields_ = [
            ("total_ranks", c_int),
            ("start_ts", c_double),
            ("time_resolution", c_double),
            ("ts_buffer_elements", c_int),
            ("ts_compression_algo", c_int),
    ]

class Record(Structure):
    _fields_ = [
            ("tstart", c_double),
            ("tend", c_double),
            ("level", c_ubyte),
            ("func_id", c_ubyte),
            ("tid", c_int),
            ("arg_count", c_ubyte),
            ("args", POINTER(c_char_p)),
    ]

Here's an example on how to use the provided classes.

from recorder_viz import RecorderReader

reader = RecorderReader("path/to/Recorder-traces-folder")

for rank in range(reader.GM.total_ranks):
    LM = reader.LMs[rank]
    print("Rank: %d, Number of trace records: %d" %(rank, LM.total_records))

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for recorder-viz, version 0.4.0
Filename, size File type Python version Upload date Hashes
Filename, size recorder-viz-0.4.0.tar.gz (30.9 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page