netflix-spectator-py-runtime-metrics 1.0.4

Library to collect runtime metrics for Python applications using spectator-py.

spectator-py-runtime-metrics

Library to collect runtime metrics for Python applications using spectator-py.

See the Atlas Documentation site for more details on spectator-py.

Instrumenting Code

from spectator.registry import Registry
from runmetrics.stats_collector import StatsCollector

if __name__ == "__main__":
    registry = Registry()
    # optionally add a stable worker id tag to runtime metrics, for forked processes
    StatsCollector(registry, worker_id=None).start()

Worker IDs

For long-running Python applications, it is common to use a forking process model to increase the ability to handle load. Usually, these rely upon the pid to uniquely identify the workers, but some of these frameworks offer either a stable worker id or the ability to create stable worker id values through the addition of callbacks.

Ideally, the worker id is a string in the range 0..N, with N<16 in most cases. Most process forking frameworks have a configuration setting for the maximum number of workers, which helps to keep these values constrained.

Using raw str(os.getpid()) values as the worker id is not recommended, because it can lead to large increases in metrics volume in the Atlas backend, if processes restart regularly over the lifetime of an instance.

References

• Python
  • gc — Garbage Collector interface
  • multiprocessing — Process-based parallelism
  • os — Miscellaneous operating system interfaces
  • resource — Resource usage information
  • threading — Thread-based parallelism
• Linux
  • getrusage(2) — Linux manual page

Local Development

Install pyenv, possibly with Homebrew, and install a recent Python version.

make setup-venv
make test
make coverage