yappi 0.54

Yet Another Python Profiler

Motivation:

CPython standart distribution is coming with three profilers. cProfile, Profile and hotshot. cProfile module is implemented as a C module based on lsprof, Profile is in pure Python and the hotshot can be seen as a small subset of a cProfile. The motivation to implement a new profiler is that all of these profilers lacks the support of multi-threaded programs. If you want to profile a multi-threaded application, you must give an entry point to these profilers and then maybe merge the outputs. None of these profilers is designed to work on long-running multi-threaded application. While implementing a game server, it turns out that is is impossible to profile an application retrieve the statistics then stop and then start later on on the fly(without affecting the profiled application). With the experience of implementing a game server in Python, we have identified most of the problems, tricky parts regarding profiler usage and so, we have come up with simple but powerful requirements.

Requirements:

• Profiler should be started/stopped at any time from any thread in the application.

• Profile statistics should be obtained from any thread at any time.

• Profile events themselves should work theoretically in O(1) time and O(n+m) space(where m is recursion count) per any profile event. This means: application behavior/details/size should not affect the profiler run-time.

• “Profiler pollution”(effect on the application run-time) should be very minimal and observable via statistics to see how much load the profiler is putting on the application. Best way to compute this pollution is to compute the percentage of total run time spent in profiler to the total application run-time.(these statistics shall also include space requirements per application)

• Profiler shall handle the error/exception situations internally. For example running out of memory should be handled by the profiler itself. No OS/Python VM exception that occurred inside profiler shall not affect the profiled application.