Yet Another Python Profiler
Project description
Yappi
A tracing profiler that is thread&coroutine&greenlet aware.
Highlights
- Fast: Yappi is fast. It is completely written in C and lots of love&care went into making it fast.
- Unique: Yappi supports multithreaded, asyncio and gevent profiling. Tagging/filtering multiple profiler results has interesting use cases.
- Intuitive: Profiler can be started/stopped and results can be obtained from any time and any thread.
- Standards Compliant: Profiler results can be saved in callgrind or pstat formats.
- Rich in Feature set: Profiler results can show either Wall Time or actual CPU Time and can be aggregated from different sessions. Various flags are defined for filtering and sorting profiler results.
- Robust: Yappi had seen years of production usage.
Motivation
CPython standard distribution comes with three deterministic profilers. cProfile
, Profile
and hotshot
. cProfile
is implemented as a C module based on lsprof
, Profile
is in pure Python and hotshot
can be seen as a small subset of a cProfile. The major issue is that all of these profilers lack support for multi-threaded programs and CPU time.
If you want to profile a multi-threaded application, you must give an entry point to these profilers and then maybe merge the outputs. None of these profilers are designed to work on long-running multi-threaded applications. It is also not possible to profile an application that start/stop/retrieve traces on the fly with these profilers.
Now fast forwarding to 2019: With the latest improvements on asyncio
library and asynchronous frameworks, most of the current profilers lacks the ability to show correct wall/cpu time or even call count information per-coroutine. Thus we need a different kind of approach to profile asynchronous code. Yappi, with v1.2 introduces the concept of coroutine profiling
. With coroutine-profiling
, you should be able to profile correct wall/cpu time and call count of your coroutine. (including the time spent in context switches, too). You can see details here.
Installation
Can be installed via PyPI
$ pip install yappi
OR from the source directly.
$ pip install git+https://github.com/sumerc/yappi#egg=yappi
Examples
A simple example:
import yappi
def a():
for _ in range(10000000): # do something CPU heavy
pass
yappi.set_clock_type("cpu") # Use set_clock_type("wall") for wall time
yappi.start()
a()
yappi.get_func_stats().print_all()
yappi.get_thread_stats().print_all()
'''
Clock type: CPU
Ordered by: totaltime, desc
name ncall tsub ttot tavg
doc.py:5 a 1 0.117907 0.117907 0.117907
name id tid ttot scnt
_MainThread 0 139867147315008 0.118297 1
'''
Profile a multithreaded application:
You can profile a multithreaded application via Yappi and can easily retrieve
per-thread profile information by filtering on ctx_id
with get_func_stats
API.
import yappi
import time
import threading
_NTHREAD = 3
def _work(n):
time.sleep(n * 0.1)
yappi.start()
threads = []
# generate _NTHREAD threads
for i in range(_NTHREAD):
t = threading.Thread(target=_work, args=(i + 1, ))
t.start()
threads.append(t)
# wait all threads to finish
for t in threads:
t.join()
yappi.stop()
# retrieve thread stats by their thread id (given by yappi)
threads = yappi.get_thread_stats()
for thread in threads:
print(
"Function stats for (%s) (%d)" % (thread.name, thread.id)
) # it is the Thread.__class__.__name__
yappi.get_func_stats(ctx_id=thread.id).print_all()
'''
Function stats for (Thread) (3)
name ncall tsub ttot tavg
..hon3.7/threading.py:859 Thread.run 1 0.000017 0.000062 0.000062
doc3.py:8 _work 1 0.000012 0.000045 0.000045
Function stats for (Thread) (2)
name ncall tsub ttot tavg
..hon3.7/threading.py:859 Thread.run 1 0.000017 0.000065 0.000065
doc3.py:8 _work 1 0.000010 0.000048 0.000048
Function stats for (Thread) (1)
name ncall tsub ttot tavg
..hon3.7/threading.py:859 Thread.run 1 0.000010 0.000043 0.000043
doc3.py:8 _work 1 0.000006 0.000033 0.000033
'''
Different ways to filter/sort stats:
You can use filter_callback
on get_func_stats
API to filter on functions, modules
or whatever available in YFuncStat
object.
import package_a
import yappi
import sys
def a():
pass
def b():
pass
yappi.start()
a()
b()
package_a.a()
yappi.stop()
# filter by module object
current_module = sys.modules[__name__]
stats = yappi.get_func_stats(
filter_callback=lambda x: yappi.module_matches(x, [current_module])
) # x is a yappi.YFuncStat object
stats.sort("name", "desc").print_all()
'''
Clock type: CPU
Ordered by: name, desc
name ncall tsub ttot tavg
doc2.py:10 b 1 0.000001 0.000001 0.000001
doc2.py:6 a 1 0.000001 0.000001 0.000001
'''
# filter by function object
stats = yappi.get_func_stats(
filter_callback=lambda x: yappi.func_matches(x, [a, b])
).print_all()
'''
name ncall tsub ttot tavg
doc2.py:6 a 1 0.000001 0.000001 0.000001
doc2.py:10 b 1 0.000001 0.000001 0.000001
'''
# filter by module name
stats = yappi.get_func_stats(filter_callback=lambda x: 'package_a' in x.module
).print_all()
'''
name ncall tsub ttot tavg
package_a/__init__.py:1 a 1 0.000001 0.000001 0.000001
'''
# filter by function name
stats = yappi.get_func_stats(filter_callback=lambda x: 'a' in x.name
).print_all()
'''
name ncall tsub ttot tavg
doc2.py:6 a 1 0.000001 0.000001 0.000001
package_a/__init__.py:1 a 1 0.000001 0.000001 0.000001
'''
Profile an asyncio application:
You can see that coroutine wall-time's are correctly profiled.
import asyncio
import yappi
async def foo():
await asyncio.sleep(1.0)
await baz()
await asyncio.sleep(0.5)
async def bar():
await asyncio.sleep(2.0)
async def baz():
await asyncio.sleep(1.0)
yappi.set_clock_type("WALL")
with yappi.run():
asyncio.run(foo())
asyncio.run(bar())
yappi.get_func_stats().print_all()
'''
Clock type: WALL
Ordered by: totaltime, desc
name ncall tsub ttot tavg
doc4.py:5 foo 1 0.000030 2.503808 2.503808
doc4.py:11 bar 1 0.000012 2.002492 2.002492
doc4.py:15 baz 1 0.000013 1.001397 1.001397
'''
Profile a gevent application:
You can use yappi to profile greenlet applications now!
import yappi
from greenlet import greenlet
import time
class GreenletA(greenlet):
def run(self):
time.sleep(1)
yappi.set_context_backend("greenlet")
yappi.set_clock_type("wall")
yappi.start(builtins=True)
a = GreenletA()
a.switch()
yappi.stop()
yappi.get_func_stats().print_all()
'''
name ncall tsub ttot tavg
tests/test_random.py:6 GreenletA.run 1 0.000007 1.000494 1.000494
time.sleep 1 1.000487 1.000487 1.000487
'''
Documentation
-
Coroutine Profiling (new in 1.2)
-
Greenlet Profiling (new in 1.3)
Note: Yes. I know I should be moving docs to readthedocs.io. Stay tuned!
Related Talks
Special thanks to A.Jesse Jiryu Davis:
PyCharm Integration
Yappi is the default profiler in PyCharm
. If you have Yappi installed, PyCharm
will use it. See the official documentation for more details.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for yappi-1.3.6-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f583403948513583886f479d7a3886e96b135650c516308d7011af2e60829539 |
|
MD5 | 740a68ddabc32cd3db462390e42a0582 |
|
BLAKE2b-256 | effcb37190fd4864f4da35f28784543f0ccfe600f5414fe8800837540458b2d4 |
Hashes for yappi-1.3.6-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6bd470bdf49c26fdeecf85336b70afd194382834ae52f6dab972b57de8f86f6c |
|
MD5 | 1e13f3a45dd48b12f1df3782075b7fe4 |
|
BLAKE2b-256 | 28ca3cf901e3cdd7df21fcb7bf2da8a87aec47d431bff3d153983d6b5aaf573b |
Hashes for yappi-1.3.6-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fac50de9a9edf8566e571e44cfb646ae57defd81b9ea807a94b939cbd895e4ab |
|
MD5 | d98b6d404bab2531a0f8e3e110def07a |
|
BLAKE2b-256 | 9d5997c969064d71ed420184d6d46a3f500d969b0c845d00738170a8ad26cce9 |
Hashes for yappi-1.3.6-cp310-cp310-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0cd3d5d0ad6c4172b9de5a67af80f0320e31700e020e0ce91bbec8e7ca558c8c |
|
MD5 | 6cf17f699ffe043695f1b5e0fe2c9811 |
|
BLAKE2b-256 | a9956ba8943e3ec56bc20af5d1df0b79b43bb5200484d947ffe1e49f62743e1c |
Hashes for yappi-1.3.6-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a24e30018ee2aec0e53a4edd7847d726ab5d2c2152d8eaa014739dca355dbe83 |
|
MD5 | 923cf0a108b38850a2ec6ceefe64bd56 |
|
BLAKE2b-256 | 746913e16fabc6f24ed82f46ce2b7895725b493726293e5e6ff3d0876cef020c |
Hashes for yappi-1.3.6-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8819c2b8dc094edf2e3f1a7ef424a628a0547f42b1bc536a43dd223b7753c86c |
|
MD5 | 64485efa84dcb7516e39794dca375ec0 |
|
BLAKE2b-256 | 63b9ee7a233f275509e1ffcedaba08cef19e35b294968a4d32522e9df590aa7b |
Hashes for yappi-1.3.6-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84241b3a5022e721b1595582097aa7f3fa0b1a560dc75bc6d9c6fe535b964971 |
|
MD5 | 1707b5efd7812ee7d26457ee729d1965 |
|
BLAKE2b-256 | 67eddc14f476372fd71dcc7c4d77a67e54366c51d415e42b5c751cf30156906f |
Hashes for yappi-1.3.6-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bcef9571559729a76d2245d0fc406a4fad60d32bfa4e4e5e124096e27e98922b |
|
MD5 | 0530a813a88c33f6a3d9529f1d183a3f |
|
BLAKE2b-256 | 3d061e8e1fd58cf69c3de3043010f06bcd96d490fce9fbbfa3f88f2a06d16ce4 |
Hashes for yappi-1.3.6-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8bfd6f38768a84f6c12af6087a6aec0d1fec2d9c0fc65136cf074ef494b25811 |
|
MD5 | 78db7464f6d5a9e62efbc384bffb2ace |
|
BLAKE2b-256 | 23216ff00e70b4fba4dd1a89fc25da3b88ab0f997b85e9d6ba6a88a52e53f30c |
Hashes for yappi-1.3.6-cp39-cp39-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e51750f86defe1f9690f2cdae759768ad536356062c47d22edfab889fb12a041 |
|
MD5 | 7b2f836511c08a1d6302b63849a8898d |
|
BLAKE2b-256 | 7881525d4d6bfb275dbce4526cf1faf65e870d32518dc9f766a8ac789eef15b0 |
Hashes for yappi-1.3.6-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f048e86e426b84357c86975c195598a7412d550c3df5f9b904ea00c57fe2548 |
|
MD5 | 49b4fb87b209d290f814b3f31b5a6c9b |
|
BLAKE2b-256 | 05b0f8e5e181aa59b52b372dd7d7fbea501a9654748b65825b0a4e77fbba7216 |
Hashes for yappi-1.3.6-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d3bb721d956bf62ffe7d8f5a37037c8bd10c9217d9bd783641367341ca57e290 |
|
MD5 | 8e65d6355a4a5eb9baf158853100147b |
|
BLAKE2b-256 | 74c1149561a0ce15c00582663c62e2d4e973c27a84dcbaa53c1522d055a6788d |
Hashes for yappi-1.3.6-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29815ea0107f25ac1be75c44515c66a5fd6ebd7131c640003292c8226c919b77 |
|
MD5 | c7b1df48b7e43cd3b74d5b1a0e7adcfc |
|
BLAKE2b-256 | f18e14818c051571fab9c81c9ba4797cd9a0a2ec9c594cb9d2de05e8ed7e3e73 |
Hashes for yappi-1.3.6-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07c50013b2435f53ea34dd35e74db4a172e52ae6b9a866d79d22805c9fdf65c4 |
|
MD5 | ff86dbc6c66229927dc1bd85f6c3ed57 |
|
BLAKE2b-256 | f55bf10220e142516b2545f1d1ba1b0deb9b9181d9af42b9ae6cdcfa77a1967b |
Hashes for yappi-1.3.6-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1b712ea0d38ffbb985463ea644cbce210fe9f84f47180470c7981764838db7b1 |
|
MD5 | d8f96f8c8110798452b4b4bbcc403ff4 |
|
BLAKE2b-256 | b0f04c8fc204d7bb0ecf6dda90db868237dce9b28cd9cbe3d1b226efb0330881 |
Hashes for yappi-1.3.6-cp38-cp38-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c2d954a63c1806c906fefb2304902c71bfeeb76ec30af53cc164cb747c9e64ec |
|
MD5 | 67aac31aa5d1a74889d679066d0262b4 |
|
BLAKE2b-256 | 94b9a0886972940524fe015e2fdb9b738ed209a654a46e48ccf06cbe45bf25c9 |
Hashes for yappi-1.3.6-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c7bf348067bb54f9cbb47a395b7be10fa6e42c214304668fc822da8998be46b4 |
|
MD5 | 9d8fa3258e9da0d13aae5135f956563e |
|
BLAKE2b-256 | 05a9e4edf0e276b42b8258a54dace6f09463b9685be5935f0bc96f706368a329 |
Hashes for yappi-1.3.6-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e55f3f7a7be76d70f60f71530ad0db83afd1284bf4a6fa8d69af56a1533f624 |
|
MD5 | e3135d85269e33396ab72d8d9013590e |
|
BLAKE2b-256 | cb8521e4a4d265463ed1c4354c632871a45cf69d6f8e4648c31a7f786aa3c203 |
Hashes for yappi-1.3.6-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8efb1630e1a8ebaeee5dcd08f58a1397a98e91241dde1e3c5ffd89bc1aa293b2 |
|
MD5 | 6a7fe25413bcee5eb018e6339bea5fed |
|
BLAKE2b-256 | 087472b5693182a063403d75ffc0141f651869e8dd2c71237a3806f1641afe4f |
Hashes for yappi-1.3.6-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b639a0788a8d8fd566a5a03bd2ce769acc42ddee3256287d57adb76dde7134c8 |
|
MD5 | 6078518b481e3ec249ebe8f3eab7c701 |
|
BLAKE2b-256 | a5548cc898c1a30b67209f4f2cebb0e01d300517b21e64e5d1c9132d4a55020d |
Hashes for yappi-1.3.6-cp37-cp37m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5aac25c46acd8f320fc19b4a6f86c966a106c9ad5736e436ed3d99ae436d13a3 |
|
MD5 | 4009aca596a65a4a68433fafe22ccab7 |
|
BLAKE2b-256 | 6742a859bacac27b9530b073f4512c6ce1a325e23a1e4a27bf27d45be6d960b6 |
Hashes for yappi-1.3.6-cp37-cp37m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b0916e90cbfe17c6e8dd82d7c04f7d628a3a09f15f1d28fd239096daa7347cca |
|
MD5 | 093c5d0d066d05241f370e8a1aa89ab7 |
|
BLAKE2b-256 | 291b264c26ea73332059787678223d27c33d7b4564bb9973408ea0ac0354084c |
Hashes for yappi-1.3.6-cp37-cp37m-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fbca637b111d4a3085376c80e9b671ecbad420574f942aa10fd93cbfa804d36f |
|
MD5 | 5483c473d1c5975669513848e055dc6f |
|
BLAKE2b-256 | 0a79ba0bbadd61347d07f5caf509c23eaf8767f8dea7b873f32df9b61f83f485 |
Hashes for yappi-1.3.6-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bfbeaca9e1a41cb1eabe1b7897b8fb236e85889c502654bded7c6628bd7bb2f7 |
|
MD5 | d968121b61a38d024542eddb791e51e9 |
|
BLAKE2b-256 | eb00ddb59004c522b7116c2d35ac2d9e71d2a3d3c15b259ef0fef3f4f29501e6 |
Hashes for yappi-1.3.6-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d5aefd0728e80cae3cb325a67d413b90cd998ea97f274ade0ff45279ca40a8d6 |
|
MD5 | 41ad3fd4def8fbe03b91f8ea82718228 |
|
BLAKE2b-256 | a65763a17a2cc3ef28e7b704d8c94d8ce3a8c13b49b92590cf91958a64f6b405 |
Hashes for yappi-1.3.6-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 95b8ee6b8718527e480be6366fd86b6bbac51ff8ae3f5dd2a54be60ab28c7e65 |
|
MD5 | 32c0e02361dae928e6b69e4670cc38b3 |
|
BLAKE2b-256 | 1621e8f831ac9c0cbd70a4567416da050ab9c214a2fe7b1840fd201739f7a706 |
Hashes for yappi-1.3.6-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8482861bcd9b7530c017067d8f6a21a9c1d26ca787dc5e6b716eb57314af449a |
|
MD5 | d92209f54bda11739319c51fa58c004b |
|
BLAKE2b-256 | 84a99f592d73e9679b51c974625bd18c63e7e43a10af9bb8aaf1471963a181b6 |
Hashes for yappi-1.3.6-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | de5c84fc12652578dda9c8acabb985c672a1f0c7e0f27e974f0ce991610d0049 |
|
MD5 | 5d0575e1511f15f8079632d6c9167467 |
|
BLAKE2b-256 | ea4d9466c7a9a8bca2653e1315b8de4fba7885a86082706b1b1eb0f303286b95 |
Hashes for yappi-1.3.6-cp36-cp36m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 57bc0d2fc86cf5ade040042f193dd04df67646d3416317625efcee78e13c6fc7 |
|
MD5 | 2773d5d401b2b16d147459590611eefc |
|
BLAKE2b-256 | 37f9ea3254d7f3e4112e941c1e5f724ed8c3f64b4522dedb3ee30eeea0711fd8 |
Hashes for yappi-1.3.6-cp36-cp36m-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba5b59978795f77ee337a03a16f6685d914390c7857375e1b98746e75ec9ae6f |
|
MD5 | 7f7e9dd9b288bf0ee0420d9145082b8c |
|
BLAKE2b-256 | 28dd5cd65a29eb903886cc06b1974b285f8589436223ca83b35984c042075196 |
Hashes for yappi-1.3.6-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 97d1edc0cf2212d0ebfffd06fd7aee0d17d280216e8f4fe320eb0436187f14e5 |
|
MD5 | deb6b81f45e09c435edd8f8f8a303659 |
|
BLAKE2b-256 | 51aab9e778c56767be1ec40b60ef9af90398bdd70ace1be52494c07655b6f4b2 |
Hashes for yappi-1.3.6-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22c567a6983f76317e128a0e669be7afb3e7c02654b8e4d76c3b4c3acef008e9 |
|
MD5 | fa9f54005a88718e69ba5eca423bc5b4 |
|
BLAKE2b-256 | a4a19bdeec7c3442c56231822e29a1bf3cb9412c39eef1ae67f5d70b683dff9a |
Hashes for yappi-1.3.6-cp36-cp36m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e159395a0a7cca4963de96155928e3f682bfbea40325bf33a22f422aa0c8a13c |
|
MD5 | 6a5e8348871cd6f97cd18dc2f0c81aac |
|
BLAKE2b-256 | 24a4a853f5c05700b55d37a2d4a944eaa1f7002c55825b59d193a8120f8458bb |