aiodebug 2.3.0

A tiny library for monitoring and testing asyncio programs

This is a tiny library for monitoring and testing asyncio programs. Its monitoring features are meant to be always on in production.

Installation

aiodebug is only compatible with Python 3.8 and higher. There are no plans to support older versions.

aiodebug is available on PyPI and you can install it with:

pip install aiodebug

or

poetry add aiodebug

aiodebug will use logwood if it is installed, otherwise it will default to the standard logging module.

Log warnings when callbacks block the event loop

import aiodebug.log_slow_callbacks

aiodebug.log_slow_callbacks.enable(0.05)

This will produce WARNING-level logs such as

Executing <Task pending coro=<foo() running at /home/.../foo.py:37>
wait_for=<Future pending cb=[Task._wakeup()]>> took 0.069 seconds

asyncio already does this in debug mode, but you probably don’t want to enable full-on debug mode in production.

Instead of defaulting to the logs, you may provide your own callback that gets called with the name of the slow callback and its execution duration, and can do anything it needs with it. This might be useful e.g. for structured JSON logging.

import aiodebug.log_slow_callbacks

aiodebug.log_slow_callbacks.enable(
        0.05,
        on_slow_callback = lambda task_name, duration: json_logger.warning(
                'Task blocked async loop for too long',
                extra = {'task_name': task_name, 'duration': duration}
        )
)

Track event loop lags in StatsD

import aiodebug.monitor_loop_lag

aiodebug.monitor_loop_lag.enable(statsd_client)

Tracks how much scheduled calls get delayed and sends the lags to StatsD.

Dump stack traces of all threads if the event loop hangs for too long

import aiodebug.hang_inspection

dumper = aiodebug.hang_inspection.start('/path/to/output/directory', interval = 0.25)  # 0.25 is the default
...
await aiodebug.hang_inspection.stop_wait(dumper)

Enabling this function may help you in case one of your threads (sometimes) runs a CPU-bound operation that completely stalls the event loop, but you don’t know which thread it is or what it is doing.

Every time the event loop hangs (doesn’t run a scheduled ‘monitoring’ task) for longer than the given interval, aiodebug will create 3 stack traces, 1 second apart, in your output directory. For example:

-rw-r--r--  1 user  group   6.7K  4 Jan 09:41 stacktrace-20220104-094154.197418-0.txt
-rw-r--r--  1 user  group   7.0K  4 Jan 09:41 stacktrace-20220104-094155.206574-1.txt
-rw-r--r--  1 user  group   6.6K  4 Jan 09:41 stacktrace-20220104-094156.211781-2.txt

Each file then contains the Python stack traces of all threads that were running or waiting at the time. You might be able to find your culprit blocking the event loop at the end of one of the traces.

Speed up or slow down time in the event loop

This is mainly useful for testing.

import aiodebug.testing.time_dilated_loop

loop = aiodebug.testing.time_dilated_loop.TimeDilatedLoop()
asyncio.set_event_loop(loop)

loop.time_dilation = 3
await asyncio.sleep(1)  # Takes 0.333s of real time

loop.time_dilation = 0.1
await asyncio.sleep(1)  # Takes 10s of real time

---

aiodebug was made by Quantlane, a systematic trading firm. We design, build and run our own stock trading platform.