Skip to main content

async/await introspection

Project description

Await, What?

Tells you what waits for what in an async/await program.

Python 3.10.0a1

It seems the API was changed in 3.10 and the C extension doesn't compile. I'll investigate...

Alpine

You'll need apk add build-base openssl-dev libffi-dev

2019 Sprint Setup

Comms: https://gitter.im/awaitwhat/community

  • Python 3.9, Python 3.8 (preferred) or Python 3.7
  • Your platform dev tools (compiler, etc).
  • Ensure that python is 3.9 or 3.8 or 3.7
  • Install poetry
  • Install graphviz
  • Clone this repository
  • Look at tests
  • Look at issues
> python --version
Python 3.9.0b4  #🧡
Python 3.8.4    #👌

> dot -V
dot - graphviz version 2.40.1

> curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python
# add ~/.poetry/bin to your PATH

> git clone git@github.com:dimaqq/awaitwhat.git
> cd awaitwhat
~/awaitwhat (dev|✔) > poetry shell    # creates a venv and drops you in it

(awaitwhat-x-py3.9) ~/awaitwhat (dev|✔) > poetry install  # installs projects dependencies in a venv
(awaitwhat-x-py3.9) ~/awaitwhat (dev|✔) > poetry build    # builds a C extension in this project

(awaitwhat-x-py3.9) ~/awaitwhat (dev|✔) > env PYTHONPATH=. python examples/test_shield.py | tee graph.dot
(awaitwhat-x-py3.9) ~/awaitwhat (dev|✔) > dot -Tsvg graph.dot -o graph.svg
(awaitwhat-x-py3.9) ~/awaitwhat (dev|✔) > open graph.svg  # or load it in a browser

TL;DR

Say you have this code:

async def job():
    await foo()


async def foo():
    await bar()


async def bar():
    await baz()


async def baz():
    await leaf()


async def leaf():
    await asyncio.sleep(1)  # imagine you don't know this


async def work():
    await asyncio.gather(..., job())

Now that code is stuck and and you want to know why.

Python built-in

Stack for <Task pending coro=<job() > wait_for=<Future pending cb=[<TaskWakeupMethWrapper >()]> cb=[]> (most recent call last):
  File "test/test_stack.py", line 34, in job
    await foo()

This library

Stack for <Task pending coro=<job() > wait_for=<Future pending cb=[<TaskWakeupMethWrapper >()]> cb=[]> (most recent call last):
  File "test/test_stack.py", line 34, in job
    await foo()
  File "test/test_stack.py", line 38, in foo
    await bar()
  File "test/test_stack.py", line 42, in bar
    await baz()
  File "test/test_stack.py", line 46, in baz
    await leaf()
  File "test/test_stack.py", line 50, in leaf
    await asyncio.sleep(1)
  File "/…/asyncio/tasks.py", line 568, in sleep
    return await future
  File "<Sentinel>", line 0, in <_asyncio.FutureIter object at 0x7fb6981690d8>: 

Dependency Graph

References

https://mail.python.org/archives/list/async-sig@python.org/thread/6E2LRVLKYSMGEAZ7OYOYR3PMZUUYSS3K/

Hi group,

I'm recently debugging a long-running asyncio program that appears to get stuck about once a week.

The tools I've discovered so far are:

  • high level: asyncio.all_tasks() + asyncio.Task.get_stack()
  • low level: loop._selector._fd_to_key

What's missing is the middle level, i.e. stack-like linkage of what is waiting for what. For a practical example, consider:

async def leaf(): await somesocket.recv()
async def baz(): await leaf()
async def bar(): await baz()
async def foo(): await bar()
async def job(): await foo()
async def work(): await asyncio.gather(..., job())
async def main(): asyncio.run(work())

The task stack will contain:

  • main and body of work with line number
  • job task with line number pointing to foo

The file descriptor mapping, socket fd, loop._recv() and a Future.

What's missing are connections foo->bar->baz->leaf. That is, I can't tell which task is waiting for what terminal Future.

Is this problem solved in some way that I'm not aware of? Is there a library or external tool for this already?

Perhaps, if I could get a list of all pending coroutines, I could figure out what's wrong.

If no such API exists, I'm thinking of the following:

async def foo():
    await bar()

In [37]: dis.dis(foo)
  1           0 LOAD_GLOBAL              0 (bar)
              2 CALL_FUNCTION            0
              4 GET_AWAITABLE
              6 LOAD_CONST               0 (None)
              8 YIELD_FROM
             10 POP_TOP
             12 LOAD_CONST               0 (None)
             14 RETURN_VALUE

Starting from a pending task, I'd get it's coroutine and:

Get the coroutine frame, and if current instruction is YIELD_FROM, then the reference to the awaitable should be on the top of the stack. If that reference points to a pending coroutine, I'd add that to the "forward trace" and repeat.

At some point I'd reach an awaitable that's not a pending coroutine, which may be: another Task (I already got those), a low-level Future (can be looked up in event loop), an Event (tough luck, shoulda logged all Event's on creation) or a dozen other corner cases.

What do y'all think of this approach?

Thanks, D.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

awaitwhat-21.1.tar.gz (9.3 kB view hashes)

Uploaded Source

Built Distribution

awaitwhat-21.1-cp39-cp39-macosx_10_16_x86_64.whl (17.8 kB view hashes)

Uploaded CPython 3.9 macOS 10.16+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page