async_generator 1.2

Async generators for Python 3.5

The async_generator library

This is a tiny library to add “async generators” to Python 3.5. What are those?

Option 1: my 5-minute lightning talk demo from PyCon 2016

Option 2: read on!

Python’s iterators are great to use – but manually implementing the iterator protocol (__iter__, __next__) can be very annoying. No-one wants to do that all the time.

Fortunately, Python has generators, which make it easy and straightforward to create an iterator by writing a function. E.g., if you have a file where each line is a JSON document, you can make an iterator over the decoded bodies with:

def load_json_lines(fileobj):
    for line in fileobj:
        yield json.loads(line)

Starting in v3.5, Python has added *async iterators* and *async functions*. These are like regular iterators and functions, except that they have magic powers that let them do asynchronous I/O without twisting your control flow into knots.

Asynchronous I/O code is all about incrementally processing streaming data, so async iterators are super handy. But manually implementing the async iterator protocol (__aiter__, __anext__) can be very annoying, which is why we want async generators, which make it easy to create an async iterator by writing an async function. For example, suppose that in our example above, we want to read the documents from a network connection, instead of the local filesystem. Using the asyncio.StreamReader interface we can write:

async def load_json_lines(asyncio_stream_reader):
    async for line in asyncio_stream_reader:
        yield json.loads(line)

BUT! the above DOESN’T WORK in Python 3.5 – you just get a syntax error. In 3.5, the only way to make an async generator is to manually define __aiter__ and __anext__.

Until now.

This is a little library which implements async generators in Python 3.5, by emulating the above syntax. The two changes are that you have to decorate your async generators with @async_generator, and instead of writing yield x you write await yield_(x):

# Same example as before, but works in Python 3.5
from async_generator import async_generator, yield_, yield_from_

@async_generator
async def load_json_lines(asyncio_stream_reader):
    async for line in asyncio_stream_reader:
        await yield_(json.loads(line))

Semantics

This library generally follows PEP 525 semantics (“as seen in Python 3.6!”), except that it adds yield from support, and it doesn’t currently support the sys.{get,set}_asyncgen_hooks garbage collection API. There are two main reasons for this: (a) it doesn’t exist on Python 3.5, and (b) even on 3.6, only built-in generators are supposed to use that API, and that’s not us. In any case, you probably shouldn’t be relying on garbage collection for async generators – see this discussion and PEP 533 for more details.

aclosing

As discussed above, you should always explicitly call aclose on async generators. To make this more convenient, this library also includes an aclosing async context manager. It acts just like the closing context manager included in the stdlib contextlib module, but does await obj.aclose() instead of obj.close(). Use it like this:

from async_generator import aclosing

async with aclosing(load_json_lines(asyncio_stream_reader)) as agen:
    async for json_obj in agen:
        ...

yield from

Starting in 3.6, CPython has native support for async generators. But, native async generators still don’t support yield from. This library does. It looks like:

@async_generator
async def wrap_load_json_lines(asyncio_stream_reader):
    await yield_from_(load_json_lines(asyncio_stream_reader))

The await yield_from_(...) construction can be applied to any async iterator, including class-based iterators, native async generators, and async generators created using this library, and fully supports the classic yield from semantics.

In fact, if you’re using CPython 3.6 native generators, you can even use this library’s yield_from_ directly inside a native generator. For example, this totally works (if you’re on 3.6):

async def f():
    yield 2
    yield 3

async def g():
    yield 1
    await yield_from_(f())
    yield 4

There are two limitations to watch out for, though:

• You can’t write a native async generator that only contains yield_from_ calls; it has to contain at least one real yield or else the Python compiler won’t know that you’re trying to write an async generator and you’ll get extremely weird results. For example, this won’t work:

  async def wrap_load_json_lines(asyncio_stream_reader):
      await yield_from_(load_json_lines(asyncio_stream_reader))

  The solution is either to convert it into an @async_generator, or else add a yield expression somewhere.

• You can’t return values from native async generators. So this doesn’t work:

  async def yield_and_return():
      yield 1
      yield 2
      # "SyntaxError: 'return' with value in async generator"
      return "all done"
  
  async def wrapper():
      yield "in wrapper"
      result = await yield_from_(yield_and_return())
      assert result == "all done"

  The solution is to convert yield_and_return to an @async_generator:

  @async_generator
  async def yield_and_return():
      await yield_(1)
      await yield_(2)
      return "all done"

Changes

1.2 (2016-11-14)

• Rewrote yield from support; now has much more accurate handling of edge cases.

• yield_from_ now works inside CPython 3.6’s native async generators.

• Added aclosing context manager; it’s pretty trivial, but if we’re going to recommend it be used everywhere then it seems polite to include it.

• 100% test coverage.

1.1 (2016-11-06)

• Support for asend/athrow/aclose

• Support for yield from

• Add a __del__ method that complains about improperly cleaned up async generators.

• Adapt to the change in Python 3.5.2 where __aiter__ should now be a regular method instead of an async method.

• Adapt to Python 3.5.2’s pickiness about iterating over already-exhausted coroutines.

• 100% test coverage.

1.0 (2016-07-03)

• Fixes a very nasty and hard-to-hit bug where await yield_(...) calls could escape out to the top-level coroutine runner and get lost, if the last trap out to the coroutine runner before the await yield_(...) caused an exception to be injected.

• Infinitesimally more efficient due to re-using internal ANextIter objects instead of recreating them on each call to __anext__.

• 100% test coverage.

0.0.1 (2016-05-31)

Initial release.