asyncio-gevent 0.2.3

asyncio & gevent in harmony

asyncio-gevent

asyncio-gevent makes asyncio and gevent compatible. It provides utilities for

• running asyncio on gevent (by using gevent as asyncio's event loop)
• running gevent on asyncio (by using asyncio as gevent's event loop, still work in progress)
• converting greenlets to asyncio futures
• converting futures to asyncio greenlets
• wrapping blocking or spawning functions in coroutines which spawn a greenlet and wait for its completion
• wrapping coroutines in spawning functions which block until the future is resolved

asyncio-gevent is a fork and rewrite of aiogevent and tulipcore for modern python 3.

Install

Install asyncio-gevent from pypi using your favourite package manager.

# If you use poetry
poetry add asyncio-gevent

# If you use pip
pip install asyncio-gevent

Usage

Running asyncio on gevent

In order to run asyncio on gevent, we need to set the (default) EventLoopPolicy to use asyncio_gevent.EventLoopPolicy.

import gevent.monkey
gevent.monkey.patch_all()

import asyncio

import asyncio_gevent

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())

async def main():
    await asyncio.sleep(1)
    print("done")

asyncio.run(main())

After setting the event loop policy, asyncio will use an event loop that uses greenlets for scheduling.

Under the hood, it uses the default selector-based event loop in asyncio with the the gevent selector implementation.

Alternatively, you can also manually set and use the event loop.

import gevent.monkey
gevent.monkey.patch_all()

import asyncio

import asyncio_gevent

loop = asyncio_gevent.EventLoop()
asyncio.set_event_loop(loop)

async def main():
    await asyncio.sleep(1)
    print("done")

loop.run_until_complete(main())

Running gevent on asyncio

This implementation is still work-in-progress. It may work for simple examples, but otherwise fail in unexpected ways.

In order to run gevent on asyncio, gevent needs to be initialised to use the asyncio event loop. This is done by setting the environment variable GEVENT_LOOP to asyncio_gevent.gevent_loop.GeventLoop and then starting python.

GEVENT_LOOP=asyncio_gevent.gevent_loop.GeventLoop python3 myscript.py

gevent will now run on asyncio.

Alternatively, you can also set the loop configuration setting, preferably right after importing gevent and before monkey patching.

import gevent
gevent.config.loop = "asyncio_gevent.gevent_loop.GeventLoop"

Converting greenlets to asyncio futures

Use asyncio_gevent.greenlet_to_future to convert a greenlet to an asyncio future. The future yields once the greenlet has finished execution.

# Preamble: Apply the gevent monkey patch and initialise the asyncio event loop policy

import gevent.monkey
gevent.monkey.patch_all()

import asyncio

import asyncio_gevent

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())

# Main example

import gevent

def blocking_function() -> int:
    gevent.sleep(10)
    return 42


async def main() -> None:
    greenlet = gevent.spawn(blocking_function)
    future = asyncio_gevent.greenlet_to_future(greenlet)
    result = await future

asyncio.run(main())

If the greenlet is already dead when the future is awaited/scheduled, then the future will resolve with the result or raise the exception thrown immediately.

If the greenlet is not yet running, the greenlet will by default be started when the future is awaited/scheduled. This is to ensure a sensible default behaviour and prevent odd concurrency issues. To prevent this auto-starting, you can pass autostart_greenlet=False as an argument to greenlet_to_future.

When a greenlet is killed without a custom exception type, it will return a GreenletExit exception. In this instance, the future get cancelled. If a custom exception type is used, the future will raise the exception.

If the future gets cancelled, then by default the greenlet is killed. To prevent the greenlet from getting killed, you can pass autokill_greenlet=False as an argument to greenlet_to_future.

Converting asyncio futures to greenlets

Use asyncio_gevent.future_to_greenlet to convert a future to a greenlet.

# Preamble: Apply the gevent monkey patch and initialise the asyncio event loop policy

import gevent.monkey
gevent.monkey.patch_all()

import asyncio

import asyncio_gevent

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())

# Main example

import gevent

async def async_function() -> int:
    await asyncio.sleep(1)
    return 42


def main() -> None:
    future = async_function()
    greenlet = asyncio_gevent.future_to_greenlet(future)
    greenlet.start()
    greenlet.join()
    assert greenlet.get() == 42

main()

The greenlet returned by this function will not start automatically, so you need to call Greenlet.start() manually.

If future is a coroutine object, it will be scheduled as a task on the loop when the greenlet starts. If no loop argument has been passed, the running event loop will be used. If there is no running event loop, a new event loop will be started using the current event loop policy.

If the future is not already scheduled, then it won't be scheduled for execution until the greenlet starts running. To prevent the future from being scheduled automatically, you can pass autostart_future=False as an argument to future_to_greenlet.

If the greenlet gets killed, the by default the future gets cancelled. To prevent this from happening and having the future return the GreenletExit objct instead, you can pass autocancel_future=False as an argument to future_to_greenlet.

If the future gets cancelled, the greenlet gets killed and will return a GreenletExit. This default behaviour can be circumvented by passing autokill_greenlet=False and the greenlet will raise the CancelledError instead.

Wrapping blocking or spawning functions in coroutines

Use asyncio_gevent.sync_to_async to wrap a blocking function or a function that may spawn greenlets and wait for their completion in a coroutine.

# Preamble: Apply the gevent monkey patch and initialise the asyncio event loop policy

import gevent.monkey
gevent.monkey.patch_all()

import asyncio

import asyncio_gevent

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())

# Main example

import gevent

def blocking_function(duration: float):
    gevent.sleep(duration)
    return 42

async_function = asyncio_gevent.sync_to_async(blocking_function)

asyncio.run(async_function(1.0))

The returned corountine function will execute the original function in a spawned greenlet and await it's completion.

Under the hood, this is just a thin convenience wrapper around asyncio_gevent.greenlet_to_future.

As a result, asyncio_gevent.sync_to_async accepts the same arguments as asyncio_gevent.greenlet_to_future to achieve the same behaviour.

asyncio_gevent.sync_to_async can also be used as a decorator

# Preamble: Apply the gevent monkey patch and initialise the asyncio event loop policy

import gevent.monkey
gevent.monkey.patch_all()

import asyncio

import asyncio_gevent

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())

# Main example

import gevent

@asyncio_gevent.sync_to_async()
def fn(duration: float):
    gevent.sleep(duration)
    return 42

asyncio.run(fn(1.0))

Wrapping coroutines in spawning functions

Use asyncio_gevent.async_to_sync to wrap a coroutine function or in a blocking function that spawns a greenlet and waits until the coroutine has returned.

# Preamble: Apply the gevent monkey patch and initialise the asyncio event loop policy

import gevent.monkey
gevent.monkey.patch_all()

import asyncio

import asyncio_gevent

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())

# Main example

import gevent

async def async_function(duration: float):
    await asyncio.sleep(duration)
    return 42

blocking_function = asyncio_gevent.async_to_sync(async_function)

blocking_function(1)

The returned function will execute the coroutine on an existing running loop or a new event loop and await it's completion.

Under the hood, this is just a thin convenience wrapper around asyncio_gevent.future_to_greenlet.

As a result, asyncio_gevent.async_to_sync accepts the same arguments as asyncio_gevent.future_to_greenlet to achieve the same behaviour.

asyncio_gevent.async_to_sync can also be used as a decorator.

# Preamble: Apply the gevent monkey patch and initialise the asyncio event loop policy

import gevent.monkey
gevent.monkey.patch_all()

import asyncio

import asyncio_gevent

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())

# Main example

import gevent

@asyncio_gevent.async_to_sync
async def fn(duration: float):
    await asyncio.sleep(duration)
    return 42

fn(1)

Known limitations

gevent.sleep

gevent.sleep will pause the current greenlet. As a result, this will, by default, result in the greenlet powering the asyncio loop to be put to sleep, so nothing will execute.

This is typically not intended.

The solution to this problem is to wrap any code that calls gevent.sleep in asyncio.sync_to_async or (equivalently) asyncio_gevent.greenlet_to_future(gevent.spawn(f)). This causes the function to be executed in another greenlet which prevents the main greenlet from getting blocked.

Example:

import gevent.monkey

gevent.monkey.patch_all()

import asyncio
import threading

import gevent

import asyncio_gevent

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())


async def f():
    print("f", 1)
    await asyncio.sleep(1)
    print("f", 2)


def g():
    print("g", 1)
    gevent.sleep(2)
    print("g", 2)


async def main():
    await asyncio.gather(f(), asyncio_gevent.sync_to_async(g)())
    # OR equivalently
    # await asyncio.gather(f(), asyncio_gevent.greenlet_to_future(gevent.spawn(g)))


if __name__ == "__main__":
    asyncio.run(main())

The output will be (as expected):

g 1
f 1
f 2
g 2

If gevent.sleep is called inside an async function, then the async function needs to first be wrapped in asyncio.async_to_sync.

import gevent.monkey

gevent.monkey.patch_all()

import asyncio

import gevent

import asyncio_gevent

asyncio.set_event_loop_policy(asyncio_gevent.EventLoopPolicy())


async def f():
    print("f", 1)
    await asyncio.sleep(1)
    print("f", 2)


async def g():
    print("g", 1)
    await asyncio.sleep(1)
    gevent.sleep(1)
    print("g", 2)


async def main():
    await asyncio.gather(
        f(), asyncio_gevent.sync_to_async(asyncio_gevent.async_to_sync(g))()
    )


if __name__ == "__main__":
    asyncio.run(main())

The output will again be (as expected):

g 1
f 1
f 2
g 2

License

MIT