pyvoy 0.1.3

A Python app server based on envoy

pyvoy

pyvoy is a Python application server implemented in Envoy. It is based on Envoy dynamic modules, embedding a Python interpreter into a module that can be loaded by a stock Envoy binary.

Features

• ASGI applications
• WSGI applications with worker threads
• A complete, battle-tested HTTP stack - it's just Envoy
  • Includes full HTTP protocol support, with HTTP/2 trailers and HTTP/3
• Any Envoy configuration features such as load shedding can be integrated as normal

Limitations

• Platforms limited to those supported by Envoy, which generally means glibc-based Linux on amd64/arm64 or MacOS on arm64
• Multiple worker processes. It is recommended to scale up with a higher-level orchestrator instead and use a health endpoint wired to RSS for automatic restarts if needed
• Certain non-compliant requests are prevented by Envoy itself
  • The full URL path, including query string, must be ASCII percent-encoded

Installation

pyvoy is published as a wheel that includes both the dynamic module and Envoy itself. You can use it in the same way as any other app server.

uv add pyvoy # or pip install

Running

pyvoy includes a CLI which supports standard options for HTTP servers. If just passing a module:attr name to point to an application, it will be served on plaintext on port 8000.

uv run pyvoy my.module:app

(if the application is named exactly app, :app can be omitted)

To see a full list of options:

uv run pyvoy -h

Docker

[!NOTE] This is an initial pattern and we will iterate on it, notably it will be good to remove the python version from the environment variables.

For production deployments to containers, we recommend running Envoy directly without the pyvoy CLI to avoid potential issues with subprocess spawning. The pyvoy CLI simply spawns Envoy with an appropriate YAML config and environment variables for loading the dynamic module. You can see the example Dockerfile for how to set up the config and environment for running Envoy directly.

Note that the pyvoy CLI with --print-envoy-config is run within the Dockerfile to easily set up the config. This is convenient for simple cases and should run well for normal deployments. But for experienced Envoy users that want to configure other aspects of Envoy, we also recommend managing the Envoy config in your codebase and adding it to the container - you can then tweak any and all Envoy parameters to meet your needs.

Development

We use poe for running development tasks. For a list of tasks, you can run

uv run poe -h

During development, the most common commands will be

uv run poe test # Run unit tests
uv run poe format # Apply possible formatting
uv run poe check # Run all checks. If this passes, CI should pass
uv run poe build # Only build pyvoy. Needed if running tests from IDE

Benchmarks

We have some preliminary benchmarks just to understand how the approach works specifically for HTTP/2. The main goal is to see if pyvoy runs in the same ballpark as other servers.

A single example from CI for a 10ms service with 10K response size shows:

Running benchmark for pyvoy with protocol=h2 sleep=10ms response_size=10000

Requests      [total, rate, throughput]         13460, 2691.78, 2686.15
Duration      [total, attack, wait]             5.011s, 5s, 10.489ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.546ms, 11.141ms, 11.066ms, 11.943ms, 12.246ms, 12.997ms, 16.798ms
Bytes In      [total, mean]                     134600000, 10000.00
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:13460
Error Set:


Running benchmark for granian with protocol=h2 sleep=10ms response_size=10000

Requests      [total, rate, throughput]         13489, 2697.08, 2691.47
Duration      [total, attack, wait]             5.012s, 5.001s, 10.423ms
Latencies     [min, mean, 50, 90, 95, 99, max]  9.253ms, 11.111ms, 11.038ms, 11.883ms, 12.172ms, 12.946ms, 16.373ms
Bytes In      [total, mean]                     134890000, 10000.00
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:13489
Error Set:

Running benchmark for hypercorn with protocol=h2 sleep=10ms response_size=10000

Requests      [total, rate, throughput]         1002, 183.30, 177.42
Duration      [total, attack, wait]             5.479s, 5.466s, 12.183ms
Latencies     [min, mean, 50, 90, 95, 99, max]  11.43ms, 163.65ms, 13.497ms, 16.099ms, 17.629ms, 5.019s, 5.023s
Bytes In      [total, mean]                     9720000, 9700.60
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           97.01%
Status Codes  [code:count]                      0:30  200:972
Error Set:
Get "http://localhost:8000/controlled": http2: server sent GOAWAY and closed the connection; LastStreamID=2001, ErrCode=NO_ERROR, debug=""

We see that hypercorn seems to not perform well with HTTP/2, with errors and resulting poor performance numbers. We will focus comparisons on granian.

Performance seems to be mostly the same between pyvoy and granian within the range of noise for a fast but still useful in real-world service.

We can try to isolate more performance of the app server itself with a less realistic service with no delay or response.

 Running benchmark for pyvoy with protocol=h2 sleep=0ms response_size=0

Requests      [total, rate, throughput]         104043, 20808.94, 20805.37
Duration      [total, attack, wait]             5.001s, 5s, 856.185µs
Latencies     [min, mean, 50, 90, 95, 99, max]  344.742µs, 1.327ms, 1.294ms, 1.736ms, 1.905ms, 2.271ms, 3.965ms
Bytes In      [total, mean]                     0, 0.00
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:104043
Error Set:

Running benchmark for granian with protocol=h2 sleep=0ms response_size=0

Requests      [total, rate, throughput]         96513, 19302.87, 19298.03
Duration      [total, attack, wait]             5.001s, 5s, 1.254ms
Latencies     [min, mean, 50, 90, 95, 99, max]  304.289µs, 1.501ms, 1.506ms, 1.827ms, 1.931ms, 2.2ms, 3.776ms
Bytes In      [total, mean]                     0, 0.00
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:96513
Error Set:

We again see very similar performance likely within the range of noise.