lowhaio 0.0.3

Lightweight and dependency-free Python asyncio HTTP/1.1 client.

lowhaio

---

Work in progress. These docs serve as a rough design spec.

---

A lightweight and dependency-free Python asyncio HTTP/1.1 client. Rather than abstracting away the HTTP protocol or aspects of the connection, the provided API is a small suite of 6 utility functions that exposes low-level behaviour. This provides both flexibility and minimisation of unnecessary code and processing in the production application. Clients can of course wrap lowhaio calls in their own functions to reduce duplication or provide abstractions as they see fit.

lowhaio is HTTPS-first: non-encrypted HTTP connections are possible, but require a bit more client code.

Responsibilities of lowhaio

• Connection pooling
• Creation of TLS/SSL connections, i.e. for HTTPS
• Streaming of requests and responses
• Parsing of HTTP headers, and closing connections on "Connection: close" headers.

While many uses do not need requests or responses streamed, it's not possible to provide a meaningful streaming API on top of a non-streaming API, so this is deliberately provided out-of-the-box by lowhaio.

Responsibilities of client code

• DNS resolution: this is a simple call to loop.getaddrinfo (or alternative such as aiodns)
• Parsing URLs: HTTP requests are expressed in terms of protocol, hostname, port and path, and often there is no need to concatanate these together only to then require the concatanation to be parsed.
• Conversion of strings to bytes or vice-versa: many usages do not need encoding or decoding behaviour beyond ASCII.
• Conversion of tuples of bytes to more complex data structures, such as dictionaries.
• Setting connection limits or timeouts: no defaults are appropriate to all situations.
• Serializing HTTP headers: this is trivial.
• Retrying failed connections, including those caused by the server closing perisistant connections
• Calling lowhaio functions in the appropriate order. Some knowledge of the HTTP protocol is required.

Usage

host = b'www.example.com'
port = 443
ip_address = await loop.getaddrinfo(host, port)
async with \
        lowhaio.connection_pool(max=20, max_wait_connection_seconds=20, max_requests_per_connection=20, max_idle_seconds=5, max_wait_send_recv_seconds=10, chunk_bytes=4096) as pool, \
        pool.connection(ip_address, port, ssl_context) as connection:

    await pool.send(connection,
        b'POST /path HTTP/1.1\r\n'
        b'Host: www.w3.org\r\n'
        b'\r\n'
        b'Body'
    )

    code, cursor = await pool.recv_status_line(connection):
    async for header_key, header_value, cursor in pool.recv_headers(connection, cursor):
        # Do something with the header values

    async for body_bytes in pool.recv_body(connection, cursor):
        # Save the body bytes

Design

Two main principles inform the design of lowhaio.

• Overal understanding of what the final application is doing is a primary concern, and this covers both lowhaio client code and lowhaio internal code. Making an elegant API is not significantly prioritised over ease of overall understanding what the application is doing in terms of data transformations or bytes sent/received. For example, the API has changed during development to simpliciy/optimise the internal code.

• All code in the client is required for all uses of the client: nothing is unnecessary. For example, HTTP and HTTPS require different code paths, and so HTTPS is chosen to be supported out of the box over unencrypted HTTP. HTTP is possible, but with more code.

Testing aims

• 100% code coverage.

• Patching in tests done only to avoid non-determinism or slow tests, e.g. to fast-forward time when testing timeout behaviour.

• No direct tests of private functions.

Recipies

Non-streaming requests and responses

Sending a dictionary of HTTP headers

Receiving a dictionary of HTTP headers

Chunked transfer encoding

Compressed content encoding

Non-encrypted connections

class NonSSLContext():
    def wrap_socket(self, sock, *args, **kwargs):
        sock.__class__ = NonSSLSocket
        return sock

class NonSSLSocket(socket):
    __slots__ = ()

    def do_handshake(self):
        pass

    def unwrap(self):
        self.__class__ = socket
        return self

async with lowhaio.socket(pool, host, port, NonSSLContext(),...):
    ...