aioreq 1.0.5

Asynchronous HTTP library.

Aioreq is a Python low-level asynchronous HTTP client library. It is built on top of TCP sockets and implements the HTTP protocol entirely on his own.

---

Documentation

Click here

Install

$ pip install aioreq

Usage

Basic usage

---

>>> import aioreq
>>> import asyncio
>>>
>>> cl = aioreq.Client()
>>>
>>> resp = asyncio.run(
...	cl.get('https://www.google.com')
...	)
>>> resp
<Response 200 OK>
>>> resp.status
200
>>> resp.status_message
'OK'
>>> resp.request
<Request GET www.google.com>
>>> headers = resp.headers # dict
>>> body = resp.content # bytes object

or you can use the default client, which is a synchoronous wrapper for an asynchronous interface

>>> aioreq.get('https://www.google.com')
<Response 200 OK>

Alternatively, the best practice is to use a Context manager.

>>> import aioreq
>>> import asyncio
>>>
>>> async def main():
...     async with aioreq.Client() as cl:
...         return await cl.get('https://google.com')
>>> asyncio.run(main())
<Response 200 OK>

More advanced usage

---

This code will asynchronously send 100 get requests to google.com, which is much faster than synchronous libraries. Also, the client persistent connection mechanism was disabled, so that a connection would be opened for each request.

>>> import asyncio
>>> import aioreq
>>>
>>> async def main():
...     async with aioreq.http.Client(persistent_connections=False) as cl:
...         tasks = []
...         for j in range(100):
...             tasks.append(
...                 asyncio.create_task(
...                     cl.get('https://www.google.com/', )
...                 )
...             )
...         await asyncio.gather(*tasks)
>>> asyncio.run(main())

Streams

---

We occasionally use the HTTP protocol to download videos, photos, and possibly files. When downloading very large files, Stream must be used instead of the default Client. When a client downloads videos or files, the server responds with all information including headers, status code, status message, and full body, which can be very large. As a result, we cannot store it in RAM. Stream only returns a portion of the body per iteration, allowing us to write it to disk, then receive another portion and solve the ram overflow problem.

There is some fundamental Stream usage.

>>> import aioreq
>>> import asyncio
>>> from aioreq import Request
>>> 
>>> async def main():
...     req = Request(url="https://www.youtube.com", method="GET")
...     async with aioreq.StreamClient(request=req) as response:
...         assert response.status == 200
...         async for chunk in response.content:
...             assert chunk   
>>> asyncio.run(main())

Middlewares

Aioreq now supports 'middleware' power.

The first steps with middleware

Aioreq provides default middlewares to each client. We can see that middlewares by importing 'default_middlewares' variable.

>>> import aioreq
>>> aioreq.middlewares.default_middlewares
('RetryMiddleWare', 'RedirectMiddleWare', 'CookiesMiddleWare', 'DecodeMiddleWare', 'AuthenticationMiddleWare')

The first item on this list represents the first middleware that should handle our request (i.e. the closest middleware to our client), while the last index represents the closest middleware to the server.

To override the default middlewares, we can pass our middlewares to the Client.

>>> client = aioreq.Client(middlewares=aioreq.middlewares.default_middlewares[2:])

This client will no longer redirect or retry responses.

Also, because aioreq stores middlewares in Client objects as linked lists, we can simply change the head of that linked list to skip the first middleware.

>>> client = aioreq.Client()
>>> client.middlewares.__class__.__name__
'RetryMiddleWare'
>>>
>>> client.middlewares = client.middlewares.next_middleware
>>> client.middlewares.__class__.__name__
'RedirectMiddleWare'
>>> 
>>> client.middlewares = client.middlewares.next_middleware
>>> client.middlewares.__class__.__name__
'CookiesMiddleWare'

or

>>> client = aioreq.Client()
>>> client.middlewares = client.middlewares.next_middleware.next_middleware
>>> # alternative for client = aioreq.Client(middlewares=aioreq.middlewares.default_middlewares[2:])

Create your own middlewares!

All 'aioreq' middlewares must be subclasses of the class 'middlewares.MiddleWare'

MiddleWare below would add 'test-md' header if request domain is 'www.example.com'

>>> import aioreq
>>>
>>> class CustomMiddleWare(aioreq.middlewares.MiddleWare):
...     async def process(self, request, client):
...         if request.host == 'www.example.com':
...             request.headers['test_md'] = 'test'
...         return await self.next_middleware.process(request, client)
...
>>> client = aioreq.Client()
>>> client.middlewares = CustomMiddleWare(next_middleware=client.middlewares)

Our CustomMiddleWare will now be the first middleware (i.e. closest to the client). Because 'aioreq' middlewares are stored as linked lists, this pattern works (i.e. same as linked list insert method).

Alternatively, we can alter the list of middlewares that the client receives.

>>> client = aioreq.Client(middlewares = (CustomMiddleWare, ) + aioreq.middlewares.default_middlewares)
>>> client.middlewares.__class__.__name__
'CustomMiddleWare'

Benchmarks

Aioreq is a very fast library, and we compared it to other Python libraries to demonstrate its speed.

I used httpx to compare speed.

---

Benchmark run

First, clone aioreq repository.

Then...

$ cd aioreq
$ python -m venv venv
$ source ./venv/bin/activate
$ pip install '.[dev]'
$ cd benchmarks
$ ./run_tests

---

Benchmark results

This is the average execution time of each library for 200 asynchronous requests where responses was received without chunked transfer encoding.

Benchmark settings.

• Url - http://www.google.com
• Requests count - 200

With Content-Length

$ cd becnhmarks
$ ./run_tests
Tests with module loading
---------------------------
aioreq benchmark

real    0m1.893s
user    0m0.777s
sys     0m0.063s
---------------------------
httpx benchmark

real    0m2.448s
user    0m0.840s
sys     0m0.151s

With Transfer-Encoding: Chunked

This is the average execution time of each library for 200 asynchronous requests where responses was received with chunked transfer encoding.

Benchmark settings.

• Url - https://www.youtube.com
• Requests count - 200

$ cd benchmarks
$ ./run_tests
Tests with module loading
---------------------------
aioreq benchmark

real    0m2.444s
user    0m1.090s
sys     0m0.089s
---------------------------
httpx benchmark

real    0m2.990s
user    0m1.254s
sys     0m0.127s

As you can see, the synchronous code lags far behind when we make many requests at the same time.

Keylog

If the SSLKEYLOGFILE environment variable is set, Aioreq will write keylogs to it.

$ export SSLKEYLOGFILE=logs

Then just run python script.

$ python aioreq_app.py
$ ls -l
total 8
-rw-r--r-- 1 user user  94 Dec  5 17:19 aioreq_app.py
-rw-r--r-- 1 user user 406 Dec  5 17:19 logs

Now, the 'logs' file contains keylogs that can be used to decrypt your TLS/SSL traffic with a tool such as 'wireshark'.

Authentication

If the auth parameter is included in the request, Aioreq will handle authentication.

>>> import aioreq
>>> import asyncio
>>> async def send_req():
...     async with aioreq.Client() as cl:
...         return await cl.get('http://httpbin.org/basic-auth/foo/bar', auth=('foo', 'bar'))
>>> resp = asyncio.run(send_req())
>>> resp.status
200

Parameter 'auth' should be a tuple with two elements: password and login.

Authentication is enabled by 'AuthenticationMiddleWare,' so exercise caution when managing middlewares manually.

Supported Features

Aioreq support basic features to work with HTTP/1.1.
More functionality will be available in future releases.
This is the latest version features.

• Keep-Alive (Persistent Connections)
• Middlewares
• Keylogs
• Authentication
• Cookies
• Automatic accepting and decoding responses. Using Accept-Encoding header
• HTTPS support, TLS/SSL Verification
• Request Timeouts