Linux aio ABI wrapper
Project description
[english] | [한국어 (korean)]
linux_aio: Python wrapper for Linux Kernel AIO
Python wrapper module that uses Linux Kernel AIO directly
What is Linux Kernel AIO?
In summary, it allows non-blocking and asynchronous use of blocking IO operations such as read(2) and write(2).
Related documents
It is different from POSIX AIO
The POSIX AIO APIs have the aio_
prefix, but the Linux Kernel AIO has the io_
prefix.
There is already a POSIX AIO API for asynchronous I/O, but Linux implements it in glibc, a user-space library, which is supposed to use multi-threading internally. So, as you can see from the experiment below, it's much worse than using the blocking IO API.
Implementation & Structure
Package linux_aio.raw
- ctypes module is used.
- It is defined to correspond
1:1
with the C header of Linux AIO.- Implemented 100% of the functionality when using C.
- All the functions shown in the man page based on Linux man pages (4.16), and all the functions added in 4.20.3 source code, as far as I can find them.
- If you know how to use the ctypes module to operate on pointers, you can also build other types of wrappers based on this package.
- It uses
syscall
for invoking ABI and cffi for gathering different syscall number by architecture on module installation. - python stub (
pyi
files - for type hint) are included.
Package linux_aio
- It based on package
linux_aio.raw
. - Unlike
linux_aio.raw
, it can be used without knowledge ofctypes
- Examples can be found in the code in the test directory.
Example
Examples can be found in the code in the test directory.
Notes & Limits
- Obviously available only on Linux
- Because it is a wrapper, it brings the constraints of Linux.
- It can not be used for files used as a kernel interface. (e.g.
cgroup
) - Sometimes it works as Blocking.
- There are some things that have been solved through development after posting.
- Some features are being added because they are still under development.
- There are also some features that are not supported when the Linux version is low
- You need to check Linux man pages (4.16) and its related API documentation
- It can not be used for files used as a kernel interface. (e.g.
Evaluation
Setup
-
Distribution: Ubuntu Server 16.04.5 LTS
-
Linux: 4.19.0
-
CPU: 2-way Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
-
MEM: total 64GB
-
Storage: SK hynix SC300B SATA 512GB
-
Python: 3.7.2 (Ubuntu ppa)
-
Attempts to read a total of
1000
times in1ms
intervals. -
The file size varies from
1KB
to100KB
, but it is small. -
Experiment with increasing the number of files read at the same time
-
Because we have experimented with high-performance server, there may be larger performance differences when testing on a typical desktop.
Comparison target
- aiofiles - Uses Thread pool
- aiofile - Uses POSIX AIO
- libaio - Uses libaio
- python built-in open()
It is not a perfectly fair comparison.
aiofiles
and aiofile
are libraries that support asyncio. Since open()
is blocking, there is a disadvantage that you can not do any other work while IO is going on. libaio
and linux_aio
are non-blocking, but must be polled.
Results
It may differ from environment to environment.
Runtime
- Unit: second
# of files | 1 | 6 | 12 | 24 |
---|---|---|---|---|
aiofiles | 1.681 | 3.318 | 5.354 | 9.768 |
aiofile | 1.543 | 1.958 | 2.493 | 3.737 |
libaio | 1.311 | 1.344 | 1.362 | 1.423 |
open() | 1.252 | 1.322 | 1.375 | 1.481 |
linux_aio | 1.305 | 1.327 | 1.353 | 1.431 |
Threads
# of files | 1 | 6 | 12 | 24 |
---|---|---|---|---|
aiofiles | 321 | 321 | 321 | 321 |
aiofile | 3 | 8 | 15 | 26 |
libaio | 1 | 1 | 1 | 1 |
open() | 1 | 1 | 1 | 1 |
linux_aio | 1 | 1 | 1 | 1 |
Memory
- Physical memory (Virtual memory)
# of files | 1 | 6 | 12 | 24 |
---|---|---|---|---|
aiofiles | 21MB (22.6GB) | 21MB (22.6GB) | 21MB (22.6GB) | 21MB (22.6GB) |
aiofile | 17MB (258MB) | 17MB (654MB) | 17MB (1080MB) | 18MB (1949MB) |
libaio | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) |
open() | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) |
linux_aio | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) |
CPU Utilization
# of files | 1 | 6 | 12 | 24 |
---|---|---|---|---|
aiofiles | 42.8% | 85.0% | 102.2% | 113.2% |
aiofile | 31.4% | 52.4% | 67.0% | 84.0% |
libaio | 14.0% | 16.0% | 17.2% | 20.6% |
open() | 13.4% | 17.6% | 21.0% | 26.2% |
linux_aio | 13.0% | 15.0% | 16.0% | 21.0% |
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for linux_aio-0.2.2-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8ecfc6555772528f871b087a450cb7f1d65bb3b230ee2683cdd8f8a0022a2e74 |
|
MD5 | c6b9bf09474d42815e45553473d01906 |
|
BLAKE2b-256 | 3d2576e0ceeaeb172f5773cb550da1c6b31a780b57f70b2a4183fd7ab2a86a51 |
Hashes for linux_aio-0.2.2-cp37-cp37m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12a6214f43ba01dfb3bf33f903e06b3b0a2d78ca25df3501968ca76d6bb54ea8 |
|
MD5 | e070b315aae4133ba8b5e32048012366 |
|
BLAKE2b-256 | e8791251416d47b4d0ae874ce9acbf21fa48e858de430328f68472db300d361d |
Hashes for linux_aio-0.2.2-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3912344f73b3bba214b0ebfca5f853197ed1be2f104158ed29e47e0c804d5dc6 |
|
MD5 | e437b48a0de5c2339557dc06790d7128 |
|
BLAKE2b-256 | a6ff171fffe252534bc2ad3a7db051ad89438e65a7ab7ed435fd10a59a7b7b20 |
Hashes for linux_aio-0.2.2-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 95ebd047b00c22cad3edb217872411cc23908cedbb711b86bf708e4deb56a5a9 |
|
MD5 | 6a9cedd91ad2622df3899b42aae62881 |
|
BLAKE2b-256 | c15722e0b121e69f16668e28d79259e86a322c6b2479fc4e76b8656c6549e561 |
Hashes for linux_aio-0.2.2-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 666a6c7627d5830ef7468ffd3c5da83e6fa0850dc02b227b157b74df2e30ba41 |
|
MD5 | 5e388388d21381d15cce43170baa6a83 |
|
BLAKE2b-256 | e93f5382fa63d7a6028ebd3c065e50e6267ebcac575a6724192082874df05801 |
Hashes for linux_aio-0.2.2-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3d3fc1b2449fe9094c9a6063d07d8a5ae21b7586875a672f8e5d2c689de8fce7 |
|
MD5 | ac4252d68205249cf1955b56203d1dc7 |
|
BLAKE2b-256 | 0f4c5258efb3403cd3f20f1c26c2a7de44403ed3136c43f8ccc3925b6227d267 |
Hashes for linux_aio-0.2.2-cp34-cp34m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a8dd1347fb9ed3d1ecd3fd6e612eb585e2710f4e82d4d92951ecf37d1787dce |
|
MD5 | 5749f37fbc8106b14d36a8e912f8753e |
|
BLAKE2b-256 | 8a195d637f9acb702db1d12690d3580ee71889bd4e05d047daaa30096a29c403 |
Hashes for linux_aio-0.2.2-cp34-cp34m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 762c6df852fa9e6d163c1800b30099a042c8380c57cee96069f009cb396786b5 |
|
MD5 | 6284c42e1ba5ce01f8dc0f327cdda548 |
|
BLAKE2b-256 | 9293bab1a029752741dc8377917e548af0c4fcf0ef515e06de9fa46bd25855c1 |