Linux aio ABI wrapper
Project description
[english] | [한국어 (korean)]
linux_aio: Python wrapper for Linux Kernel AIO
Python wrapper module that uses Linux Kernel AIO directly
What is Linux Kernel AIO?
In summary, it allows non-blocking and asynchronous use of blocking IO operations such as read(2) and write(2).
Related documents
It is different from POSIX AIO
The POSIX AIO APIs have the aio_
prefix, but the Linux Kernel AIO has the io_
prefix.
There is already a POSIX AIO API for asynchronous I/O, but Linux implements it in glibc, a user-space library, which is supposed to use multi-threading internally. So, as you can see from the experiment below, it's much worse than using the blocking IO API.
Implementation & Structure
Package linux_aio.raw
- ctypes module is used.
- It is defined to correspond
1:1
with the C header of Linux AIO.- Implemented 100% of the functionality when using C.
- All the functions shown in the man page based on Linux man pages (4.16), and all the functions added in 4.20.3 source code, as far as I can find them.
- If you know how to use the ctypes module to operate on pointers, you can also build other types of wrappers based on this package.
- It uses
syscall
for invoking ABI and cffi for gathering different syscall number by architecture on module installation. - python stub (
pyi
files - for type hint) are included.
Package linux_aio
- It based on package
linux_aio.raw
. - Unlike
linux_aio.raw
, it can be used without knowledge ofctypes
- Examples can be found in the code in the test directory.
Example
Examples can be found in the code in the test directory.
Notes & Limits
- Obviously available only on Linux
- Because it is a wrapper, it brings the constraints of Linux.
- It can not be used for files used as a kernel interface. (e.g.
cgroup
) - Sometimes it works as Blocking.
- There are some things that have been solved through development after posting.
- Some features are being added because they are still under development.
- There are also some features that are not supported when the Linux version is low
- You need to check Linux man pages (4.16) and its related API documentation
- It can not be used for files used as a kernel interface. (e.g.
Evaluation
Experiment script (requires python 3.7)
Setup
-
Distribution: Ubuntu Server 16.04.5 LTS
-
Linux: 4.19.0
-
CPU: 2-way Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
-
MEM: total 64GB
-
Storage: SK hynix SC300B SATA 512GB
-
Python: 3.7.2 (Ubuntu ppa)
-
Attempts to read a total of
1000
times in1ms
intervals. -
The file size varies from
1KB
to100KB
, but it is small. -
Experiment with increasing the number of files read at the same time
-
Because we have experimented with high-performance server, there may be larger performance differences when testing on a typical desktop.
Comparison target
- aiofiles - Uses Thread pool
- aiofile - Uses POSIX AIO
- libaio - Uses libaio
- python built-in open()
It is not a perfectly fair comparison.
aiofiles
and aiofile
are libraries that support asyncio. Since open()
is blocking, there is a disadvantage that you can not do any other work while IO is going on. libaio
and linux_aio
are non-blocking, but must be polled.
Results
It may differ from environment to environment.
Runtime
- Unit: second
# of files | 1 | 6 | 12 | 24 |
---|---|---|---|---|
aiofiles | 1.681 | 3.318 | 5.354 | 9.768 |
aiofile | 1.543 | 1.958 | 2.493 | 3.737 |
libaio | 1.311 | 1.344 | 1.362 | 1.423 |
open() | 1.252 | 1.322 | 1.375 | 1.481 |
linux_aio | 1.305 | 1.327 | 1.353 | 1.431 |
Threads
# of files | 1 | 6 | 12 | 24 |
---|---|---|---|---|
aiofiles | 321 | 321 | 321 | 321 |
aiofile | 3 | 8 | 15 | 26 |
libaio | 1 | 1 | 1 | 1 |
open() | 1 | 1 | 1 | 1 |
linux_aio | 1 | 1 | 1 | 1 |
Memory
- Physical memory (Virtual memory)
# of files | 1 | 6 | 12 | 24 |
---|---|---|---|---|
aiofiles | 21MB (22.6GB) | 21MB (22.6GB) | 21MB (22.6GB) | 21MB (22.6GB) |
aiofile | 17MB (258MB) | 17MB (654MB) | 17MB (1080MB) | 18MB (1949MB) |
libaio | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) |
open() | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) |
linux_aio | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) |
CPU Utilization
# of files | 1 | 6 | 12 | 24 |
---|---|---|---|---|
aiofiles | 42.8% | 85.0% | 102.2% | 113.2% |
aiofile | 31.4% | 52.4% | 67.0% | 84.0% |
libaio | 14.0% | 16.0% | 17.2% | 20.6% |
open() | 13.4% | 17.6% | 21.0% | 26.2% |
linux_aio | 13.0% | 15.0% | 16.0% | 21.0% |
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for linux_aio-0.3.0-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5e1819216672d9ed2ab6b55dda61f7cb9f114c1a1e3e18df626d83e4691df135 |
|
MD5 | 39f2f7b75d7b6d339fab03488d86fe96 |
|
BLAKE2b-256 | 1a3ab197ce0defcf04f61d11aa9998cea9ec1a03a42bed6844bd16156d5999e9 |
Hashes for linux_aio-0.3.0-cp37-cp37m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84edad11a429d4225cffc385c954c1d363da73a21aeda282ffe6fd37d2bcb887 |
|
MD5 | 609aaa76fe954f6372b180d59cab0a1c |
|
BLAKE2b-256 | d691b8b68190e40ba5060d86c55f155110ad4255476e16fa0ea6647562ddfb0f |
Hashes for linux_aio-0.3.0-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8bc5dcf770f097c241b4f3823148a2ae4b8adedad659bc39921b071f06435cd3 |
|
MD5 | 538b767e8dd05e0bf665789bb57f6577 |
|
BLAKE2b-256 | c3276e5bdada89e7c45e6195e38676d32f2c7b4120a211b0b20001a8248c7788 |
Hashes for linux_aio-0.3.0-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a7a98dbd04d44c014327b31d442a13d2742fa13a042c89445217af3aa75e8bca |
|
MD5 | 4975f899c15b1b987b992ddc79f97038 |
|
BLAKE2b-256 | 13126085d1c0703d6d64755c9f690e3e61ec2beb00233a18f228c489dfeca740 |
Hashes for linux_aio-0.3.0-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4eaab9be1f0fa342e120f678ed01e1183e1570b8f77792f518249c76ce3284ea |
|
MD5 | de82619a00257a6a8f0c42804038f13a |
|
BLAKE2b-256 | 817427f58633b9c2351378a2e0f47b6ab361272d074694db3fe332de7982bd53 |
Hashes for linux_aio-0.3.0-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 98ceb5b24723c67119139d1ada96b97a4e3738155c96ef7a1524f872fe5c32de |
|
MD5 | ca6f4162375ec02ecb4045e1896102c6 |
|
BLAKE2b-256 | 790fc208b80687b5801223b68ded861f32e28a3faae3b0d37a9d70510e605346 |
Hashes for linux_aio-0.3.0-cp34-cp34m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f1c2611891266ff86081b070579d0a7a47958982d457987b8d0fca5d451a287 |
|
MD5 | b6d4baa047974ed731adbba82b80e5c3 |
|
BLAKE2b-256 | 0ecd0a0beb9150217b54046bb7ddb58f60e3e1ea8285f60fee267fb041d42a56 |
Hashes for linux_aio-0.3.0-cp34-cp34m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fdf0caec605f8586739b21c8d6389246dff2e2205fd1ff4bbc2e84ab3f4a5acd |
|
MD5 | 3499f24272d2de38a05d526d3a2e3459 |
|
BLAKE2b-256 | 7c5b7901a7bf8cc2a06c3623f45b0e001435b613375e7a062cd3140a49f5297c |