Linux aio ABI wrapper
Project description
[english] | [한국어 (korean)]
linux_aio: Python wrapper for Linux Kernel AIO
Python wrapper module that uses Linux Kernel AIO directly
What is Linux Kernel AIO?
In summary, it allows non-blocking and asynchronous use of blocking IO operations such as read(2) and write(2).
Related documents
It is different from POSIX AIO
The POSIX AIO APIs have the aio_
prefix, but the Linux Kernel AIO has the io_
prefix.
There is already a POSIX AIO API for asynchronous I/O, but Linux implements it in glibc, a user-space library, which is supposed to use multi-threading internally. So, as you can see from the experiment below, it's much worse than using the blocking IO API.
Implementation & Structure
Package linux_aio
- Implemented based on linux_aio_bind package which is low-level binding of Linux kernel AIO.
- Unlike linux_aio_bind, it can be used without knowledge of
ctypes
- Examples can be found in the code in the test directory.
Example
Examples can be found in the code in the test directory.
Notes & Limits
- Obviously available only on Linux
- Because it is a wrapper, it brings the constraints of Linux.
- It can not be used for files used as a kernel interface. (e.g.
cgroup
) - Sometimes it works as Blocking.
- There are some things that have been solved through development after posting.
- Some features are being added because they are still under development.
- There are also some features that are not supported when the Linux version is low
- You need to check Linux man pages (4.16) and its related API documentation
- It can not be used for files used as a kernel interface. (e.g.
Evaluation
Experiment script (requires python 3.7)
Setup
-
Distribution: Ubuntu Server 16.04.5 LTS
-
Linux: 4.19.0
-
CPU: 2-way Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
-
MEM: total 64GB
-
Storage: SK hynix SC300B SATA 512GB
-
Python: 3.7.2 (Ubuntu ppa)
-
Attempts to read a total of
1000
times in1ms
intervals. -
The file size varies from
1KB
to100KB
, but it is small. -
Experiment with increasing the number of files read at the same time
-
Because we have experimented with high-performance server, there may be larger performance differences when testing on a typical desktop.
Comparison target
- aiofiles - Uses Thread pool
- aiofile - Uses POSIX AIO
- libaio - Uses libaio
- python built-in open()
It is not a perfectly fair comparison.
aiofiles
and aiofile
are libraries that support asyncio. Since open()
is blocking, there is a disadvantage that you can not do any other work while IO is going on. libaio
and linux_aio
are non-blocking, but must be polled.
Results
It may differ from environment to environment.
Runtime
- Unit: second
# of files | 1 | 6 | 12 | 24 |
---|---|---|---|---|
aiofiles | 1.681 | 3.318 | 5.354 | 9.768 |
aiofile | 1.543 | 1.958 | 2.493 | 3.737 |
libaio | 1.311 | 1.344 | 1.362 | 1.423 |
open() | 1.252 | 1.322 | 1.375 | 1.481 |
linux_aio | 1.305 | 1.327 | 1.353 | 1.431 |
Threads
# of files | 1 | 6 | 12 | 24 |
---|---|---|---|---|
aiofiles | 321 | 321 | 321 | 321 |
aiofile | 3 | 8 | 15 | 26 |
libaio | 1 | 1 | 1 | 1 |
open() | 1 | 1 | 1 | 1 |
linux_aio | 1 | 1 | 1 | 1 |
Memory
- Physical memory (Virtual memory)
# of files | 1 | 6 | 12 | 24 |
---|---|---|---|---|
aiofiles | 21MB (22.6GB) | 21MB (22.6GB) | 21MB (22.6GB) | 21MB (22.6GB) |
aiofile | 17MB (258MB) | 17MB (654MB) | 17MB (1080MB) | 18MB (1949MB) |
libaio | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) |
open() | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) |
linux_aio | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) | 17MB (76MB) |
CPU Utilization
# of files | 1 | 6 | 12 | 24 |
---|---|---|---|---|
aiofiles | 42.8% | 85.0% | 102.2% | 113.2% |
aiofile | 31.4% | 52.4% | 67.0% | 84.0% |
libaio | 14.0% | 16.0% | 17.2% | 20.6% |
open() | 13.4% | 17.6% | 21.0% | 26.2% |
linux_aio | 13.0% | 15.0% | 16.0% | 21.0% |
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for linux_aio-0.4.0-py37-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 606cc0f8ab930c237a483ea3097c73babfe318a03c48a256cf895adad88dd90f |
|
MD5 | f962837da9ec0bd54bd0ae66e7367c7e |
|
BLAKE2b-256 | 416824e3365ae5c54bf3ffb7e527510f3686c1ae2000cc06bc31c36b869810e7 |
Hashes for linux_aio-0.4.0-py36-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09ad1ea6c4f5dbb0fafc2e02534e5204bfbeb80b7cbe1cbaba87a357f9e94438 |
|
MD5 | e55127acf77ee5ede27c0413dbff8f81 |
|
BLAKE2b-256 | 29b3a6b2dc82a23b303cb6970f644b20e67f01a91d1f5f47455aaaf0c2b9afee |
Hashes for linux_aio-0.4.0-py35-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c2877f47881229947a30e65a754d43ceec389670e481fec524a69075462c198d |
|
MD5 | e328ec8ae9284fae0d5096a0748bbd0e |
|
BLAKE2b-256 | 2ceb8f28c0e1d7368547e366ac3490327933b0e6e97863c8d2b941d75646fad3 |
Hashes for linux_aio-0.4.0-py34-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 33d63f1af3632d1c97b19b4b84c77bd07b9c5212594c86c441efdfdf3db666a7 |
|
MD5 | 8120a8fa9fea2d8828235fd2dff1f396 |
|
BLAKE2b-256 | d0d86d1c704114bbdc0416849ec97b472e73b17e0f8379f36499c4728710f8ed |