liburing 2024.4.22

...

Liburing

Liburing is Python + Cython wrapper around C Liburing, which is a helper to setup and tear-down io_uring instances.

• Fast & scalable asynchronous I/O (storage, networking, …) interface.

• io_uring reduces number of syscalls overhead & context switches, thus improving speed.

• …

Good(old) documentation Lord of the io_uring

Check out Shakti. It uses liburing and provides an easy to use Python async await Interface.

Requires

• Linux 6.1+ (6.7+ recommended)

• Python 3.8+

Includes (battery)

• C liburing 2.5+

Install, update & uninstall (Alpha)

Use pip to install, upgrade & uninstall Python wrapper:

python3 -m pip install liburing             # install

python3 -m pip install --upgrade liburing   # upgrade

python3 -m pip uninstall liburing           # uninstall

Install directly from GitHub:

python3 -m pip install --upgrade git+https://github.com/YoSTEALTH/Liburing

To find out all the class, functions and definitions:

import liburing

print(dir(liburing))  # to see all the importable names (this will not load all the modules)
help(liburing)        # to see all the help docs (this will load all the modules.)

Find out which io_uring operations is supported by the kernel:

# example/probe.py
import liburing

probe = liburing.probe()
print(probe)

Simple File Example

# example/open_write_read_close.py
from liburing import O_CREAT, O_RDWR, AT_FDCWD, iovec, io_uring, io_uring_get_sqe, \
                     io_uring_prep_openat, io_uring_prep_write, io_uring_prep_read, \
                     io_uring_prep_close, io_uring_submit, io_uring_wait_cqe, \
                     io_uring_cqe_seen, io_uring_cqe, io_uring_queue_init, \
                     io_uring_queue_exit, trap_error


def open(ring, cqes, path, flags, mode=0o660, dir_fd=AT_FDCWD):
    _path = path if isinstance(path, bytes) else str(path).encode()
    # if `path` is relative and `dir_fd` is `AT_FDCWD`, then `path` is relative
    # to current working directory. Also `_path` must be in bytes

    sqe = io_uring_get_sqe(ring)  # sqe(submission queue entry)
    io_uring_prep_openat(sqe, dir_fd, _path, flags, mode)
    return _submit_and_wait(ring, cqes)  # returns fd


def write(ring, cqes, fd, data, offset=0):
    iov = iovec(data)  # or iovec([bytearray(data)])
    sqe = io_uring_get_sqe(ring)
    io_uring_prep_write(sqe, fd, iov.iov_base, iov.iov_len, offset)
    return _submit_and_wait(ring, cqes)  # returns length(s) of bytes written


def read(ring, cqes, fd, length, offset=0):
    iov = iovec(bytearray(length))  # or [bytearray(length)]
    sqe = io_uring_get_sqe(ring)
    io_uring_prep_read(sqe, fd, iov.iov_base, iov.iov_len, offset)
    _submit_and_wait(ring, cqes)  # get actual length of file read.
    return iov.iov_base


def close(ring, cqes, fd):
    sqe = io_uring_get_sqe(ring)
    io_uring_prep_close(sqe, fd)
    _submit_and_wait(ring, cqes)  # no error means success!


def _submit_and_wait(ring, cqes):
    io_uring_submit(ring)  # submit entry
    io_uring_wait_cqe(ring, cqes)  # wait for entry to finish
    result = trap_error(cqes.res)  # auto raise appropriate exception if failed
    # note `cqe.res` returns results, if ``< 0`` its an error, if ``>= 0`` its the value

    # done with current entry so clear it from completion queue.
    io_uring_cqe_seen(ring, cqes)
    return result  # type: int


def main():
    ring = io_uring()
    cqes = io_uring_cqe()
    try:
        io_uring_queue_init(8, ring, 0)

        fd = open(ring, cqes, '/tmp/liburing-test-file.txt', O_CREAT | O_RDWR)
        print('fd:', fd)

        length = write(ring, cqes, fd, b'hello world')
        print('wrote:', length)

        content = read(ring, cqes, fd, length)
        print('read:', content)

        close(ring, cqes, fd)
        print('closed.')
    finally:
        io_uring_queue_exit(ring)


if __name__ == '__main__':
    main()

Note

• Try not to use from liburing import * this will load all the modules at once, unless that’s what you want!

Cython Note

• You can cimport liburing directly into your project if you are planning on compiling your project as well.

• There is also src/liburing/lib directory with raw .pxd header files.

• All raw C wrapped function, enum, struct, defines starts with __, not including anything that’s ctypedef. This is to prevent naming confusion between whats C and Cython side.

• liburing must be included in both build-system.requires and project.dependencies in pyproject.toml to compile and use properly.

• Check out Shakti to see how to include liburing using cython.

TODO

• Stable Release (currently still in alpha)

• Linux 6.1 Backwards compatibility.

License

Free, Public Domain (CC0). Read more