butter 0.11

Library to interface to low level linux features (inotify, fanotify, timerfd, signalfd, eventfd, containers) with asyncio support

“Making things that little bit butter under linux”

Butter is a library to integrate some of Linux’s low level features into python eg signalfd, eventfd and timerfd. most of these functions are handy if you are looking into creating non-blocking servers, dealing with event loops or writing high performance services

Features

• Full testing of all error conditions in unit tests

• Small set of exceptions to deal with

• asyncio compatible versions of calls

• Emulationa nd reuse of built in objects wherever possible

• Both low level 1:1 calls and a high level interface available

• Default values chosen follow ‘least surprise’ principle (eg CLOCK_MONOTONIC instead of CLOCK_REALTIME to avoid issues with clock updates)

• Single codebase supporting python 2 and python 3 without modifcation for easier forward migration

Whats Available

• inotify (Complete support, includes asyncio support)

• seccomp (Limited support)

• fanotify (Limited support, includes asyncio support)

• splice

• tee

• vmsplice

• gethostname

• sethostname

• mount

• umount

• pivot_root

• getpid (bypasses glibc caching)

• getppid (bypasses glibc caching)

• eventfd (includes asyncio support)

• timerfd (includes asyncio support)

• pthread_sigmask (Avalible in python3.x but backported for python2.7)

• signalfd (includes asyncio support)

Whats Coming

Most of these exist in v0.2 as ctypes code. these are currently being rewritten to use cffi for speed and compatibility with pypy

• posix/linux aio (scatter/gather read writes with a completion based API instead of a ‘ready’ based interface, eg similar to IOCP on windows)

• Sphinx documentation

• Example code

• More unit tests

Supported Python

Butter currently supported the following python interpreters

• pypy (2.7 and 3.2 python implementations)

• cpython 3.4 (required for asyncio support)

• cpython 3.x

• cpython 2.7

Butter may work on older versions however it has not been tested on anything except the above interpreters and may break without warning

Installing

butter makes use of several C libraries to provide low level functionality. most of the compilation is taken care of by setup.py and cffi at install time however you will need to ensure that some header files are available on the machine doing the compilation (which may be different to the machine butter is being packaged for and Finally installed on)

Dev packages required:

• libseccomp

• kernel headers for a fairly recent kernel (3.x recommended)

to install butter use the following command:

$ pip install butter

this will pull in all the required dependencies and compile the required C extensions

for asyncio support, python 3.4 or newer is required. importing the asyncio modules on older versions of python will throw a syntax error. Hence why these are namespaced under butter.asyncio rather than in the base modules

Design

The ‘fd’ apis have been designed first as thin wrappers around the underlying kernel syscalls. A File-like object that then uses these syscalls is then made available that obeys the ‘event like’ interface and is based on the interface provided by File and friends. They are intended to be drop in replacements and be used with the select module

Event-like objects

Event like objects are identical to filelike objects except they have the following extra methods to replace the disabled read/readline/write/writeline functionality

• read_event: Return a single event

• read_events: Return all cached events OR read all events from the kernel

AsyncIO

Some syscalls (ones that deal with fd’s) have been made to work with the asyncio framework that became avalible in python3.4. The asyncio enabled object is named after the normal Event-like object equivlent with ‘_async’ after it, eg Inotify => Inotify_async. These asyncio Eventlike objects are designed to act and behave like the asyncio.Queue object without the ability to write events to the queue (ie they can only be read from and events are injected from the fd as required)

Exceptions

Butter was designed to have a small set of easily catchable exceptions and to reuse the builtin exceptions wherever possible. The list below is a rough guide of what exceptions you can expect to receive

• OSError/MemoryError: The Operating system was unable to fulfill your request

• ValueError: One of the arguments is invalid

• PermissionError: You do not have sufficient Privileges to use this call

• InternalError: The function hit an internal bug that was never expected to be hit but is tested for just in case. Please consider filing a bug report

• UnknownError: The Error is not specifically handled and unexpected, see exception.errno for the Error code that generated this exception

Projects using Butter

• https://pypi.python.org/pypi/icheck/ (Written by the author, inotify)

• https://pypi.python.org/pypi/igor-inotify/ (Written by the author, inotify)

Contact Details

• For pull requests, patches, feature requests and support please email the author ([e-mail])

• xmpp: [e-mail] or [e-mail]

Release History

0.11 (2015-05-02)

• Make close on exec the default behavior for python 3 platforms

• Low level functions now accept a file-like or event-like object in addition to a fd

• New ‘watch’ function as a simple wrapper around inotify for ultra simple uses cases

• wait() on eventlike objects now accept a timeout value in seconds to specify the maximum ammount of time to wait for an event to occur

API Changes

• Argument order to fanotify_mark and high level API changed to have a more logical flow

• Timerfd code has been overhauled and is no longer compatible

Bug Fixes

• ‘blocking’ on Inotify objects was an internal flag and is now prefixed with ‘_’ to denote its status

• Timerfd was created by default as CLOCK_MONOTONIC in contrast to other functions in butter

• Fixed bug that prevented compilation on 64bit machines with signalfd

0.10 (2015-03-20)

• errno to Exception mapping became a fixed part of the API (and unit tested)

• timerfd is now CLOCK_MONOTONIC by default

• Added Simpler TimerVal to replace Timerspec for timerfd

• Deprecated TimerStruct (will print a DeprecationWarning if such warnings have been turned on)

• OSError now returned instead of IOError. on python3 IOError = OSError so identical behavior will be observed

• Identical Error codes now return Identical Exceptions, except where there is a clear reason not o (eg InternalError)

• Errors due to incorrect arguments now all raise ValueError instead of ValueError or OSError

Bug Fixes

• C code now gets compiled at installation rather than first use

• Fixed up the name of an exception in error handling code leading to double exception

• pivot_root would return OSError rather than ValueError when incorect arguments were provided

• system.py was using its own definition of PermissionError, unify this with utils.py

• fanotify now raises PermissionError on EPERM instead of OSError (of which it is a subclass)

0.9.2 (2015-03-10)

• Deprecating seccomp support in favour of official libseccomp python bindings

Bug Fixes

• Add __init__.py to asyncio dir so that async methods can be imported

0.9.1 (2015-01-05)

Bug Fixes

• read_events was passing an undefined variable to actual implementations

0.9 (2014-05-24)

• Added eventfd support

• Added eventfd AsyncIO support

• Added timerfd support

• Added timerfd AsyncIO support

• Added Signalfd

• Added Signalfd AsyncIO support

• Added pthread_sigmask

• AsyncIO objects now have a close() method

• Converted all high level event objects to Eventlike objects

• Inotify events now have an is_dir_event property

• Added test suite

Bug Fixes

• Fixed issue with circular imports preventing python3.4 from working

• Fixed issue with python2.7 returning floats where python3 returned ints

0.8 (2014-05-17)

• Now works with python3.4 and higher

• ‘from butter import *’ now imports the system module

• Added trove classification

• Added friendly properties to inotify event object

• Added friendly properties to fanotify event object

• FanotifyEvents now use less memory

• AsyncIO support for inotify on supported platforms

• AsyncIO support for fanotify on supported platforms

0.7 (2014-03-16)

• Added system.py module

• Added gethostname syscall

• Added sethostname syscall

• Added mount syscall

• Added umount syscall

• Added pivot_root syscall

• Added getpid syscall

• Added getppid syscall

• Documented all new syscalls

0.6 (2014-03-12)

• splice syscall documentation

• Added tee() syscall

• Added tee() example

• Added vmsplice() syscall

• Added vmsplice() example

• Updated setup.py to newer auto detecting version

• hide ‘main’ functions in splice module

0.5 (2014-03-11)

• Added splice() syscall

0.4 (2013-12-12)

• Refactor fanotify

• Refactor inotify

• Provide fanotify.str_to_events()

• Provide inotify.str_to_events()

• Add int to signal name mapping for inotify

0.3 (2013-11-20)

• Support for inotify

• Initial support for fanotify

• Initial support for seccomp

• Add function to peer inside kernel buffer and get amount of available bytes to read

API Changes

• removed unused old (non working) signalfd, eventfd, aio

0.2 (2013-11-20)

• Initial support for signalfd

• Initial support for eventfd

• Initial support for aio