pyinotifyd 0.0.1

Monitoring filesystems events with inotify on Linux and execute tasks.

pyinotifyd

A daemon to monitore filesystems events with inotify on Linux and execute tasks (Python methods or Shell commands) with an optional delay. It is also possible to cancel delayed tasks.

Requirements

• pyinotify

Installation

# install pyinotifyd with pip
pip install pyinotifyd

# install systemd service and create config directory
pyinotifyd --install

# uninstall systemd service
pyinotifyd --uninstall

Autostart

The following init systems are supported.

systemd

# start the daemon at boot time
systemctl enable pyinotifyd.service

# start the daemon immediately
systemctl start pyinotifyd.service

OpenRC (Gentoo)

# start the daemon at boot time
rc-update add pyinotifyd default

# start the daemon immediately
rc-service pyinotifyd start 

Configuration

The config file /etc/pyinotifyd/config.py is written in Python syntax. pyinotifyd reads and executes its content, that means you can add your custom Python code to the config file.

Tasks

Tasks are Python methods that are called in case an event occurs. They can be bound directly to an event type in an event map. Although this is the easiest and quickest way, it is usually better to add a task to a scheduler and bind the scheduler to event types.

Simple

This is a very basic example task that just logs each event and task_id:

async def task(event, task_id):
    logging.info(f"{task_id}: execute example task: {event}")

FileManager

FileManager moves, copy or deletes files and/or directories following a list of rules.

A rule holds an action (move, copy or delete) and a regular expression src_re. The FileManager task will be executed if src_re matches the path of an event. If the action is copy or move, the destination path dst_re is mandatory and if action is delete and rec is set to True, non-empty directories will be deleted recursively.
With auto_create set to True, possibly missing subdirectories in dst_re are created automatically. Regex subgroups or named-subgroups may be used in src_re and dst_re.
Set the mode of moved/copied files/directories with filemode and dirmode. Ownership of moved/copied files/directories is set with user and group. Mode and ownership is also set to automatically created subdirectories.
Log messages with logname.

rule = Rule(
    action="move",
    src_re="^/src_path/(?P<path>.*).to_move$",
    dst_re="/dst_path/\g<path>",
    auto_create=False,
    rec=False,
    filemode=None,
    dirmode=None,
    user=None,
    group=None)

fm = FileManager(
    rules=[rule],
    logname="filemgr")

FileManager provides a task fm.task.

Schedulers

pyinotifyd has different schedulers to schedule tasks with an optional delay. The advantages of using a scheduler are consistent logging and the possibility to cancel delayed tasks. Furthermore, schedulers have the ability to differentiate between files and directories.

TaskScheduler

TaskScheduler schedules task with an optional delay in seconds. Use the files and dirs arguments to schedule tasks only for files and/or directories. Log messages with logname. All arguments except for task are optional.

s = TaskScheduler(
    task=task,
    files=True,
    dirs=False,
    delay=0,
    logname="sched")

TaskScheduler provides two tasks which can be bound to an event in an event map.

• s.schedule Schedule a task. If there is already a scheduled task, it will be canceled first.
• s.cancel Cancel a scheduled task.

ShellScheduler

ShellScheduler schedules Shell command cmd. The placeholders {maskname}, {pathname} and {src_pathname} are replaced with the actual values of the event. ShellScheduler has the same optional arguments as TaskScheduler and provides the same tasks.

s1 = ShellScheduler(
    cmd="/usr/local/bin/task.sh {maskname} {pathname} {src_pathname}")

Event maps

EventMap maps event types to tasks. It is possible to set a list of tasks to run multiple tasks on a single event. If the task of an event type is set to None, it is ignored. This is an example:

event_map = EventMap({
        "IN_CLOSE_NOWRITE": [s.schedule, s1.schedule],
        "IN_CLOSE_WRITE": s.schedule})

The following event types are available:

• IN_ACCESS: a file was accessed
• IN_ATTRIB: a metadata changed
• IN_CLOSE_NOWRITE: an unwritable file was closed
• IN_CLOSE_WRITE: a writable file was closed
• IN_CREATE: a file/directory was created
• IN_DELETE: a file/directory was deleted
• IN_DELETE_SELF: a watched item itself was deleted
• IN_IGNORED: raised when a watch is removed, probably useless for you
• IN_MODIFY: a file was modified
• IN_MOVE_SELF: a watched item was moved, currently its full pathname destination can only be known if its source and destination directories were both watched. Otherwise, the file is still being watched but you cannot rely anymore on the given path attribute event.path
• IN_MOVED_FROM: a file/directory in a watched directory was moved from another specified watched directory. Can trace the full move of an item when IN_MOVED_TO is available too, in this case if the moved item is itself watched, its path will be updated (see IN_MOVE_SELF)
• IN_MOVED_TO: a file/directory was moved to another specified watched directory (see IN_MOVE_FROM)
• IN_OPEN: a file was opened
• IN_Q_OVERFLOW: the event queue overflown. This event is not associated with any watch descriptor
• IN_UNMOUNT: when backing filesystem was unmounted. Notified to each watch of this filesystem

Watches

Watch watches path for event types in event_map and execute the corresponding task(s). If rec is True, a watch will be added on each subdirectory in path. If auto_add is True, a watch will be added automatically on newly created subdirectories in path.

watch = Watch(
    path="/tmp",
    event_map=event_map,
    rec=False,
    auto_add=False)

Pyinotifyd

pyinotifyd expects an instance of Pyinotifyd named pyinotifyd defined in the config file. The options are a list of watches and the shutdown_timeout. pyinotifyd will wait shutdown_timeout seconds for pending tasks to complete during shutdown. Log messages with logname.

pyinotifyd = Pyinotifyd(
    watches=[watch],
    shutdown_timeout=30,
    logname="daemon")

Logging

Pythons logging framework is used to log messages (see https://docs.python.org/3/howto/logging.html).

Configure the global loglevel. This is the default:

logging.getLogger().setLevel(logging.WARNING)

It is possible to configure the loglevel per log name. This is an example for logname TaskScheduler:

logging.getLogger("TaskScheduler").setLevel(logging.INFO)

Syslog

Add this to your config file to send log messages to a local syslog server.

# send log messages to the Unix socket of the syslog server.
syslog = logging.handlers.SysLogHandler(
    address="/dev/log")

# set the log format of syslog messages
log_format = "pyinotifyd/%(name)s: %(message)s"
syslog.setFormatter(
    logging.Formatter(formatter)

# set the log level for syslog messages
syslog.setLevel(logging.INFO)

# enable syslog
logging.getLogger().addHandler(syslog)

# or enable syslog just for TaskScheduler
logging.getLogger("TaskManager").addHandler(syslog)

Examples

Schedule Python task for all events

async def task(event, task_id):
    logging.info(f"{task_id}: execute example task: {event}")

s = TaskScheduler(
    task=task,
    files=True,
    dirs=True)

event_map = EventMap(
    default_task=s.schedule)

watch = Watch(
    path="/tmp",
    event_map=event_map,
    rec=True,
    auto_add=True)

pyinotifyd_config = PyinotifydConfig(
    watches=[watch],
    shutdown_timeout=5)

Schedule Shell commands for specific events on files

s = ShellScheduler(
    cmd="/usr/local/sbin/task.sh {pathname}",
    files=True,
    dirs=False)

event_map = EventMap(
    {"IN_WRITE_CLOSE": s.schedule})

watch = Watch(
    path="/tmp",
    event_map=event_map,
    rec=True,
    auto_add=True)

pyinotifyd_config = PyinotifydConfig(
    watches=[watch],
    shutdown_timeout=5)

Move, copy or delete newly created files after a delay

move_rule = Rule(
    action="move",
    src_re="^/src_path/(?P<path>.*)\.to_move$",
    dst_re="/dst_path/\g<path>",
    auto_create=True,
    filemode=0o644,
    dirmode=0o755)

copy_rule = Rule(
    action="copy",
    src_re="^/src_path/(?P<path>.*)\.to_copy$",
    dst_re="/dst_path/\g<path>",
    auto_create=True,
    filemode=0o644,
    dirmode=0o755)

delete_rule = Rule(
    action="delete",
    src_re="^/src_path/(?P<path>.*)\.to_delete$",
    rec=False)

fm = FileManager(
    rules=[move_rule, copy_rule, delete_rule])

s = TaskScheduler(
    task=fm.task,
    delay=30,
    files=True,
    dirs=False)

event_map = EventMap({
        "IN_CLOSE_WRITE": s.schedule,
        "IN_DELETE": s.cancel,
        "IN_DELETE_SELF": s.cancel,
        "IN_MODIFY": s.cancel,
        "IN_MOVED_TO": s.schedule,
        "IN_UNMOUNT": s.cancel})

watch = Watch(
    path="/src_path",
    event_map=event_map,
    rec=True,
    auto_add=True)

# note that shutdown_timeout should be greater than the greatest scheduler delay,
# otherwise pending tasks may get cancelled during shutdown.
pyinotifyd_config = PyinotifydConfig(
    watches=[watch],
    shutdown_timeout=35)