winnan 0.1.0

More consistent file handling across POSIX and Windows systems

winnan

1. (verb) to struggle, suffer, contend

https://en.wiktionary.org/wiki/winnan

Because everything about managing files on Windows is terrible.

Installation

$ python -m pip install --upgrade winnan

Usage

Use the winnan.open() function exactly like using Python’s built-in open() function. It aims to be a drop-in replacement.

import winnan

with winnan.open("myfile") as fileobj:
    pass

The file descriptor underlying the returned file object has the following two properties on both POSIX and Windows systems:

1. The file descriptor is non-inheritable.

2. The file can be scheduled for deletion while the file descriptor is still open.

   Saying “scheduled for deletion” rather than “deleted” is to be pedantic about how the file is put into a “delete pending” state on Windows until the last file handle is closed. See Mercurial’s developer documentation as a reference for the semantics of certain operations on the file while it is in a “delete pending” state.

Motivation

Unsurprisingly, the complications around managing files involve dealing with other processes. Depending on what the Python application is doing, there may be a need to deal with not only processes spawned by the Python application but also other processes running on the machine.

1. As documented by PEP-446, opening a file concurrently with spawning a subprocess may lead to the file descriptor being inherited unintentionally.

   The process cannot access the file because it is being used by another process.

   is perhaps the most classic representation of this issue on Windows. In modern versions of Python, the O_NOINHERIT flag is set by default when opening file descriptors on Windows. Setting it on the opened file descriptor prevents it from being inherited by a child process. The equivalent O_CLOEXEC flag is also set by default in modern versions of Python when opening file descriptors on POSIX systems.

   It is worth mentioning that due to limitations with being able to set close_fds=True when redirecting stdin, stdout, or stderr in older versions of Python, setting the O_NOINHERIT flag isn’t sufficient for preventing files descriptors from being leaked when spawning subprocesses concurrently. Consider guarding all calls to subprocess.Popen with a threading.Lock instance to avoid this as an additional issue in older versions of Python.

2. On POSIX systems, it is possible to unlink() a file while it is still open in another thread or process. On Windows, in all versions of Python, non-O_TEMPORARY files are opened with FILE_SHARE_READ | FILE_SHARE_WRITE as their sharing mode. Omitting FILE_SHARE_DELETE prevents another thread or process from attempting to delete a file while it is still open.

The purpose of this library is to mask these behavioral differences across different platforms and in older versions of Python.

References

There have been multiple attempts to address the FILE_SHARE_DELETE issue within CPython itself but they unfortunately never succeeded in being integrated.

• bpo-12939: tempfile.NamedTemporaryFile not particularly useful on Windows

• bpo-14243: Support for opening files with FILE_SHARE_DELETE on Windows

• bpo-15244: Add new io.FileIO using the native Windows API

Starting in Python 3.7, close_fds=True is able to be set even when redirecting stdin, stdout, or stderr.

• bpo-19575: subprocess: on Windows, unwanted file handles are inherited by child processes in a multi-threaded application

• bpo-19764: subprocess: use PROC_THREAD_ATTRIBUTE_HANDLE_LIST with STARTUPINFOEX on Windows Vista

• bpo-33369: Removing Popen log files in threads is racy on Windows