Assorted filesystem related utility functions, some of which have been bloating cs.fileutils for too long.

Short summary:

• atomic_directory: RUn code in a temporary directory, which will be renamed to the target directory if no exception occurs.

• findup: Walk up the filesystem tree looking for a directory where criterion(fspath) is not None, where fspath starts at dirpath. Return the result of criterion(fspath). Return None if no such path is found.

• fnmatchdir: Return a list of the names in dirpath matching the glob fnglob.

• FSPathBasedSingleton: The basis for a SingletonMixin based on realpath(self.fspath).

• HasFSPath: A mixin for an object with a .fspath attribute representing a filesystem location.

• is_valid_rpath: Test that rpath is a clean relative path with no funny business.

• longpath: Return path with prefixes and environment variables substituted. The converse of shortpath().

• needdir: Create the directory dirpath if missing. Return True if the directory was made, False otherwise.

• RemotePath: A representation of a remote filesystem path (local if host is None).

• remove_protecting: Remove the file at rmpath while protecting safepath from destruction.

• rpaths: A shim for scandirtree to yield relative file paths from a directory.

• scandirpaths: A shim for scandirtree to yield filesystem paths from a directory.

• scandirtree: Generator to recurse over dirpath, yielding (is_dir,subpath) for all selected subpaths.

• shortpath: Return fspath with the first matching leading prefix replaced.

• update_linkdir: Update linkdirpath with symlinks to paths. Remove unused names if trim. Return a mapping of names in linkdirpath to absolute forms of paths.

• validate_rpath: Test that rpath is a clean relative path with no funny business; raise ValueError if the test fails.

Functions

atomic_directory(dirpath_or_func: Union[str, Callable], *, make_placeholder=False)

RUn code in a temporary directory, which will be renamed to the target directory if no exception occurs.

Parameters:

• make_placeholder: optional flag, default False: if true an empty directory will be make at the target name and after completion it will be removed and the completed directory renamed to the target name

This may be used as a context manager or as a decorator.

As a contextmanager:

with atimoc_directory(target_directory) as tmpdirpath:
    do work inside tmpdirpath

This will rename the tmpdirpath to target_directory on exit.

As a decorator:

@atomic_directory
def produce_dir_content(tmpdirpath,.....):

This produces a function which will accept a target directory path as its first argument and calls produce_dir_content with the temporary directory. On return the temporary directory will be renamed to the target directory.

findup(dirpath: str, criterion: Union[str, Callable[[str], Any]]) -> str | None

Walk up the filesystem tree looking for a directory where criterion(fspath) is not None, where fspath starts at dirpath. Return the result of criterion(fspath). Return None if no such path is found.

Parameters:

• dirpath: the starting directory
• criterion: a str or a callable accepting a str

If criterion is a str, look for the existence of os.path.join(fspath,criterion).

Example:

# find a directory containing a `.envrc` file
envrc_path = findup('.', '.envrc')

# find a Tagger rules file for the Downloads directory
rules_path = findup(expanduser('~/Downloads', '.taggerrc')

fnmatchdir(dirpath, fnglob)

Return a list of the names in dirpath matching the glob fnglob.

is_valid_rpath(rpath, log=None) -> bool

Test that rpath is a clean relative path with no funny business.

This is a Boolean wrapper for validate_rpath().

longpath(path, prefixes=None)

Return path with prefixes and environment variables substituted. The converse of shortpath().

needdir(dirpath, mode=511, *, use_makedirs=False, log=None) -> bool

Create the directory dirpath if missing. Return True if the directory was made, False otherwise.

Parameters:

• dirpath: the required directory path
• mode: the permissions mode, default 0o777
• log: log makedirs or mkdir call
• use_makedirs: optional creation mode, default False; if true, use os.makedirs, otherwise os.mkdir

remove_protecting(rmpath, safepath)

Remove the file at rmpath while protecting safepath from destruction.

This is for situations such as "merging" two equivalent files where the "source" file (rmpath) is to be removed, leaving the destination file (safepath). It can be that these are the same file (not merely links to the same file, but the same link/name); this is surprisingly hard to detect, and removing the source will then destroy the destination.

Instead of checking carefully and unreliably, we instead make a "safe" hard link of the destination, remove the source, then try to link the safe link back to the destination. If that succeeds, we have recovered from the destruction. If that fails with FileExistsError then the destruction did not occur. Both are good. Other exceptions are released with an accompanying note about the path to the "safe" link.

Example use:

if srcpath != dstpath and same_content(srcpath, dstpath):
    remove_protecting(srcpath, dstpath)

rpaths(dirpath='.', **scan_kw)

A shim for scandirtree to yield relative file paths from a directory.

Parameters:

• dirpath: optional top directory, default '.'

Other keyword arguments are passed to scandirtree.

scandirpaths(dirpath='.', **scan_kw)

A shim for scandirtree to yield filesystem paths from a directory.

Parameters:

• dirpath: optional top directory, default '.'

Other keyword arguments are passed to scandirtree.

scandirtree(dirpath='.', *, include_dirs=False, name_selector=None, only_suffixes=None, skip_suffixes=None, sort_names=False, follow_symlinks=False, recurse=True)

Generator to recurse over dirpath, yielding (is_dir,subpath) for all selected subpaths.

Parameters:

• dirpath: the directory to scan, default '.'
• include_dirs: if true yield directories; default False
• name_selector: optional callable to select particular names; the default is to select names not starting with a dot ('.')
• only_suffixes: if supplied, skip entries whose extension is not in only_suffixes
• skip_suffixes: if supplied, skip entries whose extension is in skip_suffixes
• sort_names: option flag, default False; yield entires in lexical order if true
• follow_symlinks: optional flag, default False; passed to scandir
• recurse: optional flag, default True; if true, recurse into subdrectories

shortpath(fspath, prefixes=None, *, collapseuser=False, foldsymlinks=False)

Return fspath with the first matching leading prefix replaced.

Parameters:

• prefixes: optional list of (prefix,subst) pairs
• collapseuser: optional flag to enable detection of user home directory paths; default False
• foldsymlinks: optional flag to enable detection of convenience symlinks which point deeper into the path; default False

The prefixes is an optional iterable of (prefix,subst) to consider for replacement. Each prefix is subject to environment variable substitution before consideration. The default prefixes is from SHORTPATH_PREFIXES_DEFAULT: (('$HOME/', '~/'),).

update_linkdir(linkdirpath: str, paths: Iterable[str], trim=False)

Update linkdirpath with symlinks to paths. Remove unused names if trim. Return a mapping of names in linkdirpath to absolute forms of paths.

My example use is maintaining a small directory of wallpapers to shuffle, selected from a reference image tree.

validate_rpath(rpath: str)

Test that rpath is a clean relative path with no funny business; raise ValueError if the test fails.

Tests:

• not empty or '.' or '..'
• not an absolute path
• normalised
• does not walk up out of its parent directory

Examples:

>>> validate_rpath('')
False
>>> validate_rpath('.')

Classes

class FSPathBasedSingleton(cs.obj.SingletonMixin, HasFSPath, cs.deco.Promotable)

The basis for a SingletonMixin based on realpath(self.fspath).

FSPathBasedSingleton.__init__(self, fspath: Optional[str] = None, lock=None)

Initialise the singleton:

On the first call:

• set .fspath to self._resolve_fspath(fspath)
• set ._lock to lock (or cs.threads.NRLock() if not specified)

FSPathBasedSingleton.fspath_normalised(fspath: str)

Return the normalised form of the filesystem path fspath, used as the key for the singleton registry.

This default returns realpath(fspath).

As a contracting example, the cs.ebooks.kindle.classic.KindleTree class tries to locate the directory containing the book database, and returns its realpath, allowing some imprecision.

FSPathBasedSingleton.promote(obj)

Promote None or str to a CalibreTree.

class HasFSPath(os.PathLike)

A mixin for an object with a .fspath attribute representing a filesystem location.

The __init__ method just sets the .fspath attribute, and need not be called if the main class takes care of that itself.

HasFSPath.__init__(self, fspath)

Save fspath as .fspath; often done by the parent class.

HasFSPath.__fspath__(self)

Return the filesystem path string, for os.PathLike.

HasFSPath.fnmatch(self, fnglob)

Return a list of the names in self.fspath matching the glob fnglob.

HasFSPath.listdir(self)

Return os.listdir(self.fspath).

HasFSPath.pathto(self, *subpaths)

The full path to subpaths, comprising a relative path below self.fspath. This is a shim for os.path.join which requires that all the subpaths be relative paths.

HasFSPath.shortpath

<property object at 0x10219a4d0>

class RemotePath(RemotePath, HasFSPath, cs.deco.Promotable)

A representation of a remote filesystem path (local if host is None).

This is useful for things like rsync targets.

RemotePath.__init__(self, host, fspath)

dummy init since namedtuple does not have one

RemotePath.__str__(self)

Return the string form of this path.

RemotePath.from_str(pathspec: <staticmethod(<function RemotePath.str at 0x1021867a0>)>)

Produce a RemotePathfrompathspec, a path with an optional leading [user@]rhost:` prefix.

RemotePath.from_tuple(cls, host_fspath: tuple)

Produce a RemotePathfromhost_fspath, a (host,fspath)` 2-tuple.

RemotePath.str(host, fspath)

Return the string form of a remote path.