Assorted filesystem related utility functions, some of which have been bloating cs.fileutils for too long.

Latest release 20250325: New update_linkdir() imported from my wpr script.

Module contents:

• atomic_directory(*da, **dkw): Decorator for a function which fills in a directory which calls the function against a temporary directory then renames the temporary to the target name on completion.

  Parameters:

  • infill_func: the function to fill in the target directory
  • make_placeholder: optional flag, default False: if true an empty directory will be make at the target name and after completion it will be removed and the completed directory renamed to the target name

• findup(dirpath: str, criterion: Union[str, Callable[[str], Any]]) -> str: Walk up the filesystem tree looking for a directory where criterion(fspath) is not None, where fspath starts at dirpath. Return the result of criterion(fspath). Return None if no such path is found.

  Parameters:

  • dirpath: the starting directory
  • criterion: a str or a callable accepting a str

  If criterion is a str, look for the existence of os.path.join(fspath,criterion).

  Example:

  # find a directory containing a `.envrc` file
  envrc_path = findup('.', '.envrc')
  
  # find a Tagger rules file for the Downloads directory
  rules_path = findup(expanduser('~/Downloads', '.taggerrc')
  

• fnmatchdir(dirpath, fnglob): Return a list of the names in dirpath matching the glob fnglob.

• Class FSPathBasedSingleton(cs.obj.SingletonMixin, HasFSPath, cs.deco.Promotable): The basis for a SingletonMixinbased onrealpath(self.fspath)`.

FSPathBasedSingleton.__init__(self, fspath: Optional[str] = None, lock=None): Initialise the singleton:

On the first call:

• set .fspath to self._resolve_fspath(fspath)
• set ._lock to lock (or cs.threads.NRLock() if not specified)

FSPathBasedSingleton.fspath_normalised(fspath: str): Return the normalised form of the filesystem path fspath, used as the key for the singleton registry.

This default returns realpath(fspath).

As a contracting example, the cs.ebooks.kindle.classic.KindleTree class tries to locate the directory containing the book database, and returns its realpath, allowing some imprecision.

FSPathBasedSingleton.promote(obj): Promote None or str to a CalibreTree.

• Class HasFSPath: A mixin for an object with a .fspath` attribute representing a filesystem location.

  The __init__ method just sets the .fspath attribute, and need not be called if the main class takes care of that itself.

HasFSPath.__init__(self, fspath): Save fspath as .fspath; often done by the parent class.

HasFSPath.fnmatch(self, fnglob): Return a list of the names in self.fspath matching the glob fnglob.

HasFSPath.listdir(self): Return os.listdir(self.fspath).

HasFSPath.pathto(self, *subpaths): The full path to subpaths, comprising a relative path below self.fspath. This is a shim for os.path.join which requires that all the subpaths be relative paths.

HasFSPath.shortpath: The short version of self.fspath.

• is_valid_rpath(rpath, log=None) -> bool: Test that rpath is a clean relative path with no funny business.

  This is a Boolean wrapper for validate_rpath().

• longpath(path, prefixes=None): Return path with prefixes and environment variables substituted. The converse of shortpath().

• needdir(dirpath, mode=511, *, use_makedirs=False, log=None) -> bool: Create the directory dirpath if missing. Return True if the directory was made, False otherwise.

  Parameters:

  • dirpath: the required directory path
  • mode: the permissions mode, default 0o777
  • log: log makedirs or mkdir call
  • use_makedirs: optional creation mode, default False; if true, use os.makedirs, otherwise os.mkdir

• rpaths(dirpath='.', **scan_kw): A shim for scandirtree to yield relative file paths from a directory.

  Parameters:

  • dirpath: optional top directory, default '.'

  Other keyword arguments are passed to scandirtree.

• scandirpaths(dirpath='.', **scan_kw): A shim for scandirtree to yield filesystem paths from a directory.

  Parameters:

  • dirpath: optional top directory, default '.'

  Other keyword arguments are passed to scandirtree.

• scandirtree(dirpath='.', *, include_dirs=False, name_selector=None, only_suffixes=None, skip_suffixes=None, sort_names=False, follow_symlinks=False, recurse=True): Generator to recurse over dirpath, yielding (is_dir,subpath) for all selected subpaths.

  Parameters:

  • dirpath: the directory to scan, default '.'
  • include_dirs: if true yield directories; default False
  • name_selector: optional callable to select particular names; the default is to select names not starting with a dot ('.')
  • only_suffixes: if supplied, skip entries whose extension is not in only_suffixes
  • skip_suffixes: if supplied, skip entries whose extension is in skip_suffixes
  • sort_names: option flag, default False; yield entires in lexical order if true
  • follow_symlinks: optional flag, default False; passed to scandir
  • recurse: optional flag, default True; if true, recurse into subdrectories

• shortpath(fspath, prefixes=None, *, collapseuser=False, foldsymlinks=False): Return fspath with the first matching leading prefix replaced.

  Parameters:

  • prefixes: optional list of (prefix,subst) pairs
  • collapseuser: optional flag to enable detection of user home directory paths; default False
  • foldsymlinks: optional flag to enable detection of convenience symlinks which point deeper into the path; default False

  The prefixes is an optional iterable of (prefix,subst) to consider for replacement. Each prefix is subject to environment variable substitution before consideration. The default prefixes is from SHORTPATH_PREFIXES_DEFAULT: (('$HOME/', '~/'),).

• update_linkdir(linkdirpath: str, paths: Iterable[str], trim=False): Update linkdirpath with symlinks to paths. Remove unused names if trim. Return a mapping of names in linkdirpath to absolute forms of paths.

• validate_rpath(rpath: str): Test that rpath is a clean relative path with no funny business; raise ValueError if the test fails.

  Tests:

  • not empty or '.' or '..'
  • not an absolute path
  • normalised
  • does not walk up out of its parent directory

  Examples:

  >>> validate_rpath('')
  False
  >>> validate_rpath('.')
  

Release Log

Release 20250325: New update_linkdir() imported from my wpr script.

Release 20241122:

• FSPathBasedSingleton: add a .promote method to promote a filesystem path to an instance.
• FSPathBasedSingleton: new fspath_normalised(fspath) class method to produce a normalised form of the fspath for use as the key to the singleton registry.

Release 20241007: FSPathBasedSingleton.init: use an NRLock for the default lock, using a late import with fallback to Lock.

Release 20241005: needdir: now returns True if the directory was made, False if it already existed.

Release 20240630: FSPathBasedSingleton: recent Pythons seem to check that init returns None, subclasses must test another way.

Release 20240623:

• shortpath(foldsymlinks=True): only examine symlinks which have clean subpaths in their link text - this avoids junk and also avoids stat()ing links which might be symlinks to mount points which might be offline.
• scandirtree: clean up the logic, possibly fix repeated mention of directories.

Release 20240522: shortpath: new collapseuser=False, foldsymlinks=False parameters, rename DEFAULT_SHORTEN_PREFIXES to SHORTPATH_PREFIXES_DEFAULT.

Release 20240422: New scandirtree scandir based version of os.walk, yielding (is_dir,fspath). New shim scandirpaths.

Release 20240412: HasFSPath: explain that the init is optional in the docstring.

Release 20240316: Fixed release upload artifacts.

Release 20240201:

• FSPathBasedSingleton: drop the default_factory parameter/attribute, let default_attr specify a callable.
• Singleton._resolve_fspath: fix reference to class name.

Release 20231129:

• HasFSPath: new listdir method.
• HasFSPath.pathto: accept multiple relative subpaths.
• FSPathBasedSingleton: accept cls.FSPATH_FACTORY as a factory function for the default fspath, makes it possible to defer the path lookup.
• Replace is_clean_subpath with validate_rpath/is_valid_rpath pair.

Release 20230806:

• Reimplement fnmatchdir using fnmatch.filter.
• No longer claim Python 2 compatibility.

Release 20230401: HasFSPath.shortpath: hand call before .fspath set.

Release 20221221: Replace use of cs.env.envsub with os.path.expandvars and drop unused environ parameter.

Release 20220918:

• FSPathBasedSingleton.init: return True on the first call, False on subsequent calls.
• FSPathBasedSingleton.init: probe dict for '_lock' instead of using hasattr (which plays poorly this early on with classes with their own getattr).
• needdir: accept optional log parameter to log mkdir or makedirs.
• HasFSPath: add a default str.

Release 20220805: Doc update.

Release 20220530: FSPathBasedSingleton._resolve_fspath: new envvar and default_attr parameters.

Release 20220429:

• New HasFSPath and FSPathBasedSingleton.
• Add longpath and shortpath from cs.fileutils.
• New is_clean_subpath(subpath).
• New needdir(path).
• New fnmatchdir(dirpath,fnglob) pulled out from HasFSPath.fnmatch(fnglob).

Release 20220327: New module cs.fs to contain more filesystem focussed functions than cs.fileutils, which is feeling a bit bloated.