filename-templates 1.4.2

Make filenames from string templates

Filename Templates

Make filenames from string templates. This package exposes the FileNames class, which keeps a list of filenames and provides a wrapper around string.format with some bells and whisles to make the syntax super nice.

I wrote this to keep track of filenames during data analysis projects, where there are many files, which names follow a standard pattern. For example: data-day001.csv data-day002.csv data-day003.csv. Processing these files may produce: data-day001-processed.csv data-day002-processed.csv data-day003-processed.csv. In these cases, it is good practice to define the templates for these filenames once, for example in a configuration file, and re-use them in the different analysis scripts.

Installation

either through pip:

pip install filename-templates

or from the repository:

pip install .

To run the tests, install the pytest package first, and then:

pytest

Usage

Use the add method to add new filenames. You specify a short "alias" for them, which you can use to retrieve the full filename later:

>>> from filename_templates import FileNames
>>> fname = FileNames()
>>> fname.add('my_file', '/path/to/file1')
>>> fname.my_file
PosixPath('/path/to/file1')

Filenames can also be templates that can be used to generate filenames for different subjects, conditions, etc.:

>>> fname = FileNames()
>>> fname.add('epochs', '/data/{subject}/{cond}-epo.fif')
>>> fname.epochs(subject='sub001', cond='face')
PosixPath('/data/sub001/face-epo.fif')

Templates can contain placeholders in the way string.format allows, including formatting options:

>>> fname = FileNames()
>>> fname.add('epochs', '/data/sub{subject:03d}/{cond}-epo.fif')
>>> fname.epochs(subject=1, cond='face')
PosixPath('/data/sub001/face-epo.fif')

If a placeholder happens to be the alias of a file that has been added earlier, the placeholder is automatically filled:

>>> fname = FileNames()
>>> fname.add('subjects', '/data/subjects_dir')
>>> fname.add('epochs', '{subjects}/{subject}/{cond}-epo.fif')
>>> fname.epochs(subject='sub001', cond='face')
PosixPath('/data/subjects_dir/sub001/face-epo.fif')

If all placeholders could be automatically filled, no brackets () are required when accessing it:

>>> fname = FileNames()
>>> fname.add('subjects', '/data/subjects_dir')
>>> fname.add('fsaverage', '{subjects}/fsaverage-src.fif')
>>> fname.fsaverage
PosixPath('/data/subjects_dir/fsaverage-src.fif')

When declaring filenames, you can tag them with mkdir=True. Whenever a filename that is tagged in this manner is accessed, the parent directory will be created if it doesn't exist yet.

>>> import os.path
>>> fname = FileNames()
>>> fname.add('my_file', 'path/to/file1', mkdir=True)
>>> os.path.exists(fname.my_file.parent)
True

If computing the file path gets more complicated than the cases above, you can supply your own function. When the filename is requested, your function will get called with the FileNames object as first parameter, followed by any parameters that were supplied along with the request:

>>> from pathlib import Path
>>> fname = FileNames()
>>> fname.add('basedir', '/data/subjects_dir')
>>> def my_function(files, subject):
...     if subject == 1:
...         return files.basedir / '103hdsolli.fif'
...     else:
...         return files.basedir / f'{subject}.fif'
>>> fname.add('complicated', my_function)
>>> fname.complicated(subject=1)
PosixPath('/data/subjects_dir/103hdsolli.fif')

When many of your filenames contain the same placeholders, it may be convenient to pre-fill the placeholder once with fname.fill_placeholder(placeholder=value), after which it will be automatically filled in the future:

>>> fname = FileNames()
>>> fname.add('epochs', 'sub-{subject:02d}_ses-{session:02d}_{cond}-epo.fif')
>>> fname.add('evoked', 'sub-{subject:02d}_ses-{session:02d}_{cond}-ave.fif')
>>> fname.fill_placeholder('subject', 1)
>>> fname.fill_placeholder('session', 2)
>>> fname.evoked(cond='visual')
PosixPath('sub-01_ses-02_visual-ave.fif')

Pre-filled placeholders can still be overwritten by manually specifying them when using a filename:

>>> fname = FileNames()
>>> fname.add('epochs', 'sub-{subject:02d}-epo.fif')
>>> fname.fill_placeholder('subject', 1)
>>> fname.epochs(subject=2)
PosixPath('sub-02-epo.fif')

You can undo filling in placeholders using fname.clear_placeholder(placeholder), after which it will again need to be filled in manually when using the filename.

>>> fname = FileNames()
>>> fname.add('epochs', 'sub-{subject:02d}-epo.fif')
>>> fname.fill_placeholder('subject', 1)
>>> fname.clear_placeholder('subject')
>>> fname.epochs()
Traceback (most recent call last):
   ...
ValueError: Cannot construct filename, because these parameters are missing: {'subject'}

Instead of adding one filename at a time, you can add a dictionary of them all at once:

>>> fname = FileNames()
>>> fname_dict = dict(
...     subjects = '/data/subjects_dir',
...     fsaverage = '{subjects}/fsaverage-src.fif',
... )
>>> fname.add_from_dict(fname_dict)
>>> fname.fsaverage
PosixPath('/data/subjects_dir/fsaverage-src.fif')

The returned filenames are of type pathlib.Path, which offers a bunch of convenience methods related to filenames that you wouldn't get with ordinary strings. They can be used in all locations were you would otherwise use a string filename. However, if you want an ordinary string, there are several ways of doing so. One is to cast the filename to a string:

>>> fname = FileNames()
>>> fname.add('my_file', '/path/to/file1')
>>> str(fname.my_file)
'/path/to/file1'

Another way is to, when adding a filename, to specify that the filename should always be returned as string:

>>> fname = FileNames()
>>> fname.add('my_file', '/path/to/file1', as_str=True)
>>> fname.my_file
'/path/to/file1'

If you want all of your filenames to be strings, always, then you can pass as_str=True when creating the FileNames object:

>>> fname = FileNames(as_str=True)
>>> fname.add('my_file', '/path/to/file1')
>>> fname.my_file
'/path/to/file1'

Obviously this also works when the filename contains placeholders:

>>> fname = FileNames(as_str=True)
>>> fname.add('my_file', '/path/to/file{subject:d}')
>>> fname.add('my_file', '/path/to/file{subject:d}')
>>> fname.my_file(subject=1)
'/path/to/file1'

The filenames object should be pickleable as long as you don't use custom functions to generate the filenames:

>>> import pickle
>>> fname = FileNames()
>>> fname.add('normal_file', 'path/to/file1')
>>> fname.add('template', 'path/to/{bla}')
>>> len(pickle.dumps(fname))
267

Author

Marijn van Vliet ([e-mail])