viat 0.10.5

A tool for managing virtual file attributes

Viat

A tool for managing virtual file attributes.

The essence of the tool is that the attributes are stored in plain text formats that can be edited and committed to version control. The main unit of operation is a vault, which is determined by .viat subdirectory. In the simplest case, this subdirectory contains config.toml, storage.toml and possibly schema.json. The source repository contains several examples of how this project can be useful.

In short, in an empty vault, the command

viat set file.pdf --raw attr value

puts the following into storage.toml:

["file.pdf"]
attr = "value"

Determining which files are tracked by Viat is done via tracker providers, while storing the attributes is done via storage providers. Both protocols are very general and new providers can easily be added.

Table of contents

• Usage
• Installation
• Motivation

See also the documentation on ReadTheDocs

Usage

We give a usage tutorial here; refer to the online documentation or to the man page viat(1) for more details. The examples directory contains more thorough examples like scripts for managing dotfiles and bibliographies.

Command-line usage

First, a vault must be initialized:

viat init

The vault is determined by a .viat subfolder that contains config.toml and storage.toml files (JSON is also supported for both). We can immediately set attributes for any file on the file system:

$ viat update tractatus.pdf '{"author": "Ludwig Wittgenstein", "year": 1921}'
Warning: File 'tractatus.pdf' is not being tracked.
{"author": "Ludwig Wittgenstein", "year": 1921}

All stored attributes for the file get printed; in this case the only stored attributes are those we have just added. We also get a warning saying that the vault's tracker does not know about this file.

The role of the tracker is to enumerate the files that are explicitly tracked by the vault. The default glob-based tracking provider requires explicit patterns. We can track all PDF files in the root of the vault using the following configuration:

[tracker.glob]
patterns = ["*.pdf"]

With this, we can add new properties without warnings:

$ viat set tractatus.pdf rating 4
{"author": "Ludwig Wittgenstein", "year": 1921, "rating": 4}

The above worked because "true" is a valid JSON value; if we were to set a string instead, we would have to escape it in quotes, which is inconvenient. Instead, we can treat the value as a string by passing the --raw flag:

$ viat set --raw tractatus.pdf publisher 'Annalen der Naturphilosophie'
...

Scripting

Tracking is useful for ensuring consistency with the file system, but also for shell scripting.

For example, the following command produces a table of variables:

$ viat shell-export
path=tractatus.pdf publisher='Annalen der Naturphilosophie' rating=4 author='Ludwig Wittgenstein' year=1921

This can be utilized in bash as follows:

viat shell-export | while read line; do
    eval "export $line"
    # The attributes are now exported as variables
done

In fish shell this is even simpler:

for line in (viat shell-export)
    eval "export $line"
    # The attributes are now exported as variables
end

If we add another file, zarathustra.pdf, and if the tracker lists it after tractatus.pdf, then Viat would try to reset the missing attributes to avoid reusing variables from the previous loop iteration:

$ viat shell-export
path=tractatus.pdf publisher='Annalen der Naturphilosophie' rating=4 author='Ludwig Wittgenstein' year=1921
path=zarathustra.pdf publisher= rating= author= year=

Schemas

At this point, .viat/storage.toml should now look as follows:

["tractatus.pdf"]
author = "Ludwig Wittgenstein"
year = 1921
rating = 4
publisher = "Annalen der Naturphilosophie"

It makes sense to utilize JSON schemas. Let us add the following to .viat/schema.json:

{
  "type":"object",
  "properties": {
    "year": {"type": "number"}
  }
}

Now we can no longer set the year to anything that is not a number:

$ viat set tractatus.pdf --raw year string
Error: Validation error for 'tractatus.pdf': data.year must be number.

If we manually change the year to "string", we will get a warning when loading the vault:

$ viat get tractatus.pdf year
Warning: Validation error in stored data for 'tractatus.pdf': data.year must be number.
4

(Re)moving files

If we move tractatus.pdf to book.pdf, viat will no longer know about it:

$ viat get book.pdf rating
Warning: File 'book.pdf' is not being tracked.
Error: Attribute 'rating' has not been set for 'book.pdf'.

Such discrepancies are straightforward to determine:

$ viat stale
tractatus.pdf
$ viat tracked --no-data
book.pdf

We provide the helpers viat mv and viat rm, but otherwise avoid being too clever.

Programmatic usage

The programmatic usage should be straightforward. Here is a brief continuation of the above example:

vault = autoload_vault()

assert vault.tracker.is_tracked('tractatus.pdf')

# This context manager validates and writes the file upon exiting.
# If no mutators have been used, no validation and writing is performed.
with vault.storage as conn:
    # The inner lock-based context managers allow either read-only or read-write operations.

    # Readers are Mapping instances.
    with conn.get_reader('tractatus.pdf') as reader:
        print(mut['year'])

    # Mutators are MutableMapping instances.
    with conn.get_mutator('tractatus.pdf') as mut:
        mut['year'] = 1921

Refer to the API reference for details.

Installation

The viat PyPI package contains the core programmatic API.

To install the viat executable for the current user, you can use pipx or uv:

pipx install viat
uv tool install viat

The git tracker requires the git extra.

To install from GitHub, you can use uv:

uv tool install viat --from git+https://github.com/v--/viat

Sometimes a particular feature branch needs to be tested. For installing a fixed revision (i.e. common/branch/tag), the following should work (if extra-name is needed, use viat@rev[extra-name]):

uv tool install viat --from git+https://github.com/v--/viat@rev

To install viat from a cloned repository, you can use the following:

uv sync
uv build --wheel
# Once built, we can install using uv
uv tool install viat --from dist/*.whl
# or pipx
pipx install --include-deps dist/*.whl

Tasks inside the repository like linting and testing use are summarized in poe.toml (configuration for poethepoet). For example, building the documentation requires some hacks internally, but is wrapped in a single command:

uv run poe docs-build

[!TIP] An AUR package is available for reference, as well as a GitHub Action. If you are packaging this for some other package manager, consider using PEP-517 tools as shown in this PKGBUILD file.

Motivation

When managing lots of files, there comes a point when metadata needs to be attached to them somehow.

• Different file systems offer extended file attributes. Unfortunately, poor software support reduces their utility. For example, curl --xattr <url> will record some attributes, but they will be lost on copy (with GNU cp at least) and will not be tracked by git.

• git attributes are obviously supported by git, but other tools have to consult git in order to use them. Furthermore, there is no convenient mechanism for setting git attributes.

• XMP (extensible metadata platform) files are designed to be used by arbitrary tools and can be easily tracked using version control, but are cumbersome to manage.

Perhaps I am missing some other approaches, but at this point it should be clear that there is no established convenient way to manage file metadata. A long time ago I wrote a small script that tracked "virtual" attributes across a directory by putting them into a single JSON file. At some point I decided to refine the script, and so Viat was born.

In the meantime, a somewhat similar ufa ("user file attributes") tool appeared. It is still quite different:

• ufa does not expose a programmatic API.
• ufa is meant to be better integrated into the ambient system, supporting FUSE and a Nautilus extension.
• ufa uses SQLite databases (unlike our plain text files).
• ufa does not support validation out-of-the-box.