kisstdlib 0.0.12

System programming with conceptually and algebraically simple primitives.

Table of Contents

(Click me to see it.)

• What is kisstdlib?
• Parts and pieces
• Usage
  • describe-forest
• Development: ./test-example.sh [--help] [--wine]

What is kisstdlib?

kisstdlib is a set of modules for the Python programming language, designed mostly for system programming with a touch of everything else, that aims to enhance the standard experience while keeping everything it does conceptually and algebraically simple.

This library borrows heavily from the Haskell programming language, where appropriate. (The Python's bundled modules do that too in many places, but kisstdlib borrows more.)

In short, kisstdlib mostly implements useful extensions for Python's bundled modules, keeping the APIs backward compatible and naming things in the same style. The only exception to this is, yet unpublished, io.loop module, which implements Free-Monad-based IO.

At the moment, kisstdlib is an alpha work in progress software with an unstable API.

Parts and pieces

kisstdlib consists of:

• a bunch of Python modules under ./kisstdlib;

  they are pretty well-documented there, though there's no generated sphinx docs yet;

• a bunch of human-readable examples using those modules;

• some useful thin-wrapper programs over kisstdlib functions, useful for writing whole-program and/or fixed-output/extensional-equality tests;

  the documentation of which gets rendered into the following "Usage" section;

• as well as some testing-related infrastructure built on top of this, in ./devscript.

Usage

describe-forest

Produce a plain-text recursive deterministic find/ls/stat-like description of given file and/or directory inputs.

The output format is designed to be descriptive and easily diffable while also producing minimally dissimilar outputs for similar inputs, even when those inputs contain lots of symlinks and/or hardlinks. I.e., essentially, this is an alternative to ls -lR and/or find . -exec ls -l {} \; which generates outputs that change very little when files with multiple symlinks and/or hardlinks change.

This is most useful for testing code that produces filesystem trees.

The most verbose output format this program can produce, for a single input file

describe-forest --full path/to/README.md

looks as follows:

. reg mode 644 mtime [2025-01-01 00:01:00] size 4096 sha256 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

Note how both the path to and the name of the file do not appear in the output. This is what you would want for doing things like

if ! diff -U 0 <(describe-forest --full v1/path/to/README.md) <(describe-forest --full v2/path/to/README.md) ; then
    echo "output changed between versions!" >&2
    exit 1
fi

which this program is designed for.

For a single input directory

describe-forest --full path/to/dir

the output looks similar to this:

. dir mode 700 mtime [2025-01-01 00:00:00]
afile.jpg reg mode 600 mtime [2025-01-01 00:01:00] size 4096 sha256 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
sub dir mode 700 mtime [2025-01-01 00:03:00]
sub/afile-hardlink.jpg ref ==> afile.jpg
sub/afile-symlink.jpg sym mode 777 mtime [2025-01-01 00:59:59] -> ../afile.jpg
sub/zfile-hardlink.jpg reg mode 600 mtime [2025-01-01 00:02:00] size 256 sha256 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
unix-socket ??? mode 600 mtime [2025-01-01 01:00:00] size 0
zfile.jpg ref ==> sub/zfile-hardlink.jpg

Hardlinks, which are denoted by refs above, are processed as follows:

• each new file encountered in lexicographic walk is rendered fully,
• files with repeated dev+inode numbers are rendered by emitting ref ==> followed by the full path (or ref => followed by the relative path, with --relative-hardlink) to the previously encountered element.

This way, renaming a file in the input changes at most two lines.

Symlinks are rendered by simply emitting the path they store, unless --follow-symlinks is given, in which case the targets they point to get rendered instead.

Multiple inputs get named by numbering them starting from "0". Thus, for instance, running this program with the same input file given twice

describe-forest --full path/to/README.md path/to/README.md

produces something like:

0 reg mode 600 mtime [2025-01-01 00:01:00] size 4096 sha256 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
1 ref ==> 0

And giving the same directory with that file inside twice produces:

0 dir mode 700 mtime [2025-01-01 00:00:00]
0/afile.jpg reg mode 600 mtime [2025-01-01 00:01:00] size 4096 sha256 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
1 dir mode 700 mtime [2025-01-01 00:00:00]
1/afile.jpg ref ==> 0/afile.jpg

In its default output format, though, the program emits only sizes and sha256s, when appropriate:

. dir
afile.jpg reg size 4096 sha256 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

which is what you would usually want for writing tests. Though, if you are testing rsync or some such, feel free to use other options described below.

See devscript directory in kisstdlib's repository for examples of some shell machinery that uses this to implement arbitrary-program fixed-output tests, which is a nice and simple way to test programs by testing their outputs against outputs of different versions of themselves.

Also, internally, this programs is actually a thin wrapper over describe_forest function of kisstdlib.fs Python module, which can be used with pytest or some such.

• positional arguments:

  • PATH : input directories

• options:

  • --version : show program's version number and exit
  • -h, --help : show this help message and exit
  • --markdown : show --help formatted in Markdown
  • --numbers : emit number prefixes even with a single input PATH
  • --literal : emit paths without escaping them even when they contain special symbols
  • --modes : emit file modes
  • --mtimes : emit file mtimes
  • --no-sizes : do not emit file sizes
  • --full : an alias for --mtimes --modes
  • --relative, --relative-hardlinks : emit relative paths when emitting refs
  • -L, --dereference, --follow-symlinks : follow all symbolic links; replaces all sym elements of the output with description of symlink targets
  • --time-precision INT : time precision (as a negative power of 10); default: 0, which means seconds, set to 9 for nanosecond precision
  • --hash-length INT : cut hashes by taking their prefixes of this many characters; default: print them whole

Development: ./test-example.sh [--help] [--wine]

Check that all kisstdlib/example/*.py are fixed-output.