skilleter-thingy 0.3.34

A collection of useful utilities, mainly aimed at making Git more friendly

Thingy

Licence: GPL v3

Author: John Skilleter v0.99

A collection of shell utilities and configuration stuff for Linux and MacOS. Untested on other operating systems.

Permanently (for the foreseeable future!) in a beta stage - usable, with a few rough edges, and probably with bugs when used in way I'm not expecting!

Install from PyPi via pip or pipx as the skilleter-thingy package; for instance - pipx install skilleter_thingy

The commands are categorised into:

• Git working tree management
• Miscellaneous commands
• Git Extensions
• General-purpose commands

IMPORTANT

The readable utility (for cleaning log files) has been moved into a standalone package skilleter-readable.

If you already have a version installed from Skilleter-Thingy then, before installing the stand-alone version, you should update skilleter-thingy to the latest version then remove the ~/.local/share/pipx/venvs/skilleter-readable/bin/readable symlink.

Git Working Tree Management

Multigit

Multigit is a tool for managing a collection of related git working trees.

This is intended for use in a situation where you have a collection of related git working trees organised in a directory hierarchy which aren't managed using git submodules or any other tool. It allows you to run git commands on multiple working trees at once, without navigating around the different working trees to do so and to select which working trees commands are run in.

For ease of use it is recommended that you create an alias for multigit called mg. How you do this is dependent on the shell you use, for instance in Bash, you could create an alias in your ~/.bashrc file:

alias mg="multigit $@"

Initialisation

To use multigit, start by creating working trees for the repositories that you want to use in a directory tree - the working trees can be at different levels or even nested, for example:

multigit_tree
|
+----working tree 1
|
+---+subdirectory
|   |
|   +----working tree 2
|   |
|   +----working tree 3
|
+----working tree 4

Start by running ensuring that the default branch (e.g. main) is checked out in each of the working trees and, in the top-level directory, run multigit init to create the configuration file which, by default is called multigit.toml - this is just a text file that sets the configuration for each working tree in terms of name, origin, default branch, tags and location.

Multigit Command Line

The multigit command line format is:

multigit OPTIONS COMMAND

Where COMMAND is an internal multigit command if it starts with a + and is a git command otherwise (including the additional git commands described in this document).

By default, when multigit is invoked with a git command, it runs a the command in each of the working trees selected by the command line options passed to multigit (if no options are specified then the command is run in all the working trees.

The command takes a number of options that can be used to select the list of working trees that each of the subcommands that it supports runs in:

--repos REPO / -r REPO Allows a list of working trees to be specfied, either by path, name or a wildcard matching either.

--modified / -m Run only in working trees containing locally modified files

--branched / -b Run only in working trees where the current branch that is checked out is NOT the default branch

--tag TAG / -t TAG Run only in working trees that are tagged with the specified tag

--sub / -s Run only in working trees that are in the current directory tree.

--continue / -o Continue if a git command returns an error (by default, executation terminates when a command fails)

--path PATH / -C PATH Run as if the command was started in PATH instead of the current working directory

These options are AND-ed together, so specifying --modified --branched --tag WOMBAT will select only working trees that are modified AND branched AND tagged with WOMBAT, but the parameters to the --repos and --tag options are OR-ed together, so specifying --tag WOMBAT --tag EMU will select repos that are tagged as WOMBAT OR EMU.

Multigit tags are stored in the configuration file, not within the working tree and each working tree can have multiple tags.

Multigit Commands

Multigit supports a small list of subcommands, each of which are prefixed with a + to distinguish them from Git commands:

+clone REPO <BRANCH> - Clone REPO (which should contain a multigit configuration file), checking out BRANCH, if specified, then clone all the repos specified in the configuration, checking out the default branch in each one.

+init - Create or update the configuration file

+dir NAME - Given the name of a working tree, output relative path(s) to one or more matching working trees. If NAME does NOT contains wildcard characters (* or ?) then matching takes place as if * were prefixed and appended to it, otherwise, the wildcards are used as specified. If NAME is not specified then the location of the directory where the multigit configuration file resides is output.

+list - Return a list of the top level directories of each of the Git repos

+config - Print the name and location of the multigit configuration file.

+tag TAG - If no tag specified list tags applied to the specified working trees. If a tag is specified, then apply the tag to the specified working trees.

+untag TAG - Remove the tag from the specified working trees (do nothing if the tag is not applied in the first place).

+run COMMAND - Run the specified command in each of the specified working trees

+shell - Run an interactive shell in each of the specified working trees

+add REPO DIR - Clone REPO into the DIR directory and, if successful, add it to the multigit configuration

Any command not prefixed with + is run in each of the working trees (filtered by the various multigit options) as a git command.

For example; multigit -m commit -ab would run git commit -a in each of the working trees that is branched and contains modified files.

The +dir command can be used with shell aliases (or their equivalent in the user's shell of choice) to create an alias to run, for example, cd (multigit +dir "$@") (Bash) or cd (multigit +dir $argv) (for the Fish shell) that would cd to the top level directory.

Miscellaneous Git Utilities

gitprompt

Output a string containing colour-coded shell nesting level, current directory and git working tree status. It is intended to be used in the shell prompt; for instance, for Bash via adding:

export PS1=$(gitprompt)

to the ~/.bashrc file.

The appearance of the prompt is controlled by configuration settings in the .gitconfig file:

[prompt]
prefix = 0..2

The prefix value determines what git status information is shown in the prompt:

• 0 - No status information
• 1 - A single letter followed by a file count showing the number of stashed, untracked, modified, etc. files in the working tree
• 2 - As 1, but using a word, rather than a single letter

If a rebase, bisect or merge is process, this is also shown in the prompt.

The name of the repo and the current branch are also appended and the prompt is also colour-coded according to the state of the working tree.

Git Extensions

Due to the way that the git command works, these can be run as they were additional git subcommands, although, due to a limitation in git, the only things that does not work is the --help option where the command has to be run with a hyphen between git and the subcommand - for example git ca --help does not work, but git-ca --help does.

Branch Names

Where one of the git extensions takes an existing branch name as a parameter, the branch name can be abbreviated and the abbreviated form is expanded according to the following rules:

• If the specified branch name exactly matches an existing branch, tag or commit ID then that is used (this includes remote branches where appropriate, if no local branches match).
• Otherwise, the branch name is compared to existing branches (again, including remote branches where appropriate, if no local branches match) and, if the specified name uniquely partially matches an existing branch (optionally using * and ? wildcard characters) that branch is used. If it matches multiple branches than an error is reported.

For example, given a repo with the following branches:

    origin/wombat
    origin/platypus
    wombat
    emu
    battery
    chaos

Then:

• 'emu' will match 'emu'
• 'wombat' will match 'wombat' but not 'origin/wombat' since the local branch takes precedence
• 'at' will match both 'wombat' and 'battery' and will report an error
• 'pus' will match 'origin/platypus'

This is most useful where branches contain ticket numbers so, for instance given a branch called feature/SKIL-103 you can check it out using git co 103 assuming no other local branches contain 103 in their name.

Note that the concept of the default branch DEFAULT mentioned above only applies when using the multigit command, although some of the commands will treat branches called master or main as special cases (see the individual command documentation).

git br

List or delete branches that have been merged

    usage: git-br [-h] [--all] [--delete] [--path PATH] [branches ...]

    positional arguments:
      branches              Filter the list of branches according to one or more patterns

    options:
      -h, --help            show this help message and exit
      --all, -a             List all branches, including remotes
      --delete, -d          Delete the specified branch(es), even if it is the current one (list of branches to delete must be supplied as parameters)
      --path PATH, -C PATH  Run the command in the specified directory

git ca

Improved version of 'git commit --amend'. Updates files that are already in the commit and, optionally, adds and commits additional files.

    usage: git-ca [-h] [--added] [--all] [--everything] [--ignored] [--patch] [--verbose] [--dry-run] [files ...]

    positional arguments:
      files             List of files to add to the commit

    options:
      -h, --help        show this help message and exit
      --added, -A       Update files in the current commit, including files added with `git add`
      --all, -a         Append all locally-modified, tracked files to the current commit
      --everything, -e  Append all modified and untracked files to the current commit (implies `~--all`)
      --ignored, -i     Include files normally hidden by `.gitignore`
      --patch, -p       Use the interactive patch selection interface to chose which changes to commit.
      --verbose, -v     Verbose mode
      --dry-run, -D     Dry-run

git cleanup

List or delete branches that have already been merged and delete tracking branches that are no longer on the remote.

    git-cleanup [-h] [--delete] [--master MASTER] [--force] [--unmerged] [--yes] [--debug] [branches ...]

    positional arguments:

    branches              List of branches to check (default is all branches)

    options:
    -h, --help            show this help message and exit
    --delete, -d          Delete all branches that have been merged
    --master MASTER, -m MASTER, --main MASTER
                          Specify the master branch (Attempts to read this from GitLab or defaults to "develop" if present or "master" or "main" otherwise
    --force, -f           Allow protected branches (e.g. master) to be removed
    --unmerged, -u        List branches that have NOT been merged
    --yes, -y             Assume "yes" in response to any prompts (e.g. to delete branches)
    --debug               Enable debug output

git co

Equivalent to git checkout but with enhanced branch matching as described above. The command does not support the full range of options supported by the git checkout comamnd which should still be used where additional functionality is required.

    git-co [-h] [--branch] [--update] [--rebase] [--force] [--exact] [--debug] branchname

    positional arguments:

    branchname    The branch name (or a partial name that matches uniquely against a local branch, remote branch, commit ID or tag)

    options:
    -h, --help    show this help message and exit
    --branch, -b  Create the specified branch
    --update, -u  If a remote branch exists, delete any local branch and check out the remote version
    --rebase, -r  Rebase the branch onto its parent after checking it out
    --force, -f   When using the update option, recreate the local branch even if it is owned by the current user (based on the author of the most recent commit)
    --exact, -e   Do not use branch name matching - check out the branch as specified (if it exists)
    --debug       Enable debug output

git common

Find the most recent common ancestor for two commits

    usage: git-common [-h] [--short] [--long] [--path PATH] [commit1] [commit2]

    positional arguments:
      commit1               First commit (default=HEAD)
      commit2               Second commit (default=master)

    options:
      -h, --help            show this help message and exit
      --short, -s           Just output the ancestor commit ID
      --long, -l            Output the log entry for the commit
      --path PATH, -C PATH  Run the command in the specified directory

git hold

Archive, list or recover one or more Git branches

    usage: git-hold [-h] [--list] [--restore] [--path PATH] [branches ...]

    positional arguments:
      branches              Branches

    options:
      -h, --help            show this help message and exit
      --list, -l            List archived branches
      --restore, -r         Restore archived branches
      --path PATH, -C PATH  Run the command in the specified directory

git parent

Attempt to determine the parent branch for the specified branch (defaulting to the current one).

    git-parent [-h] [--all] [--verbose] [branch]

    Attempt to determine the parent branch for the specified branch (defaulting to the current one)

    positional arguments:
      branch         Branch, commit or commit (defaults to current branch; main)

    options:
      -h, --help     show this help message and exit
      --all, -a      Include feature branches as possible parents
      --verbose, -v  Report verbose results (includes number of commits between branch and parent)

git retag

Apply or update a tag, optionally updating it on the remote as well. If the specified tag exists, it is deleted and re-applied, otherwise it is recreated.

    usage: git-retag [-h] [--push] [--path PATH] tag

    positional arguments:
      tag                   The tag

    options:
      -h, --help            show this help message and exit
      --push, -p            Push the tag to the remote
      --path PATH, -C PATH  Run the command in the specified directory

git update

Update the rworking tree from the remote, rebase local branch(es) against their parents and optionally run git cleanup.

usage: git_update.py [-h] [--default] [--cleanup] [--all] [--everything] [--parent PARENT] [--stop] [--ignore IGNORE] [--main MAIN] [--verbose] [--debug]
                     [--path PATH]

Rebase branch(es) against their parent branch, updating both in the process

options:
  -h, --help           show this help message and exit
  --default, -d        Checkout the main or master branch on completion
  --cleanup, -c        After updating a branch, delete it if there are no differences between it and its parent branch
  --all, -a            Update all local branches, not just the current one
  --everything, -A     Update all local branches, not just the current one and ignore the default ignore list specified in the Git configuration
  --parent, -p PARENT  Specify the parent branch, rather than trying to work it out
  --stop, -s           Stop if a rebase problem occurs, instead of skipping the branch
  --ignore, -i IGNORE  List of one or more wildcard branch names not to attempt to update
  --main, -m MAIN      List of one or more wildcard branch names that are considered main branches and should be pulled but not rebased onto anything
  --verbose, -v        Enable verbose output
  --debug, -D          Enable debug output
  --path, -C PATH      Run the command in the specified directory

The [update] section of the Git config can be used to specify the default values for the --ignore and --main parameters and both config and command line
can use wildcard values, for example "release/*".

git wt

Output the top level directory of the git working tree or return an error if we are not in a git working tree.

git-wt [-h] [--parent] [--dir DIR] [level]

positional arguments:
  level              Number of levels below the top-level directory to report

options:
  -h, --help         show this help message and exit
  --parent, -p       If we are already at the top of the working tree, check if the parent directory is in a working tree and output the top-level directory of that tree.
  --dir DIR, -d DIR  Find the location of the top-level directory in the working tree starting at the specified directory

git review

Menu-driven Git code review tool

git-review [-h] [--commit COMMIT] [--branch BRANCH] [--change] [--debug] [--dir DIR] [--difftool DIFFTOOL] [commits ...]

positional arguments:
  commits               Commit(s) or paths to compare

options:
  -h, --help            show this help message and exit
  --commit COMMIT, -c COMMIT
                        Compare the specified commit with its parent
  --branch BRANCH, -b BRANCH
                        Compare the specified commit to branch point on specified branch
  --change, -C          Compare the current commit with its parent
  --debug, -d           Start a debug session over Telnet using pudb
  --dir DIR             Work in the specified directory
  --difftool DIFFTOOL   Override the default git diff tool

ggit

Run a git command in all working trees under the current directory (somewhat superceded by multigit).

ggrep

Run 'git grep' in all repos under the current directory (somewhat superceded by multigit).

General Commands

addpath

Add or remove entries from a path list (e.g. as used by the PATH environment variable)

usage: addpath.py [-h] [--add ADD] [--prefix PREFIX] [--suffix SUFFIX] [--remove REMOVE] [--separator SEPARATOR] path

Add or remove entries from a path list (e.g. as used by the PATH environment variable)

positional arguments:
  path                  The path to modify

options:
  -h, --help            show this help message and exit
  --add ADD             Add an entry to the front of the path (do nothing if it is already present in the path)
  --prefix PREFIX       Add an entry to the front of the path (or move it there if it is already present)
  --suffix SUFFIX       Add an entry to the end of the path (or move it there if it is already present)
  --remove REMOVE       Remove an entry from the path (do nothing if it is not present
  --separator SEPARATOR Override the default path separator

consolecolours

Display all available colours in the console.

ffind

Simple file find utility - replaces the find command with something that is more human-friendly.

ffind [-h] [--path PATH] [--long] [--colour] [--no-colour] [--all] [--zero] [--iname] [--follow] [--git] [--diff] [--regex] [--fullpath] [--human-readable] [--grep GREP] [--abspath]
            [--unquoted] [--quiet] [--invert] [--exec EXEC] [--count] [--count-only] [--type TYPE] [--file] [--dir] [--block] [--char] [--pipe] [--symlink] [--socket] [--any] [--verbose]
            [--debug]
            [patterns ...]

positional arguments:
  patterns              List of things to search for.

options:
  -h, --help            show this help message and exit
  --path PATH, -p PATH  Search the specified path, rather than the current directory
  --long, -l            Output details of any files that match (cannot be used with -0/--zero)
  --colour, -C, --color
                        Colourise output even if not outputting to the terminal
  --no-colour, -N, --no-color
                        Never colourise output
  --all                 Search all directories (do not skip .git, and similar control directories)
  --zero, -0            Output results separated by NUL characters
  --iname, -i           Perform case-independent search
  --follow, -F          Follow symlinks
  --git, -g             Only search for objects in the current git repository
  --diff, -D, --diffuse
                        Run Diffuse to on all the found objects (files only)
  --regex, -R           Use regex matching rather than globbing
  --fullpath, -P        Match the entire path, rather than just the filename
  --human-readable, -H  When reporting results in long format, use human-readable sizes
  --grep GREP, -G GREP  Only report files that contain text that matches the specified regular expression
  --abspath, -A         Report the absolute path to matching entities, rather than the relative path
  --unquoted, -U        Do not use quotation marks around results containing spaces
  --quiet, -q           Do not report permission errors that prevented a complete search
  --invert, -I          Invert the wildcard - list files that do not match
  --exec EXEC, -x EXEC  Execute the specified command on each match (optionally use ^ to mark the position of the filename)
  --count, -K           Report the number of objects found
  --count-only, -c      Just report the number of objects found
  --type TYPE, -t TYPE  Type of item(s) to include in the results, where b=block device, c=character device, d=directory, p=pipe, f=file, l=symlink, s=socket. Defaults to files and directories
  --file, -f            Include files in the results (the default if no other type specified)
  --dir, -d             Include directories in the results
  --block               Include block devices in the results
  --char                Include character devices in the results
  --pipe                Include pipes in the results
  --symlink, --link     Include symbolic links in the results
  --socket              Include sockets in the results
  --any, -a             Include all types of item (the default unless specific types specified)
  --verbose, -v         Output verbose data
  --debug               Output debug data

linecount

Summarise number of files, lines of text and total size of files in a directory tree

usage: linecount [-h] [--ext]

options:
  -h, --help  show this help message and exit
  --ext, -e   Identify file type using the file extension (faster but less accurrate)

py-audit

Query api.osv.dev to determine whether a specified version of a particular Python package is subject to known security vulnerabilities

py-audit [-h] [requirements ...]

positional arguments:
  requirements  The requirements file (if not specified, then the script searches for a requirements.txt file)

options:
  -h, --help    show this help message and exit

remdir

Recursively delete empty directories

remdir [-h] [--dry-run] [--debug] [--verbose] [--ignore IGNORE] [--keep KEEP] dirs [dirs ...]

positional arguments:
  dirs                  Directories to prune

options:
  -h, --help            show this help message and exit
  --dry-run, -D         Dry-run - report what would be done without doing anything
  --debug               Output debug information
  --verbose             Output verbose information
  --ignore IGNORE, -I IGNORE
                        Files to ignore when considering whether a directory is empty
  --keep KEEP, -K KEEP  Directories that should be kept even if they are empty

rpylint

Run pylint on all the Python source files in a directory tree

usage: rpylint [-h] [paths ...]

positional arguments:
  paths       List of files or paths to lint

options:
  -h, --help  show this help message and exit

tfparse

Read JSON Terraform output and convert back to human-readable text

This allows multiple errors and warnings to be reported as there's no way of doing this directly from Terraform

usage: tfparse [-h] [--abspath] [infile ...]

positional arguments:
  infile         The error file (defaults to standard input if not specified)

options:
  -h, --help     show this help message and exit
  --abspath, -a  Output absolute file paths

trimpath

Intelligently trim a path to fit a given width (used by gitprompt)

venv-create

Create a script to create/update a virtual environment and run a python script in it.

usage: venv-create [-h] name

positional arguments:
  name        Name of the script to create

options:
  -h, --help  show this help message and exit

xchmod

Command to run chmod only on files that need it (only modifies files that don't have the required permissions already).

usage: xchmod [-h] [--debug] [--verbose] [--recursive] mode paths [paths ...]

positional arguments:
  mode             Mode to set
  paths            List of directory paths to search

options:
  -h, --help       show this help message and exit
  --debug          Output the list of files (if any) that need to be made publically writeable
  --verbose        List files as they are updated
  --recursive, -R  Operate recursively

yamlcheck

YAML validator - checks that a file is valid YAML (use yamllint to verify that it is nicely-formatted YAML).

usage: yamlcheck [-h] [--dump] [--block] [--flow] [--hiera] files [files ...]

positional arguments:
  files       YAML source file

options:
  -h, --help  show this help message and exit
  --dump      Dump the YAML data after parsing it
  --block     Force block style when dumping the YAML data
  --flow      Force flow style when dumping the YAML data
  --hiera     Process the file as Puppet Hiera data