dotsecrets 0.4.1

DotSecrets is a tool to facilitate storing your dotfiles in Git, including those with private information. The private information is filtered before committing to the repository. DotSecrets is able to symlink your dotfiles into your home directory similar to Stow.

DotSecrets

DotSecrets [1] is a tool written by Olaf Conradi to facilitate storing your dotfiles in Git, including those with private information.

By storing your configuration files in a public Git repository, you can synchronize your settings between multiple devices or share them with others. Any private information is kept in a single file stored outside the repository. It’s up to the user to transport this file between devices.

This tool is similar in functionality to Briefcase [2] but differs significantly. DotSecrets uses Git filtering to manage private information and uses a different file hierarchy and naming convention.

For more information on the filtering capabilities of Git, see the git attributes manual [3] in section Effects under filter attribute.

Symbolic linking and unlinking is supported by organizing your dotfiles in topics (specific topic names as top level directory within your repository). The stow and unstow commands automate linking and unlinking them. You decide how to name your topics. Either by application or device name, or a combination of the two. This practise is explained in a blog article [4] by Brandon Invergo back in 2012.

Dependencies

DotSecrets depends on ruamel.yaml [5] for reading configuration files and dploy [6] for stow functionality.

Installation

Run:

$ pip3 install dotsecrets

You should then have a dotsecrets script available in a new shell.

When installing directly from the Git repository [1] use:

$ pip3 install .

You might need to symlink it into your ~/bin folder:

$ ln -s ~/.local/bin/dotsecrets ~/bin

Usage

DotSecrets is to be used together with Git to manage your dotfiles.

Git filters are used to clean and smudge secrets. Each filter is configured using regular expressions grouped per filter name. The filters are named after the file path relative to the Git root directory.

Filters

Filter rules are defined in a file called .dotfilters.yaml inside the dotfiles repository.

Its syntax is as follows:

version: 2
filters:
  "mutt/.mutt/muttrc":
    rules:
      passwd:
        description: Mutt passwords
        numbered: true
        regex: password(\s*)=(\s*)(?#WSUpToHash)
        substitute: password\1=\2(?#Key)
  "irssi/.irssi/config":
    encoding: latin1
    rules:
      nickname:
        description: IRC nickname
        regex: nick(\s*)=(\s*)(?#QuotedString);
        substitute: nick\1=\2"(?#Key)";
      realname:
        regex: real_name(\s*)=(\s*)(?#QuotedString);
        substitute: real_name\1=\2"(?#Key)";

This file contains filter rules for each file that contains secrets. The first example defines a filter for replacing passwords in mutt configuration files. A secret is detected by a regular expression matching on each line containing the word password followed by an equal sign and each character (except whitespace) up to an optional hash # comment.

A match is replaced by the following: password = $DotSecrets: password_1$. The key is appended with the number of matches because numbered is defined as true. This allows for multiple matches and substitutions as long as the ordering in the file is retained.

The second example shows a filter for hiding your nickname in an Irssi configuration file. The encoding field will make sure the file is opened in the specified encoding (default encoding is utf-8). The regular expression matches any line containing the word nick followed by whitespace and one or more alphanumeric characters. A match is replaced by nick = "$DotSecrets: nickname$";.

Similar for the filter to hide your real name in the same file. The regular expression matches any line containing real_name followed by an equal sign, quoted text and a final semi-colon. A match is replaced by real_name = "$DotSecrets: realname$";.

Please note that the description, numbered and encoding fields are optional.

The regular expressions and substitutions follow the Python regular expression syntax [7]. Substitutions can reference regex groups (...) using \number syntax. To make it easier to define complex regular expressions, the following shortcuts are available. They are defined as regex comments (?#...):

Shortcut	Description
(?#QuotedString)	Matches balanced single or double quoted strings and is able to cope with escaped quote symbols within the string
(?#QuotedOrSingleWord)	Same as QuotedString or an unquoted single word of non-whitespace characters
(?#WSUpToHash)	Matches whitespace up to the hash symbol #
(?#WSUpToSemicolon)	Matches whitespace up to the semi colon symbol ;
(?#Key)	Used to substitute the secret

Secrets

Secret information, like passwords, answers to security questions, and other sensitive information is stored in a file called dotsecrets.yaml inside the XDG configuration directory (typically ~/.config/dotsecrets/dotsecrets.yaml).

Its syntax is as follows:

version: 2
filters:
  "mutt/.mutt/muttrc":
    secrets:
      password_1:
        description: Password for GMail
        secret: s3cr3t
      password_2:
        description: Password for Hotmail
        secret: f00bar
  "irssi/.irssi/config":
    secrets:
      nick:
        secret: mynick
      realname:
        secret: My Real Name

This configuration file contains two filters for mutt and irssi. Each filter contains one or more secrets. These secrets are used to filter the files in the Git repository for sensitive data. Each secret has an optional description field.

Linking filters and secrets

Git attributes are used to link file patterns to Git filters. The filters are defined in git config files.

Contents of .gitattributes:

* filter=dotsecrets

When checking in files with Git, the clean command is run for those files that match the pattern given in .gitattributes. When checking out files that have a filter defined, the smudge command substitutes the secrets again.

To add these filters run the following commands:

$ git config filter.dotsecrets.clean "dotsecrets clean %f"
$ git config filter.dotsecrets.smudge "dotsecrets smudge %f"
$ git config filter.dotsecrets.required true

They result in the following addition to your .git/config file:

[filter "dotsecrets"]
    clean = dotsecrets clean %f
    smudge = dotsecrets smudge %f
    required = true

Upon filtering (typically on git checkin, checkout or diff) the %f argument is replaced by the file path relative to the Git root directory. This is why filters must be named accordingly.

Initialize Repository

Upon a fresh checkout of the dotfiles repository, the git filter and git attributes configuration are not yet in place. The init command is available to initialize the configuration (when needed) and do the initial smudge on files defined as having secrets.

Example:

$ git clone [e-mail]:username/dotfiles.git
$ cd dotfiles
$ dotsecrets init

The following error indicates you do not yet have a secrets file in place. Either create the file as described in the secrets section or copy your existing file from another machine:

$ dotsecrets init
[Errno 2] No such file or directory: '/home/user/.config/dotsecrets/dotsecrets.yaml'

Stow and Unstow

Using the stow command each topic is linked into your home directory. The unstow command will unlink them. The topics to act upon are specified on the command line. To act on all available topics pass the --all argument. Add --dry-run to simulate which actions will be taken without doing them.

To stow and unstow the current working directory must be set inside the dotfilters repository.

Example:

$ dotsecrets stow mutt irssi

This will stow both topics.

Use the following to simulate the actions for linking mutt. The output is a list of actions needed:

$ dotsecrets stow --dry-run mutt
dploy stow: link /home/user/.mutt => dotfiles/mutt/.mutt

To remove the symbolic links from your home directory, run:

$ dotsecrets unstow --dry-run mutt
dploy stow: unlink /home/user/.mutt => dotfiles/mutt/.mutt

Adding new dotfiles

Defining regular expressions for new filters might require some practise. To test your filter definitions a test command is available:

$ dotsecrets test irssi/.irssi/config
--- /home/user/dotfiles/irssi/.irssi/config 2019-07-15 22:40:03.782600150 +0200
+++ /home/user/dotfiles/irssi/.irssi/config.dotclean        2019-07-17 21:23:22.813039617 +0200
@@ -286,8 +286,8 @@

 settings = {
   core = {
-    real_name = "My Real Name";
-    nick = "mynick";
+    real_name = "$DotSecrets: realname$";
+    nick = "$DotSecrets: nick$";
   };
   "fe-text" = { actlist_sort = "refnum"; scrollback_lines = "2000"; };
   "fe-common/core" = {

Two intermediate files are created: config.dotclean and config.dotsmudge. The difference is shown between the original source (which contains secrets) and the cleaned up file (which will contain markers). Next, the cleaned source is smudged to replace the markers with the secrets from your secrets store. The resulting file should be identical to the original source file. If that is not the case, the difference is shown.

Suppose a typo was made in the secrets store:

$ dotsecrets test irssi/.irssi/config
--- /home/user/dotfiles/irssi/.irssi/config 2019-07-15 22:40:03.782600150 +0200
+++ /home/user/dotfiles/irssi/.irssi/config.dotclean        2019-07-17 21:23:22.813039617 +0200
@@ -286,8 +286,8 @@

 settings = {
   core = {
-    real_name = "My Real Name";
-    nick = "mynick";
+    real_name = "$DotSecrets: realname$";
+    nick = "$DotSecrets: nick$";
   };
   "fe-text" = { actlist_sort = "refnum"; scrollback_lines = "2000"; };
   "fe-common/core" = {
--- /home/user/dotfiles/irssi/.irssi/config 2019-07-17 21:27:21.118130339 +0200
+++ /home/user/dotfiles/irssi/.irssi/config.dotsmudge       2019-07-17 21:36:48.327586627 +0200
@@ -287,7 +287,7 @@
 settings = {
   core = {
     real_name = "My Real Name";
-    nick = "mynick";
+    nick = "myname";
   };
   "fe-text" = { actlist_sort = "refnum"; scrollback_lines = "2000"; };
   "fe-common/core" = {
Source '/home/user/dotfiles/irssi/.irssi/config' and smudged source differ
Please adjust filter definition or validate your stored secrets

In the example above, key nick was set to myname not mynick in the secrets store. When the execution finishes, the intermediate files are deleted. If you want to retain those files for closer inspection, specify the --keep flag on the command line.

When you are satisfied with the output you can add the original source under version control. The clean filter will be applied before the commit.

References

[1] (1,2)

https://github.com/oohlaf/dotsecrets

[2]

https://github.com/jim/briefcase

[3]

https://git-scm.com/docs/gitattributes

[4]

http://brandon.invergo.net/news/2012-05-26-using-gnu-stow-to-manage-your-dotfiles.html

[5]

https://pypi.org/project/ruamel.yaml

[6]

https://pypi.org/project/dploy

[7]

https://docs.python.org/3/library/re.html#regular-expression-syntax

Changes

0.4.1 (2022-11-28)

• Add type to cmd FILE arguments by Marzzzello (PR-17)

0.4 (2020-07-15)

• Forgot to mark the changes list as final with a date

• Decided version 0.4 would be better

• Fixed git tag and minor tweaks to release document

0.3.3 (2020-07-15)

• Remove PosixPath arguments from shutil methods for compatibility with Python 3.5

• Reconfigure text encodings is only available on stdin and stdout on Python 3.7+, changing to use io.TextIOWrapper

• More small tweaks for Python 3.5

0.3.2 (2020-07-14)

• Make code compatible with Python 3.5

0.3.1 (2019-07-25)

• Testing on PyPI resulted in stricter dependencies

0.3 (2019-07-25)

• Code changed to Python 3 only

• Based filters on file names which simplifies config a lot

• Breaking change in YAML file config format

• Breaking change in git filter config (one config to rule all secrets)

• Changed YAML parser to ruamel.yaml

• Added stow and unstow command to manage linking of dotfiles

• Added test command to simplify adding new dotfiles

0.2 (2017-08-30)

• Make code compatible with Python 3

0.1 (2016-07-03)

• Add console script

• Code clean up, fix pylint warnings

• Setup script fixes by Jesse (PR-1)

0.0 (2013-05-11)

• Initial version