Auto rsync

Command to automate execution of various rsync commands based on profiles defined on a YAML configuration file.

• Instead of doing:

  rsync -avySH --delete --backup --backup-dir=../deleted/$timestamp/ "/media/Media/Photos" "user@host.com:/media/backup/filesets/$hostname.photos"
  

  Just do:

  autorsync -p photos
  

• Instead of doing:

  rsync -avySH --delete --backup --backup-dir=../deleted/$timestamp/ "/media/Media/Photos" "user@host.com:/media/backup/filesets/$hostname.photos"
  rsync -avySH --delete --backup --backup-dir=../deleted/$timestamp/ "/media/Media/Books" "user@host.com:/media/backup/filesets/$hostname.books"
  rsync -avySH --delete --backup --backup-dir=../deleted/$timestamp/ "/media/Media/Music" "user@host.com:/media/backup/filesets/$hostname.music"
  

  Just do:

  autorsync
  

Or, in more general terms, instead of doing long rsync commands for your everyday backups, or putting them in adhoc scripts, write the clear ~/autorsync.yaml file and let autorsync do the work for you.

Installation

pip3 install auto-remote-sync --user

(Sorry for the long package name, ideally this would be simply autorsync, but name autosync was already taken and PyPi would not allow similar names).

Check PyPi and GitHub for autorsync.

Usage

Organize Profiles in ~/autorsync.yaml

Here is an example with some defaults and a few profiles:

DEFAULTS:
    source_part1: '{{home}}/Media'
    target_part1: user@remote.host.com:/media/backup/filesets
    delete: True
    backup: True
    backup_dir: ../deleted/{{time.strftime('%Y.%m.%d-%H.%M.%S')}}/
    background: False
    extra_part1: --rsh "ssh -i ~/.ssh/id_operator" --no-atimes

profiles:
    - name: books
      source_part2: Books/
      target_part2: '{{hostname}}.books/files'
      background: True
      extra: --copy-links

    - name: nextcloud.data
      source: /var/lib/nextcloud/data
      target_part2: '{{hostname}}.nextcloud_files'
      extra_part2: --copy-links --itemize-changes

    - name: navidrome.data
      env:
          NAVI_DB: /var/lib/navidrome/data/navidrome.db
      before: |
          systemctl stop navidrome
          sqlite3 $NAVI_DB "PRAGMA wal_checkpoint(FULL); PRAGMA wal_checkpoint(TRUNCATE); PRAGMA journal_mode=DELETE; vacuum;"
      after: |
          sqlite3 $NAVI_DB "PRAGMA journal_mode=WAL;"
          systemctl restart navidrome
      source: /var/lib/navidrome/
      target_part2: '{{hostname}}.navidrome/files'
      extra_part2: --exclude=cache/

    - name: databases
      before: |
        TARGET=/path/to/db-backup/`date --rfc-3339=date`
        mkdir -p "$TARGET"
        runuser -u postgres -- psql --csv -t -c "SELECT datname FROM pg_database WHERE datistemplate=false;" | while read db; do
           runuser -u postgres -- pg_dump -F c "$db" | gzip -c --best > "$TARGET/$db.sql.gz"
        done
      source: /home/aviram/Backup/databases/
      target_part2: '{{hostname}}-databases/'

Notes about this configuration

• All profiles inherit parameters from DEFAULTS. If parameter isn’t set in the profile, the value defined in DEFAULTS will be used.
• For each profile, the Source is defined by source parameter, or, if not defined, by source_part1/source_part2
• Target follows same logic: target or target_part1/target_part2
• delete makes rsync delete files in target that are absent in source
• backup and backup_dir makes rsync save backups on target of deleted or modified files on source. Value on backup_dir is a path relative to target folder
• extra lets you add extra rsync switches and can be used in the DEFAULTS section (to affect all profiles) or just into a specific profile. You can also use extra_part1 and extra_part2 between profiles and DEFAULTS, which will cause your switches to be concatenated.
• before and after are shell scripts to be executed before and after rsync command.
• env is a list of environment variables that will be set for before and after scripts.
• You can use Jinja logic in source*, target* and extra* parts, surrounded by {{}}. Currently these are the available variables:
  • time, a Python datetime.datetime object which includes local timezone
  • hostname, such as “rocket”
  • Hostname, such as “rocket.mydomain.com” (FQDN)
  • username, UNIX user name as “joanbaez”
  • home, user’s home folder as “/home/joanbaez”
  • userid, user ID as “504”
  • gecos, user long name as “Joan Baez”

Se my real /root/autorsync.yaml file that is used everyday to run my incremental offsite backup via cron.

By the way, this is my root user crontab:

# Several backups everyday 4:15 AM
15 4 * * * $HOME/.local/bin/autorsync

As simple as that.

Example usage

• Show all profiles:

  autorsync -l
  

• Run rsync for all profiles:

  autorsync
  

• Run rsync only for profile books

  autorsync -p books
  

• Simulate rsync only for profile books (force rsync’s --dry-run)

  autorsync -n -p books
  

  or

  autorsync --dry-run -p books
  

• Run rsync for 2 profiles from a non-default configuration file:

  autorsync -c /etc/autorsync.yaml -p "books, photos"