restic-replica

The current project status should be considered as "beta".

Info

A command line tool to copy snapshots between Restic repositories, written in python.
What is Restic? An awesome backup tool.
Why?

• Restic does not natively support configuration files.
• Restic's copy operation does not allow filtering the snapshots to be copied by age, or number of snapshots.

Installation

pip install restic-replica

Usage

1. Initialize both the source and destination repositories using Restic, if they have not been already initialized.
   Note: Make sure to use --copy-chunker-params when initializing the destination repository.
2. Make sure at least one backup has been stored in the source repository.
3. Run restic-replica once to generate an empty configuration file.

❯ restic-replica
ERROR: Missing configuration file
An example configuration file has been created at /home/user/.restic-replica/config.toml. Update the configuration in this file to match your system, and then re-run this program.

4. Update the configuration file with the correct values for your repositories. Ensure the repository_uri, a password option, and any necessary environment variables have been updated for both source and destination.
5. Optional: run restic-replica with the --dry-run flag to check the connection to both repositories, and verify the list of snapshots to be copied is as expected.
6. Run restic-replica again to copy snapshots from the source repository to the destination repository.

Policy

A policy can be defined within the configuration file to apply age or number-based filtering, controlling which snapshots will be copied between the source and destination repositories.

The policy is applied to individual snapshot groups (e.g. "keep-last = 2" will copy the last 2 snapshots from each group). Snapshots are grouped by host and path by default, but this can be customised or disabled.

The filters defined in the policy work the same as the filters used with restic forget, in that all of the calendar-based filters (e.g. keep-daily, keep-weekly, etc.) work on natural time boundaries, and are not relative to when restic-replica is run.

Weeks span from Monday 00:00 to Sunday 23:59, days from 00:00 to 23:59, etc. The most recent snapshot within a calendar period is always selected. For example; if multiple snapshots exist on a given day, and a keep-daily filter selects a snapshot from that day, the most recent snapshot taken on that day is selected.

Use of the --dry-run option is recommended to verify that the specified filter options match expectations.

Note: Commenting out the filter options (e.g, keep-last, keep-daily, etc.) in the configuration file will disable the policy. In this state, all snapshots will be copied, unless they are excluded by host, path, or tag filters.

The "exclude-current-period" option

This option provides a workaround to an issue, where running restic-replica multiple times in the same calendar period can lead to inconsistent snapshot selection when using calendar based filters. This happens because the most recent snapshot from the calendar period is always selected, and which snapshot this is can change.

An example showing the problem:

• A new restic snapshot is taken daily, and backed up to a repository
• The goal is to copy all the monthly snapshots from this source repository to another, destination repository
• To accomplish this, a keep-monthly filter is applied. One snapshot from each month will be selected for copying, including the current month.
  • The snapshot selected from the current month will be the most recent snapshot taken when restic-replica is run.
  • As the source takes a new snapshot each day, the snapshot selected will be the snapshot taken on the day restic-replica is run.
• If restic-replica is run for a second time, more than a day later but still within the same calendar month, it will again select the most recent snapshot as the snapshot for the current month.
• However, time has passed, and there is now a more recent daily snapshot in the source repository. During the second run, it is this snapshot that will be selected for copying from the current month, not the snapshot that was selected during the previous run.
• The result of this is that two snapshots for the current month will be copied to the destination repository.
• If this pattern repeats, the number of snapshots copied to the destination repository from the "current month" will continually increase. This is not ideal, as the original goal was to copy only a single snapshot from each month.

Setting the exclude-current-period option to true allows us to work around this problem.
When enabled, snapshots from the "current calendar period" are excluded from any enabled calendar-based filters (e.g. keep-daily, keep-weekly, etc.), i.e. they are ignored.

Note: "current calendar period" is defined with respect to each calendar-based filter individually. For keep-daily it would be the current day, for keep-weekly the current week, etc.

Using the example above; when the keep-monthly filter is applied and exclude-current-period is enabled, any snapshots taken during the current month are treated as though they do not exist. e.g. if restic-replica is run on the 25th, any snapshots taken on the 1st through 25th are ignored by the keep-monthly filter, and will not be selected for copying.

This avoids the problem, but comes with a tradeoff, in that the destination repository can be up to one calendar period "out of date" when compared to the source. The further through the calendar period, the larger the potential difference is between the data stored on the source and destination repositories.

Note: this option has no effect on keep-last.

Multiple calendar based filters and "exclude-current-period"

When the exclude-current-period option is enabled, it applies to all enabled calendar-based filters (e.g. keep-daily, keep-weekly, etc.).
However, the exclusion period is applied individually to each enabled filter.

For example:

• If both keep-monthly and keep-daily filters are enabled, the keep-monthly filter will ignore any snapshots from the current month.
• The keep-daily filter will only ignore snapshots taken today. It will still consider including snapshots taken on the other days in the current month for copying.

Grouping

When determining which snapshots to copy, the configured policy will be applied to each snapshot group individually. Snapshots can be grouped by host, path, and/or tag; by default they are grouped by host and path. All three of these grouping options may be set independently of one another, i.e. all combinations are supported. If all grouping options are disabled, the policy will be applied to all snapshots in the repository.

Snapshot grouping is performed by restic; see the group-by option in restic snapshots --help.

Note: grouping has no effect if a policy is not configured.

Filters

Additional restrictions on the snapshots that will be considered for copying can be set by applying filters on hostname, path, and/or tags.

• Hostname filtering; snapshots match if they are for any of the specified hosts.
• Path filtering; snapshots must contain all of the (absolute) paths specified. If any path is missing, the snapshot is excluded.
• Tag filtering; tags can be provided individually, or in a comma separated group, or a combination thereof.
  • Individual tags: snapshots match if they contain any one of the individual tags
  • Tag Groups: snapshots match if they contain all of the tags in the group
  • e.g. ["tag_1, tag_2"] will only match snapshots with both tags. ["tag_1", "tag_2"] will match snapshots with either, or both tags.

Host, path, and tag filtering is performed by restic; see the --host, --path, and --tag options in restic snapshots --help.

Development

This project uses uv for package management.

To build the wheel and sdist:

uv build

To run the unit test suite:

uv run pytest