backups-rotate 0.0.1

kind of like logrotate, but for backups

backups-rotate

You run a website with a database, and you want to backup your database every day, and keep the last 7 days of backups. This script will do that for you.

WORK IN PROGRESS

not yet published on pypi

What it does

You can define a policy on several frequency levels, e.g.

• one per year - no matter how far back in time
• and/or, keep one backup for each of the last 3 quarters
• and/or, the last 2 months
• and/or, the last 4 weeks
• and/or, the last 5 days
• and/or, the last 6 hours

Installation

pip install backups-rotate

Configuration

The tool comes with 6 predefined "periods": year, quarter, month, week, day, hour.

You define your folder and policy in a YAML file, like e.g (for the policy described above):

---
# this defines the area named "database" that can be used
# on the command line to specify what needs to be cleaned up

- name: database
  folder: /usr/lib/backups
  patterns:
    - "mydb*.sql"
    - "mydb*.sql.gz"
  policy:
    year: infinite
    quarter: 3
    month: 2
    week: 4
    day: 5
    hour: 6
  # optionally, you can specify the format of the timestamp in the file name
    # datetime_format: "%Y-%m-%d_%H-%M-%S"
  # OR, still optionally, you can specify to use the modification time
  # instead of the creation time
    # use_modification_time: true

Usage

Here are a few ways to run the tool

# run on all areas
backups-rotate

# on a specific area
backups-rotate database

# not doing anything, just showing what would be done
backups-rotate database --dry-run

# do it, and be verbose about it
backups-rotate database --verbose

# using a config file (default is /etc/backups-rotate.yaml)
backups-rotate --config my_policy.yaml database

which will remove, from the specified folder, all backups that are older than the policy you defined.

How policies are implemented

When run on a specific area, the tool will:

• gather all matching files in the specified folder
• then for each frequency present in the policy, starting with the shortest one:
  • split time into bins of that frequency, starting from now and going backwards
  • attach every file to corresponding bin
  • then for the number attached to that frequency (can be infinite), and starting from the most recent bin:
    • take the most recent file and mark it as kept
  • then list (if dry-run) or remove (if not dry-run) all files that are not marked as kept

Example

Based on sample2() in tests/samples.py, assume you have one file per hour between 2024-01-01 00:00 and 2024-11-15 23:00, then applying the policy above would keep:

2024-06-30 23:00    # for quarter
2024-09-30 23:00    # for quarter
2024-10-27 23:00    # for week
2024-10-31 23:00    # for month
2024-11-03 23:00    # for week
2024-11-10 23:00    # for week
2024-11-11 23:00    # for day
2024-11-12 23:00    # for day
2024-11-13 23:00    # for day
2024-11-14 23:00    # for day
2024-11-15 18:00    # for hour
2024-11-15 19:00    # for hour
2024-11-15 20:00    # for hour
2024-11-15 21:00    # for hour
2024-11-15 22:00    # for hour
2024-11-15 23:00    # for hour day week month quarter year

if instead the policy was defined with month: infinite, then the policy would also retain

2024-01-31 23:00
2024-02-29 23:00
2024-03-31 23:00
2024-04-30 23:00
2024-05-31 23:00
2024-07-31 23:00
2024-08-31 23:00

noting that the following 2 were already kept for quarter:

2024-06-30 23:00
2024-09-30 23:00

Timestamps

By default, time is taken from the file's creation time. If you want to use the file's modification time instead, you can use the use-modification-time flag in your yaml config.

Also if your files are named with a timestamp, you can use the datetime-format option to specify the format of the timestamp in the file name (using Python's datetime format); files that do not match the format will cause an error.

This means you can use either

Tests

pip install pytest
pytest