Skip to main content

Archive (safe)-Removal Rolling Toolbox

Project description

A2R

The name a2r, pronounced as a double r, stands for Archive safe-Remove Rolling. It's a toolbox that provides different tool for:

  • Archive automatically Oceanic experiment outputs base on yaml configuration file
  • (safe) Rm files if a local copy already exists
  • Automatic Rolling with different options on multiple directory with a single command. Also in this case the info about what to delete and how are provided using a yaml configuration file

Installation

Via pip

pip install a2r

Via conda/mamba

mamba install a2r

Usage

Usage: a2r [OPTIONS] COMMAND [ARGS]...

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  rolling
  saferm
  update-md5

Rolling

To keep a directory under rolling, use the command:

Usage: cli.py rolling [OPTIONS]

Options:
  -c, --config PATH  [required]
  --dry-run          Dry run mode.
  -d, --debug        Enable debug mode.
  --help             Show this message and exit.

Rolling Configuration File

Here a template of a rolling configuration file:

- !CleanPath
    path: $PATH_UNDER_ROLLING
    fmt: ????????   # YYYYMMDD
    safe:
        to_keep: X
        reference_paths:
            - <REF1>
            - <REF2>
    conditional:
        to_keep: Y
        expected_files:
            - file.exe
            - tmp.nc
    force:
        to_keep: Z

let's comment each section.

CleanPath object

With this, we start to declare a list of !CleanPath objects with two mandatory attributes:

- !CleanPath
    path: $PATH_UNDER_ROLLING
    fmt: ????????   # YYYYMMDD
  • path: it's the path to keep under rolling
  • fmt: it's a string bash that represent the format of file/dir to delete. It can contains jolly characters ? or *
Safe
  safe:
      to_keep: X
      reference_paths:
          - <REF1>
          - <REF2>
  • safe: means for safe rolling, which means delete a file only if an identical local copy already exists
  • to_keep: how much dir/file to not include in the rolling
  • reference_paths: a list of path where to find if a local copy already exists

To consider that the safe mode doesn't remove the dir under rolling.

Conditional #### Conditional
  conditional:
      to_keep: Y
      expected_files:
          - file.exe
          - tmp.nc
  • conditional: specify to remove a dir if some conditions are meet
  • to_keep: how much dir/file to not include in the rolling
  • expected_files: specify the exact list of files expected to find in rolling path to trigger the rm operations. The filename can contains jolly character ? and *
Force
    force: # optional
        to_keep: Z
  • force: Enable path rm without any check
  • to_keep: how much dir/file to not include in the rolling

Update md5 dir

A requirement to safe clean, is to compute in advance the md5 of all files in the rolling path using the command:

update-md5 [OPTIONS]

Options:
  -p, --path PATH  [required]
  -d, --debug      Enable debug mode.
  --help           Show this message and exit.

The command will create in the directory a file called .dir_md5.txt with the following structure:

md5hash filename1
md5hash filename2

Safe Clean

Safe Rm command is the equivalent of safe rm rolling section: it removes files from clean path only if exists an identical copy in keep path

Usage: cli.py saferm [OPTIONS]

Options:
  --keep PATH    [required]
  --clean PATH   [required]
  -f, --force    Delete files without confirm request
  -d, --dry-run  Disable file removal
  -d, --debug    Enable debug mode.
  --help         Show this message and exit.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

a2r-0.1.0.tar.gz (14.8 kB view details)

Uploaded Source

Built Distribution

a2r-0.1.0-py3-none-any.whl (18.0 kB view details)

Uploaded Python 3

File details

Details for the file a2r-0.1.0.tar.gz.

File metadata

  • Download URL: a2r-0.1.0.tar.gz
  • Upload date:
  • Size: 14.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.10.14 Linux/6.5.0-1019-oem

File hashes

Hashes for a2r-0.1.0.tar.gz
Algorithm Hash digest
SHA256 b9a78e82bcd754b31b4477454ba58e7a7b264b11e9b2cd3e286eebe321a1d2fe
MD5 07096ad79b0092c59971ba2dc037fc94
BLAKE2b-256 8cc654923fff01c54f59b125c35c678325e3f0076ef43bf436151cadd23171ed

See more details on using hashes here.

File details

Details for the file a2r-0.1.0-py3-none-any.whl.

File metadata

  • Download URL: a2r-0.1.0-py3-none-any.whl
  • Upload date:
  • Size: 18.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.10.14 Linux/6.5.0-1019-oem

File hashes

Hashes for a2r-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 47f2832f318624c35ae877426b0ace05f987f087dfbe15cbc6bd3521cacff5c7
MD5 1f0ecaf5bf436e8ff059f6088b1d625c
BLAKE2b-256 94ff61d3d0339f96dd701fb899760f6b18194725d5d44bf405906aeedca07e8e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page