s3-storage-stats 1.0.4

S3 Storage Stats

s3-storage-stats

Installation

Linux:

pip3 install s3-storage-stats

This will install all necessary dependencies and create the executable '/usr/local/bin/s3-storage-stats'

Usage

Make sure the user that runs it is able to read the s3 configuration file.

s3-storage-stats -h
usage: s3-storage-stats [-h] {checksums,reports} ...

positional arguments:
  {checksums,reports}
    checksums          obtain and output object/file checksums
    reports            obtain and output storage reports

optional arguments:
  -h, --help           show this help message and exit

Sub-commands

Checksums

The checksums sub-command has two sub-commands itself:

get

A client gives the filename and the type of checksum hash to obtain. If this information is found, it will be printed to stdout. If not, 'None' will be printed out.

In its simplest form it requires two positional arguments:

s3-storage-stats checksums get -f [FILE] -t [HASH_TYPE]

A more complex example specifying configuration file, logging file, logging level and verbosity:

s3-storage-stats checksums get -f [FILE] -t [HASH_TYPE] -c /etc/xrootd/s3_config --loglevel=WARNING --logfile='/var/log/s3-storage-stats/s3_storage_stats.log' -v

Help:

s3-storage-stats checksums get -h
usage: s3-storage-stats checksums get [-h] [-c CONFIG_PATH] [--force] [-v]
                                      [--logfile LOGFILE] [--logid LOGID]
                                      [--loglevel {DEBUG,INFO,WARNING,ERROR}]
                                      -t HASH_TYPE -f FILE

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG_PATH, --config CONFIG_PATH
                        Path to s3 endpoint .conf file or directory. Accepts
                        one argument. Default: '/etc/xrootd/s3cfg'.
  --force               Force command execution.
  -v, --verbose         Show on stderr events according to loglevel.

Logging options:
  --logfile LOGFILE     Set logfiles path. Default:
                        /var/log/xrootd/s3_proxy/storage_stats.log
  --logid LOGID         Add this log id to every log line.
  --loglevel {DEBUG,INFO,WARNING,ERROR}
                        Set log output level. Default: WARNING.

Checksum required options:
  -t HASH_TYPE, --hash_type HASH_TYPE
                        Type of checksum hash. ['adler32', md5] Required.
  -f FILE, --file FILE  URL of object/file to request checksum of. Required.

---

put

A client gives the filename, checksum, and type of checksum has to add this information to the file's metadata. Nothing is returned unless the process encounters errors.

In its simplest form it requires three positional arguments:

s3-storage-stats checksums get -f [FILE] -t [HASH_TYPE] --checksum [CHECKSUM]

A more complex example specifying configuration file path, logging file and level, and verbosity:

s3-storage-stats checksums put -f [FILE] -t [HASH_TYPE] --checksum [CHECKSUM] -c /etc/xrootd/s3_config --loglevel=WARNING --logfile='/var/log/s3-storage-stats/s3_storage_stats.log' -v

Help:

s3-storage-stats checksums put -h
usage: s3-storage-stats checksums put [-h] [-c CONFIG_PATH] [--force] [-v]
                                      [--logfile LOGFILE] [--logid LOGID]
                                      [--loglevel {DEBUG,INFO,WARNING,ERROR}]
                                      -t HASH_TYPE -f FILE --checksum CHECKSUM

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG_PATH, --config CONFIG_PATH
                        Path to s3 endpoint .conf file or directory. Accepts
                        one argument. Default: '/etc/xrootd/s3cfg'.
  --force               Force command execution.
  -v, --verbose         Show on stderr events according to loglevel.

Logging options:
  --logfile LOGFILE     Set logfiles path. Default:
                        /var/log/xrootd/s3_proxy/storage_stats.log
  --logid LOGID         Add this log id to every log line.
  --loglevel {DEBUG,INFO,WARNING,ERROR}
                        Set log output level. Default: WARNING.

Checksum required options:
  -t HASH_TYPE, --hash_type HASH_TYPE
                        Type of checksum hash. ['adler32', md5] Required.
  -f FILE, --file FILE  URL of object/file to request checksum of. Required.
  --checksum CHECKSUM   String with checksum to set. ['adler32', md5] Required

---

Reports

The reports command calculates and outputs the total bytes used in the s3 storage followed by the number of files.

In its simplest form, the reports command requires no addition arguments:

s3-storage-stats reports

A more complex example specifying configuration file path, logging file and level, and verbosity:

s3-storage-stats reports -c /etc/xrootd/s3_config --loglevel=WARNING --logfile='/var/log/s3-storage-stats/s3_storage_stats.log' -v

Help:

usage: s3-storage-stats reports [-h] [-c CONFIG_PATH] [--force] [-v]
                                [--logfile LOGFILE] [--logid LOGID]
                                [--loglevel {DEBUG,INFO,WARNING,ERROR}]
                                {} ...

positional arguments:
  {}

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG_PATH, --config CONFIG_PATH
                        Path to s3 endpoint .conf file or directory. Accepts
                        one argument. Default: '/etc/xrootd/s3cfg'.
  --force               Force command execution.
  -v, --verbose         Show on stderr events according to loglevel.

Logging options:
  --logfile LOGFILE     Set logfiles path. Default:
                        /var/log/xrootd/s3_proxy/storage_stats.log
  --logid LOGID         Add this log id to every log line.
  --loglevel {DEBUG,INFO,WARNING,ERROR}
                        Set log output level. Default: WARNING.