eudat.accounting.client 1.0.1

Upload accounting records to an eudat accounting server

Upload accounting records to an eudat accounting server

Administrators of (storage) resources provided through the EUDAT Common Data Infrastructure can use this tool to conveniently report current resource consumption per registered resource. Accounting records are submitted per individual resource identified by its (P)ID which is available as soon as the resource has been registered at EUDAT’s Data Project Management Tool (DPMT).

Default settings are such that only the resource id and the consumed value need to be provided. The default unit is byte and the default resource type is set to storage. The full documentation of options supported is described in the next section.

If the tools provided here do not address your use case you can either file a feature request on the tracker or use a more low-level approach as outlined in the server documentation.

Full documentation and API

Installation

The easiest way to install the tool is via pip or easy_install. It is usually best to do this in a virtualenv:

$ pip install eudat.accounting.client

Command line interface

As a result of the above there are now two console script called addRecord and iRODScollector. Invoke it with -h to see its usage pattern and options.

addRecord

$ bin/addRecord -h
usage: addRecord [-h] [--version] [-b BASE_URL] [-u USER] [-p PASSWORD]
                 [-d DOMAIN] [-s SERVICE] [-n NUMBER] [-o OBJECT_TYPE]
                 [-k KEY] [-T TYPE] [-m MEASURE_TIME] [-C COMMENT] [-t] [-v]
                 account value [unit]

positional arguments:
  account               account to be used. Typically the (P)ID of the
                        resource to be accounted
  value                 The value to be recorded
  unit                  The unit of measurement for the value provided.
                        Default: "byte"

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -b BASE_URL, --base_url BASE_URL
                        base URL of the accounting server to use. Default:
                        https://accounting.eudat.eu
  -u USER, --user USER  user id used for logging into the server. If not
                        provided it is looked up in the environment variable
                        "ACCOUNTING_USER". Default: "" - aka not set
  -p PASSWORD, --password PASSWORD
                        password used for logging into the server. If not
                        provided it is looked up in the environment variable
                        "ACCOUNTING_PW". Default: "" - aka not set
  -d DOMAIN, --domain DOMAIN
                        name of the domain holding the account. Default: eudat
  -s SERVICE, --service SERVICE
                        UID (or PID) of the registered service component
                        reporting the record. Default: "" - not set
  -n NUMBER, --number NUMBER
                        number of objects associated with this accounting
                        record. This is EUDAT specific. Default: "" - not set
  -o OBJECT_TYPE, --object_type OBJECT_TYPE
                        object type for the number of objects specified with
                        "-n". This is EUDAT specific. Default: "registered
                        objects"
  -k KEY, --key KEY     key used to refer to the record. If not set the
                        accounting server will create the key. Specifying an
                        existing key will overwrite the existing record.
                        Default: "" - not set
  -T TYPE, --type TYPE  type of the resource accounted. Default: storage
  -m MEASURE_TIME, --measure_time MEASURE_TIME
                        measurement time of the accounting record if different
                        from the current time. Default: "" - not set
  -C COMMENT, --comment COMMENT
                        arbitrary comment (goes into the meta dictionary).
                        Default: "" - not set
  -t, --test            Dry run. Don't push data to server - run only locally
                        Default: off
  -v, --verbose         return the key of the accounting record created.
                        Default: off

iRODScollector

$ bin/iRODScollector -h
usage: iRODScollector [-h] [--version] [-c CONFIGPATH] [-k KEY] [-T TYPE]
                      [-m MEASURE_TIME] [-C COMMENT] [-t] [-v]

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -c CONFIGPATH, --configpath CONFIGPATH
                        path to configuration file. Default:
                        "./irodscollector.cfg" (in the current working
                        directory)
  -k KEY, --key KEY     key used to refer to the record. If not set the
                        accounting server will create the key. Specifying an
                        existing key will overwrite the existing record.
                        Default: "" - not set
  -T TYPE, --type TYPE  type of the resource accounted. Default: storage
  -m MEASURE_TIME, --measure_time MEASURE_TIME
                        measurement time of the accounting record if different
                        from the current time. Default: "" - not set
  -C COMMENT, --comment COMMENT
                        arbitrary comment (goes into the meta dictionary).
                        Default: "" - not set
  -t, --test            Dry run. Don't push data to server - run only locally
                        Default: off
  -v, --verbose         return the key of the accounting record created.
                        Default: off

A template configuration file is included in the distribution and looks like this:

$ cat irodscollector.ini

#
# template of a configuration file for EUDAT's irodscollector
#

# section containing the logging options
[Logging]
log_file=eudatacct.log

# section containing the properties to access the accounting server
# to get statistical data and report them
[Report]
# base URL of the accounting server to be used
base_url=https://accounting.eudat.eu
# domain: either eudat or test or demo
domain=eudat
# uid of the corresponding registered storage resource on DPMT
# (same as storage_space_uuid on RCT)
account=<insert uid here>
# username of the provider on the accouniting server
# owning the account specified above
# contact [e-mail] if you need one
user=<username of provider>
# if you have an access token from RCT already reuse that here
password=<password or access token>
service_uuid=<unsuported at the moment>

# section contains the list of collections to be accounted together, replace
# the examples with your collections, the script sums the values of all
# collections and sends it to EUDAT's accounting service.
[Collections]
clist=
  /zone/some/path
  /zone/other/path

Copy this to irodscollector.cfg and adapt it to your site.

Most of this should be self-explaining. Note that you need to provide credentails for the accounting service. If you do not have any contact the EUDAT accounting manager.

In addition, you need to make sure that the user invoking this script has a suitable iRODS environment set up.

Basic usage information as well as error messages are logged to a file named .accounting.log in the current working directory from where addRecord has been invoked.

Developer notes

Please use a virtualenv to maintain this package, but I should not need to say that.

The package can be installed directly from GitHub:

$ pip install git+git://github.com/EUDAT-DPMT/eudat.accounting.client

The code is organized in a nested namespace package, i.e., the real action is happening in the subdirectory

$ cd src/eudat/accounting/client

Start looking around there.

Run the tests (not really that meaningful so far):

$ python setup.py test
$ python run_tests.py

Authors

• Raphael Ritz, MPCDF (main author)

• Claudio Cacciari, CINECA (initial iRODScollector)

• Pavel Weber, KIT (improved iRODScollector)

Links

Project home page

https://github.com/EUDAT-DPMT/eudat.accounting.client

Source code

https://github.com/EUDAT-DPMT/eudat.accounting.client

Issues tracker

https://github.com/EUDAT-DPMT/eudat.accounting.client/issues

1.0.1 (2017-08-25)

• addRecord now also supports the dry run option ‘-t’ This addresses issue #3 [raphael-ritz]

1.0.0 (2017-03-15)

• Adding iRODScollector based on Claudio’s and Pavel’s work [raphael-ritz]

• Python 3 compatibility [raphael-ritz]

1.0.0rc1 (2016-07-22)

• Initial release [raphael-ritz]