NOTED: a framework to optimise network traffic via the analysis of data from File Transfer Services
Project description
NOTED: a framework to optimise network traffic via the analysis of data from File Transfer Services
Features:
- Get data from
CRIC database
to understand the topology of the network for a given {src, dst} pair:- Get
rc_site
,federation
,endpoints
andIPv4/IPv6 addresses
. - Generate a dataframe with the associated parameters per {src, dst} endpoint.
- Get
- Based on the parameters inserted by the user in
src/noted/config/config.yaml
:- Get
max_threshold
,min_threshold
andaction
to execute once the link is saturated. - Generate the
query
files to accessFTS parameters
in CERN Monit Kibana database per {src, dst} pair. Curl
toFTS CERN Monit Kibana
based on the query files.
- Get
- If the
bidirectional
link is chosen by the user: launch two threads per {src, dst} for monitoring the transfers: {src -> dst}, {dst -> src}. Otherwise only one thread is created {src -> dst} - While monitoring:
- If
throughput > max_threshold
:- If NOTED interprets that the link will be
saturated
for a long period of time:- Generate an
alert
(email notification). Provide dynamic circuit
withSENSE-O AutoGole
for the given {src, dst} pair.- Wait until NOTED interprets that the link will
not be saturated
anymore. Cancel dynamic circuit
withSENSE-O AutoGole
.
- Generate an
- If NOTED interprets that the link will be
Log
file to store the transfers per {src, dst} pair.
- If
Copyright:
© Copyright 2022 CERN. This software is distributed under the terms of
the GNU General Public Licence version 3 (GPL Version 3), copied verbatim
in the file "LICENCE.txt". In applying this licence, CERN does not waive
the privileges and immunities granted to it by virtue of its status as an
Intergovernmental Organization or submit itself to any jurisdiction.
Compilation steps:
MY_AUTH_TOKEN=my_personal_auth_token
echo $MY_AUTH_TOKEN
find . -name params.ini
sed -i "" "s/auth_token =.*/auth_token = $MY_AUTH_TOKEN/g" ./venv-noted/lib/python3.8/site-packages/noted/params/params.ini
or
nano ./venv-noted/lib/python3.8/site-packages/noted/params/params.ini
noted src/noted/config/config-example.yaml [--verbosity debug/info/warning]
Program description:
python3 main.py -h
usage: main.py [-h] [-v VERBOSITY] config_file
NOTED: a framework to optimise network traffic via the analysis of data from File Transfer Services.
positional arguments:
config_file the name of the configuration file [config-example.yaml]
optional arguments:
-h, --help show this help message and exit
-v VERBOSITY, --verbosity VERBOSITY defines the logging level [debug, info, warning]
Structure of the project:
noted_transfer_broker
.
├── README.md
└── noted_transfer_broker
├── COPYRIGHT.txt
├── LICENCE.txt
├── README.md
├── build
│ └── lib
│ └── noted
│ ├── __init__.py
│ ├── main.py
│ └── plot_transfers.py
├── dist
│ └── noted-dev-1.0.2.tar.gz
├── setup.cfg
├── setup.py
├── src
│ ├── noted
│ │ ├── __init__.py
│ │ ├── config
│ │ │ └── config-example.yaml
│ │ ├── documentation
│ │ │ ├── noted_main_function_documentation.pdf
│ │ │ ├── noted_transfer_broker_class_documentation.pdf
│ │ │ ├── reduced_noted_main_function_documentation.pdf
│ │ │ └── reduced_noted_transfer_broker_class_documentation.pdf
│ │ ├── html
│ │ │ ├── TransferBroker.html
│ │ │ └── main.html
│ │ ├── logs
│ │ │ ├── noted_email.txt
│ │ │ ├── transfer_broker_1660577038.log
│ │ │ └── transfer_broker_1660577485.log
│ │ ├── main.py
│ │ ├── modules
│ │ │ ├── plot_transfers.py
│ │ │ └── transferbroker.py
│ │ ├── params
│ │ │ └── params.ini
│ │ ├── query
│ │ │ ├── query_monit_prod_fts_raw_queue_dst_rcsite
│ │ │ └── query_monit_prod_fts_raw_queue_src_rcsite
│ │ ├── requirements.txt
│ │ ├── sense-o
│ │ │ ├── sense-cancel.sh
│ │ │ └── sense-provision.sh
│ │ └── transfers
│ │ ├── transfer_broker_all_transfers_1660577038.txt
│ │ ├── transfer_broker_dst_rcsite_1660577038.txt
│ │ └── transfer_broker_src_rcsite_1660577038.txt
│ └── noted_dev.egg-info
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ ├── entry_points.txt
│ ├── requires.txt
│ └── top_level.txt
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
noted-dev-1.1.12.tar.gz
(289.1 kB
view hashes)
Built Distribution
noted_dev-1.1.12-py3.9.egg
(300.1 kB
view hashes)