A package to collect and analyze basic performance data from clusters
This is work in progress!
FrUCToSA stands for Frankfurt University Cluster Tool of Software Analysis.
It is a simple tool to
- collect, and
data from software running on a cluster (like Goethe-HLR) —and from the cluster itself— and to analyze that data with performance “in mind”.
FrUCToSA is made of two elements:
- LiMon: a Light Monitor that collects data.
- PerA: a Performance Analyzer that analyzes the data and classifies it. Warning: TBD!
The package provides several programs:
- fructosad: main program. It works in the background (as a service/daemon) orchestrating all the system. Warning: TBD!
- lagent: LiMon agent. Customizable service/daemon that collects performance data from the a node on the cluster. It runs locally and collects data from sensors that can be activated and configured via a configuration file.
- lmaster: LiMon master. Another service/daemon that controls the agents and collects data from them. It runs in a master node of the cluster; it is controled via a configuration file.
- perad: another service/daemon that analyzes the data and classifies it according to the performance Warning: TBD!
FrUCToSA is developed in Python at the CSC (Goethe Universitaet Frankfurt) under the GPL3 license.
Just install the FrUCToSA package from PyPI:
$ pip install FrUCToSA
Configure lmaster and lagent
[optional] Start Graphite (the data will be sent to it)
[optional] Start Grafana and connect it to Graphite: this is an easy way to have nice plots and dashboards.
Start lmaster (see the output of lmaster -h, to learn about some parameters to control lmaster):
# lmaster start
A configuration file is needed. By default the configuration file is /etc/fructosa/lmaster.conf but that can be changed from the command line.
A typical configuration might be:
[lmaster] host = localhost incoming data port = 7888 [Graphite] host = localhost carbon receiver pickle port = 2004 [logging] filename = /tmp/limon.log maxBytes = 1073741824 backupCount = 10 level = DEBUG
No option is mandatory. In the file fructosa/constants.py the defaults are defined.
Start lagent on one or several computers (again, see the output of lagent -h, for more details):
# lagent start
A configuration file is needed. By default the configuration file is /etc/fructosa/lagent.conf, but can be changed from the command line. In this configuration file is where the needed sensors are activated. A typical configuration with the master and the agent running locally looks like this:
[lmaster] host = localhost incoming data port = 7888 [logging] filename = /tmp/limon.log maxBytes = 1073741824 backupCount = 10 level = DEBUG [sensor:CPUPercent] time_interval = 10 [sensor:VirtualMemory] time_interval = 30 [sensor:CPUTimes] time_interval = 30 [sensor:CPUTimesPercent] time_interval = 10 [sensor:CPUCount] time_interval = 300 [sensor:CPUStats] time_interval = 30 [sensor:CPUFreq] time_interval = 300 [sensor:SwapMemory] time_interval = 60 [sensor:DiskPartitions] time_interval = 60 [sensor:DiskUsage] time_interval = 30 #path = / [sensor:DiskIOCounters] time_interval = 20 [sensor:NetIOCounters] time_interval = 10 [sensor:NetConnections] time_interval = 20 [sensor:NetIFAddrs] time_interval = 30 [sensor:NetIFStats] time_interval = 30 [sensor:SensorsTemperatures] time_interval = 30 [sensor:SensorsFans] time_interval = 30 [sensor:SensorsBattery] time_interval = 30 [sensor:BootTime] time_interval = 300 [sensor:Users] time_interval = 10
This is just an example with many sensors active. No option is mandatory. But if lagent must measure anything, some sensor must be explicitly given. In the file fructosa/constants.py the defaults are defined. The time given in the time_interval option is understood to be in seconds. If the agent runs on a different computer as the master, the host entry in the lmaster section must be adjusted accordingly.
All the communications between lmaster and lagent are bona fide. There are no authentication or encryption mechanisms for now.
- Change constants.py: lagent, lmaster -> limon
- lagent should have an option to display the available sensors and some help for each sensor.
- Sensors should accept options: the mechanism is almost there, but need to be completed.
- Add sensors to read data from GPUs.
- Connect to Slurm.
- Add configuration options to manage Graphite:
- send data to it or not? (yes by default)
- openrc scripts to manage the whole system:
- start graphite
- start grafana (?)
- start lmaster
- start lagents where needed
- stop lagents
- stop lmaster
- stop grafana (?)
- stop graphite
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size FrUCToSA-0.1.1-py3-none-any.whl (34.8 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size FrUCToSA-0.1.1.tar.gz (84.7 kB)||File type Source||Python version None||Upload date||Hashes View|