What is plumd?
Plumd is an extensible metrics collection tool that implements a reader/writer plugin framework designed from the start to support dimensional metrics.
- Linux /proc reader for detailed system metrics
- Nginx stub_status reader
- Memcache stats reader
- Redis info reader including per slave and db metrics plus command counts and timings (includes a builtin redis client)
- PHP-FPM reader including per php-fpm pool metrics, apc and opcache metrics
- MySQL reader for basic status and table metrics (manual install of pymysql required)
- Graphite writer supporting tcp and udp plaintext protocols
Consider Plumd beta quality.
That said, it is quite stable and does not consume any significant amount of resources even when the Graphite servers are down for extended periods. It has been in use in production environments of several hundred servers for many months now without incident.
The main rough areas are:
- the default values for the Linux /proc readers need tuning
- there may be potential issues with handling of counters wrapping (requires testing)
- unit tests for plugin related code need to be created
- metric names, plugin names, etc may change until v1.0 is released
Plumd is designed for Linux like systems and runs on Python 2.7, 3.x and PyPy. The only dependency is PyYaml and optionally pymysql if you are using the mysql reader.
- pip install plumd
- create: /etc/plumd.yaml - see test.main.yaml
- create: /etc/plumd.conf.d/proc.yaml - see proc.yaml
- create: /etc/plumd.conf.d/graphite.yaml - see graphitetcp.yaml
- test: plumd –config /etc/plumd.yaml
- configure plumd to run from your favorite process manager
- note: plumd does not require root access
Plumd has been designed with several requirements in mind. It must:
- Be reasonably lightweight
- Support dimensional metrics
- Be easily extended to read new metrics and write to new backends
- Support routing metrics to specific backends
The system runs in a single process and uses threads to isolate each plugin. A typical instance consumes ~14mb of memory and ~0.5% cpu during polls on most systems.
Creating a reader plugin is as simple as sub-classing plumd.Reader and overriding the poll() method. To use it install your plugin on the system, create a yaml configuration and restart plumd.
You can instantiate a given plugin multiple times by creating multiple configurations for it with different parameters. By default Readers write to all loaded Writers however can be configured to write to specific Writer(s).
Planned for v1.0
- flake8 / formatting
- fixup cyclo complexity of several somewhat hastily created classes/functions
- create file size reader - walk directory path looking for file patterns (globs), metric per path/file with file size (eg. for mysql db/table sizes)
- create couchbase reader
- create linux lvs reader
- documentation updates
- fixup defaults in linux proc, mysql readers - prune metrics to record, move to gauges/rates as appropriate
- add functionality for plugins to enable/disable themselves on startup (eg. disable conntrack reader if iptables is not running)
- metadata handling: raise exception on conflicts
- investigate possible issues with counter wraps in rate calcs
Planned Features / Changes - Short term
- installer helper - create configuration under specified directory and populate configuration files with defaults from all built-in and configured plugins
- add functionality for plugins to report all available metrics
- improve mysql reader, add calculated values, others
Potential Features / Changes - Long Term
- feature: writer plugins: statsd, chronix.io, opentsdb, influxdb, elastic, message queues, others
- feature: server plugins for eg. Prometheus compatible endpoint