Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

pmacct-to-elasticsearch

pmacct-to-elasticsearch is a python script designed to read output from pmacct daemons, to process it and to store it into ElasticSearch. It works with both memory and print plugins and, optionally, it can perform manipulations on data (such as to add fields on the basis of other values).

img/data_flow.png
  1. pmacct daemons collect IP accounting data and process them with their plugins;

  2. data are stored into in-memory-tables (memory plugins), JSON or CSV files (print plugins);

  3. crontab jobs (memory plugins) or trigger scripts (print plugins) are invoked to execute pmacct-to-elasticsearch;

  4. pmacct’s output records are finally processed by pmacct-to-elasticsearch, which reads them from stdin (memory plugins) or directly from file.

Optionally, some data transformations can be configured, to allow pmacct-to-elasticsearch to add or remove fields to/from the output documents that are sent to ElasticSearch for indexing. These additional fields may be useful to enhance graphs and reports legibility, or to add a further level of aggregation or filtering.

Installation

Install the program using pip:

pip install pmacct-to-elasticsearch

Then clone the repository and run the ./install script to setup your system:

cd /usr/local/src/
git clone https://github.com/pierky/pmacct-to-elasticsearch.git
cd pmacct-to-elasticsearch/
./install

Configuration

Please refer to the CONFIGURATION.md file. The TRANSFORMATIONS.md file contains details about data transformations configuration.

A simple tutorial on pmacct integration with ElasticSearch/Kibana using pmacct-to-elasticsearch can be found at http://blog.pierky.com/integration-of-pmacct-with-elasticsearch-and-kibana.

Future work

  • Add support of more pmacct output formats (Apache Avro, …).

Author

Pier Carlo Chiodi - https://pierky.com/

Blog: https://blog.pierky.com/ Twitter: @pierky

Changelog

0.3.0a5

  • Fix issue with index creation on ElasticSearch 5.x.

    Thanks to Kristoffer Olsson and Daniel Lindberg for reporting this and for their extensive support.

  • Improved template for index creation.

0.3.0a4

  • Fix an issue with transformations.

0.3.0a3

  • More command line arguments.

    The command line arguments under the Configuration options group can be used to override settings done on the plugin configuration file.

0.3.0a2

  • CSV output support.

    The InputFormat option in the plugin configuration file can be used to instruct pmacct-to-elasticsearch to parse CSV output from pmacct.

0.3.0a1

  • Multithreading support.

    The ReaderThreads option in the plugin configuration file sets the number of threads used to process pmacct’s output.

Metadata

Release files for pmacct-to-elasticsearch 0.3.0a5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pmacct-to-elasticsearch 0.3.0a5
File Size Uploaded Provenance Actions
pmacct-to-elasticsearch-0.3.0a5.tar.gz 12.4 kB Without provenance Details for pmacct-to-elasticsearch-0.3.0a5.tar.gz

Release files / pmacct-to-elasticsearch-0.3.0a5.tar.gz

Download URL pmacct-to-elasticsearch-0.3.0a5.tar.gz
Size 12.4 kB
Tags Source
SHA-256 checksum
How to use checksums
a658ee07c0d3effc94e2a11cbc1a4a2fc2c6271300ed804cf26309dd0db64f60
BLAKE2b-256 checksum
How to use checksums
05d51fbe0fc2e6414fb67fcd056ded8a4d6e47b73c951ab2e54aac887c074858
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for pmacct-to-elasticsearch 0.3.0a5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page