Skip to main content

Network Telemetry And Monitoring

Project description


A Global-Scale, Network Telemetry Ecosystem

Table of Contents


Panoptes is a Python based network telemetry ecosystem that implements discovery, enrichment and polling. Key features include:

  • A modular design with well defined separation of concerns,
  • Plugin architecture that enables the implementation of any telemetry collection, enrichment, or transformation,
  • Horizontally scalable: supports clustering to add more capacity, and
  • Network telemetry specific constructs like SNMP abstractions, built in counter to gauge conversion.




Panoptes has been extensively tested on Redhat Linux, though it should run on any distribution that's compatible with LSB.


Panoptes currently supports Python 2.7 only. You can download the latest stable version of Python 2.7 from here


Before downloading and installing Panoptes, you would need the following services installed and configured


Panoptes has been tested with Redis 3.0.7, which can be downloaded from here


Panoptes has been tested with Zookeeper version 3.4.5 - download it from here


Panoptes has been tested with Kafka version - get it here


Install by running the following commands:

useradd panoptes
sudo -u panoptes
mkdir /home/panoptes/log
git clone
virtualenv -p2.7 panoptes_virt
source ~/panoptes_virt/bin/activate
cd ~/panoptes_virt
pip install -e.[deploy]


Panoptes is configured with ini style configuration files

/home/panoptes/conf/panoptes.conf is the main configuration file; you can find an example of this at examples/panoptes.ini


After adjusting the config files to your environment, start the following services. Note that these services run in the foreground and should be run under a job control system like supervisord or daemontools for production usage.

The services should be started in the order list below.

Discovery Plugin Scheduler

mkdir -p /home/panoptes/log/discovery/scheduler
cd ~/panoptes
celery beat -A yahoo_panoptes.discovery.discovery_plugin_scheduler -l info -S yahoo_panoptes.framework.celery_manager.PanoptesCeleryPluginScheduler

Discovery Plugin Agent

mkdir -p /home/panoptes/log/discovery/agent
cd ~/panoptes
celery worker -A yahoo_panoptes.discovery.discovery_plugin_agent -l info -f /home/panoptes/log/discovery/agent/discovery_plugin_agent_celery_worker.log -Q discovery_plugin_agent -n discovery_plugin_agent.%h

Resource Manager

mkdir -p /home/panoptes/log/resources/
cd ~/panoptes
python -m yahoo_panoptes.resources.manager

Enrichment Plugin Scheduler

mkdir -p /home/panoptes/log/enrichment/scheduler
cd ~/panoptes
celery beat -A yahoo_panoptes.enrichment.enrichment_plugin_scheduler -l info -S yahoo_panoptes.framework.celery_manager.PanoptesCeleryPluginScheduler --pidfile

Enrichment Plugin Agent

mkdir -p /home/panoptes/log/enrichment/agent
cd ~/panoptes
celery worker -A yahoo_panoptes.enrichment.enrichment_plugin_agent -l info -f /home/panoptes/log/enrichment/agent/enrichment_plugin_agent_celery_worker.log -Q enrichment_plugin_agent -n enrichment_plugin_agent.%h

Polling Plugin Scheduler

mkdir -p /home/panoptes/log/polling/scheduler
cd ~/panoptes
celery beat -A yahoo_panoptes.polling.polling_plugin_scheduler -l info -S yahoo_panoptes.framework.celery_manager.PanoptesCeleryPluginScheduler --pidfile

Polling Plugin Agent

mkdir -p /home/panoptes/log/polling/agent
echo 'SET panoptes:secrets:snmp_community_string:<site> <snmp community string>' | redis-cli
celery worker -A yahoo_panoptes.polling.polling_plugin_agent -l info -f /home/panoptes/log/polling/agent/polling_plugin_agent_celery_worker_001.log -Q polling_plugin_agent -n polling_plugin_agent_001.%h -Ofair --max-tasks-per-child 10


We welcome issues, questions, and pull requests.



This project is licensed under the terms of the Apache 2.0 open source license. Please refer to LICENSE for the full terms.

Project details

Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for yahoo-panoptes, version 1.0.22
Filename, size File type Python version Upload date Hashes
Filename, size yahoo_panoptes-1.0.22.tar.gz (114.9 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page