An Elasticsearch tail
Project description
Follow Elasticsearch continuously
Overview
elasticsearch_follow is library helping to query Elasticsearch continuously.
It needs https://github.com/elastic/elasticsearch-py as a dependency.
elasticsearch_follow acts as a wrapper for elasticsearch-py and handles various use-cases, like following logs by polling elasticsearch continuously and fetching loglines via a generator. It is possible to easily fetch lines surrounding a given logline.
How to poll Elasticsearch continuously
The polling logic is implemented in the class ElasticsearchFollow, which needs an Elasircsearch object from elasticsearch-py. The class Follower takes an ElasticsearchFollow-object and has a method to create a generator which yields loglines until all elements of a query have been returned. After this a new generator has to be created and used.
How to fetch log-lines from Elasticsearch
To just fetch loglines, one can use ElasticsearchFetch which has a search_surrounding. This returns a list of lists, where each list contains the queried loglines and the lines before and after as requested by the parameters num_before and num_after.
Installation
You can install the elasticsearch
package with pip:
pip install elasticsearch_follow
See also: https://pypi.org/project/elasticsearch-follow/
Example usage
This package introduces the command line tool es_tail
which can be used for
following logs written to Elasticsearch and directly fetching log lines by a query.
It is possible to configure the output via an format string.
# Follow the logs written to the indexes starting with logstash.
# Print the fieds @timestamp and message
es_tail -c "http://localhost:9200" tail --index "logstash*" -f "{@timestamp} {message}"
# Fetch all logs in the last hour with the field loglevel contains ERROR and fetch the two lines before and after.
# Print the fieds @timestamp and message
es_tail -c "http://localhost:9200" fetch --index "logstash" -f "{@timestamp} {message}" --query loglevel:ERROR -A 2 -B 2 -F "now-1h"
# It is also possible to print nested fields
es_tail -c "http://localhost:9200" fetch --index "logstash" -f "{@timestamp} {message} {kv[field]} {kv[nested][field]}" -F "now-1h"
The command line options can also be given via environment variables by using the prefix ES_TAIL
.
For example
export ES_TAIL_USERNAME='username'
export ES_TAIL_PASSWORD='password'
es_tail -c http://localhost:9200 tail
Example usage of the library
from elasticsearch import Elasticsearch
from elasticsearch_follow import ElasticsearchFollow, Follower
es = Elasticsearch()
es_follow = ElasticsearchFollow(elasticsearch=es)
# The Follower is used to get a generator which yields new
# elements until it runs out. time_delta give the number of
# seconds to look into the past.
follower = Follower(elasticsearch_follow=es_follow, index='some-index', time_delta=60)
while True:
entries = follower.generator()
for entry in entries:
print(entry)
time.sleep(0.1)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for elasticsearch_follow-0.2.6.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7c91224dc27cb5d847cbeac6d283c65a4bcccb54360c03f5046fce224f47c68d |
|
MD5 | 3c614fb4e26aa2311bb47dd1f1f31deb |
|
BLAKE2b-256 | 56a971bea0ae1bbb148c858f8d1c0f0d71b7a4c8d065c7b3a8353b8dcf0eb419 |
Hashes for elasticsearch_follow-0.2.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e96fc9b6123a4dd3754d548b7e13ec999e78a9b66cb58d7721b5d7c9287a6946 |
|
MD5 | 41a386ab2f5971600b293ecf2f5ef4a3 |
|
BLAKE2b-256 | 3c5ac8f92dd9ea351661a4c6847dba1fb7ffc6fa0dfa36893506b4f515ca0d19 |