A Python library for easy interaction with the Humio API
Project description
Humio API (unofficial lib)
This project requires
Python>=3.6.1
This is an unofficial library for interacting with Humio's API. If you're looking for the official Python Humio library it can be found here: humiolib. This library mostly exists because the official library was incredibly basic back in 2019 when I needed this.
Installation
pip install humiocli
Main features
- Asyncronous and syncronous streaming queries supported by
httpx
. - Queryjobs which can be polled once, or until completed.
- Chainable relative time modifiers (similar to Splunk e.g.
-1d@h-30m
). - List repository details (NOTE: normal Humio users cannot see repos without read permission).
- Easy env-variable based configuration.
- Ingest data to Humio, although you probably want to use Filebeat for anything other than one-off things to your sandbox.
- CLI companion tool available at humiocli.
- Create and update parsers.
- (Work in progress) An updateable timeseries, which can follow a moving timewindow using relative modifiers, optionally querying only the changed timewindow since previous update.
Usage
For convenience your Humio URL and token should be set in the environment variables HUMIO_BASE_URL
and HUMIO_TOKEN
. These can be set in ~/.config/humio/.env
and loaded by humioapi.loadenv()
.
Query repositories
Create an instance of HumioAPI to get started
import humioapi
humioapi.setup_excellent_logging('INFO')
api = humioapi.HumioAPI(**humioapi.loadenv())
repositories = api.repositories()
Iterate over syncronouys streaming searches sequentially
import humioapi
humioapi.setup_excellent_logging('INFO')
api = humioapi.HumioAPI(**humioapi.loadenv())
stream = api.streaming_search(
query="log_type=trace user=someone",
repos=['frontend', 'backend', 'integration'],
start="-1week@day",
stop="now"
)
for event in stream:
print(event)
Itreate over asyncronous streaming searches in parallell, from a syncronous context
import asyncio
import humioapi
humioapi.setup_excellent_logging('INFO')
api = humioapi.HumioAPI(**humioapi.loadenv())
loop = asyncio.new_event_loop()
try:
asyncio.set_event_loop(loop)
tasks = api.async_streaming_tasks(
loop,
query="log_type=trace user=someone",
repos=['frontend', 'backend', 'integration'],
start="-1week@day",
stop="now",
concurrent_limit=10,
)
for event in humioapi.consume_async(loop, tasks):
print(event)
finally:
try:
loop.run_until_complete(loop.shutdown_asyncgens())
finally:
asyncio.set_event_loop(None)
loop.close()
Jupyter Notebook
pew new --python=python36 humioapi
# run the following commands inside the virtualenv
pip install git+https://github.com/gwtwod/humio-api.git
pip install ipykernel seaborn matplotlib
python -m ipykernel install --user --name 'python36-humioapi' --display-name 'Python 3.6 (venv humioapi)'
Start the notebook by running jupyter-notebook
and choose the newly created kernel when creating a new notebook.
Run this code to get started:
import humioapi
humioapi.setup_excellent_logging('INFO')
api = humioapi.HumioAPI(**humioapi.loadenv())
results = api.streaming_search(query='log_type=trace user=someone', repos=['frontend', 'backend'], start="@d", stop="now")
for i in results:
print(i)
To get a list of all readable repositories with names starting with 'frontend':
repos = sorted([k for k,v in api.repositories().items() if v['read_permission'] and k.startswith('frontend')])
Making a timechart (lineplot):
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
sns.set(color_codes=True)
sns.set_style('darkgrid')
results = api.streaming_search(query='log_type=stats | timechart(series=metric)', repos=['frontend'], start=start, stop=stop)
df = pd.DataFrame(results)
df['_count'] = df['_count'].astype(float)
df['_bucket'] = pd.to_datetime(df['_bucket'], unit='ms', origin='unix', utc=True)
df.set_index('_bucket', inplace=True)
df.index = df.index.tz_convert('Europe/Oslo')
df = df.pivot(columns='metric', values='_count')
sns.lineplot(data=df)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.