Python bindings for C++ library for reading Prometheus on-disk data
Project description
pypdu
This module provides basic read-only access to the data contained in Prometheus on-disk files from Python.
pypdu
may be installed from pip (on linux and macOS):
pip install pypdu
Example usage:
#!/usr/bin/env python3
import pypdu
data = pypdu.load("/path/to/stats_data")
for series in data:
print(series.name) # equivalent to series.labels["__name__"]
print(series.labels)
print(len(series.samples)) # number of samples can be computed
# without iterating all of them
for sample in series.samples:
print(f"{sample.timestamp} : {sample.value}")
Or the series and samples can be unpacked:
for name, labels, samples in data:
print(name)
print(labels)
print(len(samples))
for timestamp, value in samples:
print(f"{timestamp} : {value}")
Filtering time series
If only a subset of the time series are desired, pypdu
can filter them based on label values, and avoid parsing unneeded series at all:
for series in data.filter({"__name__":"sysproc_page_faults_raw"}):
This will usually perform better than filtering "manually" in python after the fact.
Multiple labels can be specified:
data.filter({"__name__":"sysproc_page_faults_raw", "proc":"memcached"})
ECMAScript regexes can also be used:
data.filter({"proc":pypdu.regex("^go.*")})
Or even arbitrary Python callbacks:
data.filter({"proc":lambda x: x.startswith("go")})
As shorthand, when filtering on __name__
alone, just a string may be provided.
data.filter("sysproc_page_faults_raw")
Single series lookup
If there is only one time series matching your filter, for convenience you can do:
foobar_series = data[{"__name__":"foobar"}]
This is roughly equivalent to:
foobar_series = next(iter(data.filter({"__name__":"foobar"})))
If there are multiple time series matching your filter, this will silently discard all but the lexicographically first (sorted by the key and value of all labels).
If none match, a KeyError
is raised.
All types of filter demonstrated above with .filter(...)
may be used in this manner also.
Histograms
PrometheusData(...).histograms
allows iterating all histograms represented by the time series in a data directory.
The histograms are exposed as HistogramTimeSeries
, grouping all the component ..._bucket
time series together. Indexing into this series provides access to the histogram at a single point in time.
e.g.,
data = pypdu.load("<...>")
for histSeries in data.histograms:
print("Labels: ", histSeries.labels)
print("Number of samples: ", len(histSeries))
for hist in histSeries:
print("TS: ", hist.timestamp)
print(hist.buckets())
Iterates over every histogram found in the Prometheus data, then iterates over every sample contained in that time series.
Example output:
Labels: {'__name__': 'cm_http_requests_seconds', 'instance': 'ns_server', 'job': 'ns_server_high_cardinality'}
Number of samples: 3826
TS: 1621268098827
[(0.001, 8.0), (0.01, 25.0), (0.1, 25.0), (1.0, 25.0), (10.0, 25.0), (inf, 25.0)]
TS: 1621268158827
[(0.001, 39.0), (0.01, 118.0), (0.1, 126.0), (1.0, 127.0), (10.0, 127.0), (inf, 127.0)]
TS: 1621268218827
[(0.001, 43.0), (0.01, 132.0), (0.1, 140.0), (1.0, 141.0), (10.0, 141.0), (inf, 141.0)]
TS: 1621268278827
[(0.001, 48.0), (0.01, 145.0), (0.1, 153.0), (1.0, 154.0), (10.0, 154.0), (inf, 154.0)]
TS: 1621268338827
[(0.001, 53.0), (0.01, 158.0), (0.1, 166.0), (1.0, 167.0), (10.0, 167.0), (inf, 167.0)]
TS: 1621268398827
[(0.001, 55.0), (0.01, 171.0), (0.1, 179.0), (1.0, 180.0), (10.0, 180.0), (inf, 180.0)]
TS: 1621268458827
[(0.001, 60.0), (0.01, 191.0), (0.1, 199.0), (1.0, 200.0), (10.0, 200.0), (inf, 200.0)]
TS: 1621268518827
[(0.001, 66.0), (0.01, 204.0), (0.1, 212.0), (1.0, 213.0), (10.0, 213.0), (inf, 213.0)]
TS: 1621268578827
[(0.001, 71.0), (0.01, 217.0), (0.1, 225.0), (1.0, 226.0), (10.0, 226.0), (inf, 226.0)]
TS: 1621268638827
[(0.001, 73.0), (0.01, 230.0), (0.1, 238.0), (1.0, 239.0), (10.0, 239.0), (inf, 239.0)]
...
Labels: ...
HistogramTimeSeries
(in the above example, this is histSeries
), can be indexed into - currently
only by a sample index, but in the future, selecting the histogram closest to a given timestamp may be supported.
E.g., the first and last point in time view available for a specific histogram can be found with:
first = histSeries[0]
last = histSeries[-1]
From which the timestamp and buckets could be read:
>>> print(last.timestamp) # time since epoch in ms
1631007596974
>>> print(last.bucket_bounds()))
[0.001, 0.01, 0.1, 1.0, 10.0, inf]
>>> print(last.bucket_values())
[4279.0, 4371.0, 4666.0, 5044.0, 5044.0, 5044.0]
>>> print(last.buckets()) # convenience zip of (bounds, values)
[(0.001, 4279.0), (0.01, 4371.0), (0.1, 4666.0), (1.0, 5044.0), (10.0, 5044.0), (inf, 5044.0)]
Alternative installation steps
setup.py
pypdu
may be installed without pip
. To use, clone the repository as in the installation instructions.
Then run:
python setup.py install
This should also work on Windows, but is untested.
manual .so
Alternatively, following the cmake
steps in the installation instructions to build the project produces a module with a platform-dependent name - for example on MacOS this may be pypdu.cpython-39-darwin.so
.
This can be found either in <build dir>/src/pypdu
or in your chosen installation prefix. This can be used without installing with setup.py
, simply ensure the containing directory is in your PYTHONPATH
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pypdu-0.0.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 132f88f39fe42967047087539f3d77e13e8fc77842b725dfa64db0fcc20e36bb |
|
MD5 | 4b108b3496757b736b4c8a459935cd9a |
|
BLAKE2b-256 | 104d32b970a0928c09f8cf4bbfc14bd077c7ff8ce945b87ff005921337df8904 |
Hashes for pypdu-0.0.9-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9cd1379244952b26ed043d358245c55228c887e0ca2997603b1e1dd38ee73fac |
|
MD5 | 6196b4d27ac241acf504a1411e13b567 |
|
BLAKE2b-256 | feeb1a9a46dd2e7c7cfeb5729086829214060828c98c40624f4dfccee86a8bec |
Hashes for pypdu-0.0.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ef53079db9787b42ee920e489d92c3487a2bf72f68a7da61467edf74668e605f |
|
MD5 | 24df28cdec8674871bbc5252b2a25c00 |
|
BLAKE2b-256 | 2be7f9a6f2ab76607f8bc8a2847ab5a1efe34dbe5ae8e2455b35c133af335ba6 |
Hashes for pypdu-0.0.9-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 568c8c94fe9c39a8d802fd6e442fbd929d8e2d9d8f2cddb672bab330ce572a53 |
|
MD5 | 38c70ae6d0ea9b78f453cd6fb7272ab3 |
|
BLAKE2b-256 | 298e685c847c94c8556beae2a1832aa94c395c7fb89dad9cf857f319e9890356 |
Hashes for pypdu-0.0.9-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 298c43b95abf46066d076ec665af5cd16cf430112d00a608430080541c81f83e |
|
MD5 | f4b7102a852de34de0ebdc6c9ef35dfd |
|
BLAKE2b-256 | be890d2fd8e30f09401a697d31044fe72cfce2d672c85c44457d1567a9d0d7b2 |
Hashes for pypdu-0.0.9-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1efaaa81008f88b72d99c5c1708d74f025e93550116446a48ccd61478fe64e48 |
|
MD5 | 8bb50181caab4388d33276368f85990f |
|
BLAKE2b-256 | f6fb2dc23733cfb43baaeff1f9579b695f5df89058971132988274685779e616 |
Hashes for pypdu-0.0.9-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36f2e47c73d7b1d7074a55e6629e815e6950131ff3bb5c02d71be2853ceacbb2 |
|
MD5 | 9f72ed6908fb678bded9380639474b0c |
|
BLAKE2b-256 | 6314e39dc6cf7c97e38c16a32972b5019bef3576aefe5b5ae493047455944699 |
Hashes for pypdu-0.0.9-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7de1885968cfb10aae05ea65f04bdf0e7792b2c2c53ac92a82aab7c4bf4c99fe |
|
MD5 | d7a1efe598675ea20bd9eb8af5c25bb0 |
|
BLAKE2b-256 | c7e746896f04a6664ddc53ccbcc0b16f48e72d8581ed51c23dd5fae087a4a032 |
Hashes for pypdu-0.0.9-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1c78c2f4bcc8c6fcf502dfb121e34ab70978f60f854698f288e655bbf388f5c5 |
|
MD5 | cb2374a8510df2e77a3313a33cb11792 |
|
BLAKE2b-256 | 6fa8bf779c32a6c56ae4b3916a8c4cec288378b31b0f125967f20674f266aec9 |
Hashes for pypdu-0.0.9-cp36-cp36m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d88433b45f70640cbdde545e0d576c7ab05dd98e816a45a354cc672b5f96bea0 |
|
MD5 | 662e8839696c0230701183546c0a207a |
|
BLAKE2b-256 | 55aa643f50c00bdef8d23d89cda1721fd9cd24db5465e1c12c650d5fc6015c83 |
Hashes for pypdu-0.0.9-cp27-cp27mu-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 761b614d35471e08c15f3c6f9649dad2c606f685768addae7fb7fad250e9f0bb |
|
MD5 | 266118450113a7fd0b58118587a74d98 |
|
BLAKE2b-256 | 0b6b0e460f10a8884fc78f6cadfd72f0d42b3570e061d0572c05e568ed3e1543 |
Hashes for pypdu-0.0.9-cp27-cp27m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 95d996bcf74ab5f43e8e0c2edfba6d13be1da0c9f6c54de9b87caacead6204e9 |
|
MD5 | 01e03a317e05c0e78b6214fb4e4fde18 |
|
BLAKE2b-256 | baf7d884734f0cc290c9a99df1eeef19a7095fee1b91b3bc57e2e2c064b2d08a |