Python bindings for C++ library for reading Prometheus on-disk data
Project description
pypdu
This module provides basic read-only access to the data contained in Prometheus on-disk files from Python.
pypdu
may be installed from pip (on linux and macOS):
pip install pypdu
Example usage:
#!/usr/bin/env python3
import pypdu
data = pypdu.load("/path/to/stats_data")
for series in data:
print(series.name) # equivalent to series.labels["__name__"]
print(series.labels)
print(len(series.samples)) # number of samples can be computed
# without iterating all of them
for sample in series.samples:
print(f"{sample.timestamp} : {sample.value}")
Or the series and samples can be unpacked:
for name, labels, samples in data:
print(name)
print(labels)
print(len(samples))
for timestamp, value in samples:
print(f"{timestamp} : {value}")
Filtering time series
If only a subset of the time series are desired, pypdu
can filter them based on label values, and avoid parsing unneeded series at all:
for series in data.filter({"__name__":"sysproc_page_faults_raw"}):
This will usually perform better than filtering "manually" in python after the fact.
Multiple labels can be specified:
data.filter({"__name__":"sysproc_page_faults_raw", "proc":"memcached"})
ECMAScript regexes can also be used:
data.filter({"proc":pypdu.regex("^go.*")})
Or even arbitrary Python callbacks:
data.filter({"proc":lambda x: x.startswith("go")})
As shorthand, when filtering on __name__
alone, just a string may be provided.
data.filter("sysproc_page_faults_raw")
Single series lookup
If there is only one time series matching your filter, for convenience you can do:
foobar_series = data[{"__name__":"foobar"}]
This is roughly equivalent to:
foobar_series = next(iter(data.filter({"__name__":"foobar"})))
If there are multiple time series matching your filter, this will silently discard all but the lexicographically first (sorted by the key and value of all labels).
If none match, a KeyError
is raised.
All types of filter demonstrated above with .filter(...)
may be used in this manner also.
Histograms
PrometheusData(...).histograms
allows iterating all histograms represented by the time series in a data directory.
The histograms are exposed as HistogramTimeSeries
, grouping all the component ..._bucket
time series together. Indexing into this series provides access to the histogram at a single point in time.
e.g.,
data = pypdu.load("<...>")
for histSeries in data.histograms:
print("Labels: ", histSeries.labels)
print("Number of samples: ", len(histSeries))
for hist in histSeries:
print("TS: ", hist.timestamp)
print(hist.buckets())
Iterates over every histogram found in the Prometheus data, then iterates over every sample contained in that time series.
Example output:
Labels: {'__name__': 'cm_http_requests_seconds', 'instance': 'ns_server', 'job': 'ns_server_high_cardinality'}
Number of samples: 3826
TS: 1621268098827
[(0.001, 8.0), (0.01, 25.0), (0.1, 25.0), (1.0, 25.0), (10.0, 25.0), (inf, 25.0)]
TS: 1621268158827
[(0.001, 39.0), (0.01, 118.0), (0.1, 126.0), (1.0, 127.0), (10.0, 127.0), (inf, 127.0)]
TS: 1621268218827
[(0.001, 43.0), (0.01, 132.0), (0.1, 140.0), (1.0, 141.0), (10.0, 141.0), (inf, 141.0)]
TS: 1621268278827
[(0.001, 48.0), (0.01, 145.0), (0.1, 153.0), (1.0, 154.0), (10.0, 154.0), (inf, 154.0)]
TS: 1621268338827
[(0.001, 53.0), (0.01, 158.0), (0.1, 166.0), (1.0, 167.0), (10.0, 167.0), (inf, 167.0)]
TS: 1621268398827
[(0.001, 55.0), (0.01, 171.0), (0.1, 179.0), (1.0, 180.0), (10.0, 180.0), (inf, 180.0)]
TS: 1621268458827
[(0.001, 60.0), (0.01, 191.0), (0.1, 199.0), (1.0, 200.0), (10.0, 200.0), (inf, 200.0)]
TS: 1621268518827
[(0.001, 66.0), (0.01, 204.0), (0.1, 212.0), (1.0, 213.0), (10.0, 213.0), (inf, 213.0)]
TS: 1621268578827
[(0.001, 71.0), (0.01, 217.0), (0.1, 225.0), (1.0, 226.0), (10.0, 226.0), (inf, 226.0)]
TS: 1621268638827
[(0.001, 73.0), (0.01, 230.0), (0.1, 238.0), (1.0, 239.0), (10.0, 239.0), (inf, 239.0)]
...
Labels: ...
HistogramTimeSeries
(in the above example, this is histSeries
), can be indexed into - currently
only by a sample index, but in the future, selecting the histogram closest to a given timestamp may be supported.
E.g., the first and last point in time view available for a specific histogram can be found with:
first = histSeries[0]
last = histSeries[-1]
From which the timestamp and buckets could be read:
>>> print(last.timestamp) # time since epoch in ms
1631007596974
>>> print(last.bucket_bounds()))
[0.001, 0.01, 0.1, 1.0, 10.0, inf]
>>> print(last.bucket_values())
[4279.0, 4371.0, 4666.0, 5044.0, 5044.0, 5044.0]
>>> print(last.buckets()) # convenience zip of (bounds, values)
[(0.001, 4279.0), (0.01, 4371.0), (0.1, 4666.0), (1.0, 5044.0), (10.0, 5044.0), (inf, 5044.0)]
Alternative installation steps
setup.py
pypdu
may be installed without pip
. To use, clone the repository as in the installation instructions.
Then run:
python setup.py install
This should also work on Windows, but is untested.
manual .so
Alternatively, following the cmake
steps in the installation instructions to build the project produces a module with a platform-dependent name - for example on MacOS this may be pypdu.cpython-39-darwin.so
.
This can be found either in <build dir>/src/pypdu
or in your chosen installation prefix. This can be used without installing with setup.py
, simply ensure the containing directory is in your PYTHONPATH
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pypdu-0.0.11-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 81174ad1da7c1a8cffdd7bacbad06076f2968dbcc9327a5d87ccce3a000f7758 |
|
MD5 | 74a3b91cef7b4dda148f73cbd60878ff |
|
BLAKE2b-256 | a672989bbd61e87d48aba4db8b87142c54c6a9ca3b1b1cfa36ef3e6559214c35 |
Hashes for pypdu-0.0.11-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b65db7c8affcd5c958348b87c1d0179b61cddcb7f5cf290958bc12bd50812a16 |
|
MD5 | e846ac92a08628c6717bef97fbbb3c49 |
|
BLAKE2b-256 | 420bea0c924c19ef6679e8800dd0025729c4d7ca9c4792adee8b3e1dbcfe16fd |
Hashes for pypdu-0.0.11-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e34127835ece6b0c269adcb4e7b85f12107a8e8bf5397b36fdb409289c0710eb |
|
MD5 | e8ba47542735f08000b408f580fc060e |
|
BLAKE2b-256 | 064659615c6f07e4c7191254dce3c234068ef1c85242ec351fdb3293bb231b6e |
Hashes for pypdu-0.0.11-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3f683b74711b63890e614a1462bad262031bcab0f7170e5b2fd5cd6f96e2bcb9 |
|
MD5 | b6898480bc85d96935bea8272f18c455 |
|
BLAKE2b-256 | 432d6678b3dc6fe1e418bcd05d1eb477f16c78851f1a456ae4f92158f8a21017 |
Hashes for pypdu-0.0.11-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b8d316e2013fb0ec5b4bab83486cdfecfd0659d0e70c44ce26c1ff49f6dcb315 |
|
MD5 | f78602d75426262f70bd20a25783f1ac |
|
BLAKE2b-256 | 855ffa5f557056a08ba7baced1335dacdfcdeac053a622ff529f037fee90bca0 |
Hashes for pypdu-0.0.11-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36f25f99de182900a21db8bc1b153555dc4450177eb783c120112cb66851897f |
|
MD5 | d197a99fd0b8b1bc0a73883910462bc4 |
|
BLAKE2b-256 | 5b228efad8bd9028045376fa5f138d495e676b15e17b27be6eb2877e614e3dcf |
Hashes for pypdu-0.0.11-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 61f64d0a23e6e3c9766369c61ebe3882c5b7a37b7673e440cb5bd6f60325d3ce |
|
MD5 | b0b5fb3065eeffe7de7e8e5e4288ce06 |
|
BLAKE2b-256 | 1f1ad3b6548ec36a73494208c6efb324c4147e0edfece5da614d5834b40e1bd0 |
Hashes for pypdu-0.0.11-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 27555f8321a3a297cd4ef643935f47d3a93ebe9fad61360a3e9c3bf34c502922 |
|
MD5 | ad3d1deaef61cbe02d4d8d7adc4af6ec |
|
BLAKE2b-256 | 0aebbdbde66ca8b3ebc14c36595e0edb99a028566a6fc4fb8460700ae9d599a7 |
Hashes for pypdu-0.0.11-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2ce20fcd2cf5aa616450216ed6d4813a89d9c462fe1d1d3284eca3693fa39847 |
|
MD5 | bfdff3f86f29ca56bcbb492c2d994dea |
|
BLAKE2b-256 | f916bff3d05ae44d22276ec3811c3e5e5208f9e31a416b25d9ad8d61f54f9548 |
Hashes for pypdu-0.0.11-cp36-cp36m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c11ec51cce0246ec7496802abb9352f0eb3aefbe362d26a9db968b76aaa00da4 |
|
MD5 | 68d74e0d988c62d6c19fbe2b358baa57 |
|
BLAKE2b-256 | b01eda37a8d9e19634f77712ccb875b1cfaaf8ee9f1df9b37a95c306d4633835 |
Hashes for pypdu-0.0.11-cp27-cp27mu-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 30726d380ba7911137acb5c9ef87e7b5fa83a233b20ab61150037608217a82e7 |
|
MD5 | f6bc0ecbdce11b679a77e5d8ee4d7888 |
|
BLAKE2b-256 | c34debf06fe38281324c9269dcffa8c4a27fab379757bd48f368fe8ba8e303db |
Hashes for pypdu-0.0.11-cp27-cp27m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84807d3497ebb321814b2440e0720a7932317b8243b4ab5fcfa19d579e52b7a8 |
|
MD5 | cbd4c687b8046bd818a8ddec607e759e |
|
BLAKE2b-256 | 8554e05b4d6d0e06fc3235765538332f88c25b69daa0f26b1bc68a6921f473ce |