A wrapper for the C++ Apache DataSketches library
Reason this release was yanked:
Was a release candidate, not intended (or officially approved) for final release
Project description
# Python Wrapper for Apache DataSketches
This is the official version of the [Apache DataSketches](https://datasketches.apache.org) Python library.
In the analysis of big data there are often problem queries that don’t scale because they require huge compute resources and time to generate exact results. Examples include count distinct, quantiles, most-frequent items, joins, matrix computations, and graph analysis.
If approximate results are acceptable, there is a class of specialized algorithms, called streaming algorithms, or sketches that can produce results orders-of magnitude faster and with mathematically proven error bounds. For interactive queries there may not be other viable alternatives, and in the case of real-time analysis, sketches are the only known solution.
This package provides a variety of sketches as described below. Wherever a specific type of sketch exists in Apache DataSketches packages for other languages, the sketches will be portable between languages (for platforms with the same endianness).
## Building and Installation
Once cloned, the library can be installed by running python -m pip install . in the project root directory, which will also install the necessary dependencies, namely [pybind11](https://github.com/pybind/pybind11) and numpy.
If you prefer to call the setup.py build script directly, you must first install pybind11[global], as well as any other dependencies listed under the build-system section in pyproject.toml.
The library is also available (or soon will be) from PyPI via python -m pip install datasketches.
## Usage
Having installed the library, loading the Apache Datasketches library in Python is simple: import datasketches.
## Available Sketch Classes
KLL (Absolute Error Quantiles) - kll_ints_sketch - kll_floats_sketch
REQ (Relative Error Quantiles) - req_ints_sketch - req_floats_sketch
Frequent Items - frequent_strings_sketch - Error types are frequent_items_error_type.{NO_FALSE_NEGATIVES | NO_FALSE_POSITIVES}
Theta - update_theta_sketch - compact_theta_sketch (cannot be instantiated directly) - theta_union - theta_intersection - theta_a_not_b
HLL - hll_sketch - hll_union - Target HLL types are tgt_hll_type.{HLL_4 | HLL_6 | HLL_8}
CPC - cpc_sketch - cpc_union
VarOpt Sampling - var_opt_sketch - var_opt_union
Vector of KLL - vector_of_kll_ints_sketches - vector_of_kll_floats_sketches
## Known Differences from C++
The Python API largely mirrors the C++ API, with a few minor exceptions: The primary known differences are that Python on modern platforms does not support unsigned integer values or numeric values with fewer than 64 bits. As a result, you may not be able to produce identical sketches from within Python as you can with Java and C++. Loading those sketches after they have been serialized from another language will work as expected.
The Vector of KLL object is currently exclusive to python, and holds an array of independent KLL sketches. This is useful for creating a set of KLL sketches over a vector and has been designed to allow input as either a vector or a matrix of multiple vectors.
We have also removed reliance on a builder class for theta sketches as Python allows named arguments to the constructor, not strictly positional arguments.
## Developer Instructions
The only developer-specific instructions relate to running unit tests.
### Unit tests
The Python unit tests are run with tox. To ensure you have all the needed package, from the package base directory run: ` python -m pip install --upgrade tox tox `
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for datasketches-3.2.0-pp37-pypy37_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1d38096095cd3b9a9c33cec8436fbaef9f65949fff74b8ec9f8a7bce7d1f2a39 |
|
MD5 | 68b3c12b689ee6cc42cb238d1ea8c047 |
|
BLAKE2b-256 | a39ed8140ad2a82ad29cad0e5feace26c056d9abed11740de288ac6ae62a1d0a |
Hashes for datasketches-3.2.0-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 56d2b5e806ce6321190e893b3bfd8bc00ffddff0c2561ffcbd371bb3992ec705 |
|
MD5 | 74ed615326e4cd95b4bd7ae2935df53a |
|
BLAKE2b-256 | 061272543e9ac9b527ff7a480159f57d20f9d02db2cbfe1fe2cef8bda2369d78 |
Hashes for datasketches-3.2.0-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 779c30bf138a810e96a178882ec209aa9b44f9ebe0671b7c0c8cb0832eb8a768 |
|
MD5 | 3ac006e69e2cd3c67f5f59f9683f03da |
|
BLAKE2b-256 | b25003cb83c694736daf6681ea9a3eb84e136e4159ed8ebc4b7c8ef6d24b7b7f |
Hashes for datasketches-3.2.0-pp37-pypy37_pp73-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af3e1f9b5c2b4e8304d9bf8bb4363db404be5002473a3ea8e426c109394b6925 |
|
MD5 | a95adfc30813e84dbd5cc2d1393407a0 |
|
BLAKE2b-256 | 5cdbe59c53640e78f7f2b3dedd0f95bc9a008133badf404155c4a5b92ed66f3c |
Hashes for datasketches-3.2.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f5b20893ba57aae99edaf49bb4b1c7af240fcd19b81c2cd8a06c268a73e0b797 |
|
MD5 | aa5aed3ad3fb2b29d690031a9f65baba |
|
BLAKE2b-256 | fe9652896cbdb8fee33c9fdf58df272cbb8e231ca14ed3019310e513ebbaf4fc |
Hashes for datasketches-3.2.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b0f3f848439dbee610b37c945c03ece65affa229216a685ab73c9fa438417a63 |
|
MD5 | 52cf7400194567e0ed18d3713e4febc3 |
|
BLAKE2b-256 | ea34a8a4c2a9e3b02a45fe6adcb2b173f02bbb0f34d80dece0d656c2cfbedb7a |
Hashes for datasketches-3.2.0-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5ac5a3c7a48cf5a6239e8f71e81ccabe42d7336dd73326b66ff2f290383a4c07 |
|
MD5 | 3887a6497ff23ba5737efc42d7a604d2 |
|
BLAKE2b-256 | 7869127eb8d2e35497af1310fdb2420d4cf205d8ce51ba9d983a5dfd2ecfbdcf |
Hashes for datasketches-3.2.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 27b61d034872a0e1ed73f70f7d3882e1c391fa4660d4685bc395a93343bb2613 |
|
MD5 | a89a9d8f78b839212211a2da5aac872f |
|
BLAKE2b-256 | 2742ba0c7c8438b3b655471de3a588557495e9fb3dfa25bc1dc3c8c8146295f2 |
Hashes for datasketches-3.2.0-cp310-cp310-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 401f311e125149182d6aa9227a7ef8dbcac86d2287a536ab3acafb147442c88a |
|
MD5 | 89ef6c77f51eaaaa29a05ff1d8f3c33e |
|
BLAKE2b-256 | f9040fbb1c07f7660c20baac362f31e5427d8769fb8873e40a4ad55e91d44986 |
Hashes for datasketches-3.2.0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4b4300e991bef43dcb5b3e24d0fcd128c7a2c266e9b4ec04242edff8b3698c22 |
|
MD5 | a5b0eb072f6b522a8529a5f5b3eaa9fb |
|
BLAKE2b-256 | d0066d4a2af8d083a9d03e5f75415ae2e87c5ed8ebee3aa89e11f997b321b162 |
Hashes for datasketches-3.2.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa49cef658c9d440dec7a054ec4892088d71ee3d1566db0fd1e80d243c2e9e3c |
|
MD5 | c5d875c33511aea941c132305fe5dc37 |
|
BLAKE2b-256 | e60ab861ac16c726ec71293fc65d5cac0c45d2c531de81acbec65009e08cd867 |
Hashes for datasketches-3.2.0-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f5953ae6b7028cad0663f0109f74cdf70c5f162d004932629e85d16ba67b1c2b |
|
MD5 | 7da113e259717aedd73beddbd3eb73aa |
|
BLAKE2b-256 | 54d973256f0694dffd96000deafd0f03856ef3ddbd4934387b31d5cdb5c381fa |
Hashes for datasketches-3.2.0-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | de43c9958609b456112f72c7c498f87c0a00c7e2a28799fb986ea70202ed34f9 |
|
MD5 | 624b773a80ef8c9e7be52d2fcdeb38eb |
|
BLAKE2b-256 | 9cad8ef4738bd65cc9bd7d763605cc1803f084a50e2033fdfd9568e5f5eaeb53 |
Hashes for datasketches-3.2.0-cp39-cp39-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 837a80d0112e77a2b90c9453a4b74006f5d0a5bdf93205f60d014b6197501b0e |
|
MD5 | 9822a1a906cd9d079adaf9e6fe7d89d8 |
|
BLAKE2b-256 | 96de11fa0345ae17e817bffd1592c0c782137f335ebdef63ee0c5f46c4c8eb5a |
Hashes for datasketches-3.2.0-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d3c82ff9662f46930af271529bbe3a39eb6a889da6cf7f17d61998043d8a7732 |
|
MD5 | c4d92d3546d41f480b5cfc6e0a052c1f |
|
BLAKE2b-256 | 9dd0ee39a316b4efa148356a9cf149cd6585759ede3b91334232612e6d0191ed |
Hashes for datasketches-3.2.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7eaf0655a73f713ccdb8f9974451b1d21d5f0e9b8d3a17390d228aed6502cc15 |
|
MD5 | 124ac2f53027505468c2c55bc18f2c38 |
|
BLAKE2b-256 | 177cc9a43c7be5e01950cdfbc4711023fb87015a68129b734ffca1a7453521ca |
Hashes for datasketches-3.2.0-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2c6b9b2a6b847206aa7cf5192254ef49931f5eb125fa7474fb7d6d473d57701c |
|
MD5 | eae7105632d63826c46339edc307eb0f |
|
BLAKE2b-256 | 540530fcc10b5b9d819e4f6d1af8d23c478e9eb3a4b0bbe60bac18f87718a3f1 |
Hashes for datasketches-3.2.0-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e42e6e5afb34747ce99603d58f9a84b085a746d1352fa71ce0becbe761b5718d |
|
MD5 | 3cfd491d31f648d3219d21c514e04c74 |
|
BLAKE2b-256 | 7e90ebc062a620c05425c27800428f496d9d636a8ea42c854472280595a298ca |
Hashes for datasketches-3.2.0-cp38-cp38-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 65ade4b187f760fd323c0293719cb39baa4d3260e39087cbcaac1332fee2eacf |
|
MD5 | 43454e748a82f897d59335eb745ffa64 |
|
BLAKE2b-256 | f872d736a0a150318dfe05c3810d71dde0c9a70a294da38be40cbb149aa054eb |
Hashes for datasketches-3.2.0-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 26ac656a965b20c0fdef9f4d7fcc8d85f60ab15ac7270069ee3adab59ac9fc29 |
|
MD5 | 2140d4a1cd8ab33b9f7eafd66160a6d5 |
|
BLAKE2b-256 | f5f2d44af0289847e6b3da2cc015088cea7ab9f767e935688324a4d6876e6806 |
Hashes for datasketches-3.2.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 64509bbafc0be6b53ec0fde14f48e295a37c1af419ff9112c6706b681399ea82 |
|
MD5 | 1521f5277e4a11558ef95066cc4455e8 |
|
BLAKE2b-256 | e3275ef3e0da4bd4c96d11253c301b7f685e8319457c71d8a56a9db5ee5eee04 |
Hashes for datasketches-3.2.0-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a0ea88dfb79d02af3e5d0e3bb54cf19d6f99d72fee4dd47aeb75a31385bba5c7 |
|
MD5 | 970aa5195f8dda345fbd75001d01a9b4 |
|
BLAKE2b-256 | 964e8f3e2f01013080b56f9df5a42fa12967f65924c2432004f3f843f71a1d00 |
Hashes for datasketches-3.2.0-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f3c69d7fc37e92cc2c6e762e38f187186865852c852a50073260ed74928776b |
|
MD5 | 68e2c6c155874a02878815210dcae50f |
|
BLAKE2b-256 | 85f57946c428fb62e8ad530778f58bd0688f72ec48fc7a6eab2fa88f13f4c80d |
Hashes for datasketches-3.2.0-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 443a8b090db29b2d35718b4f24eff1b6fbc43968cd665dcc2b38df808d6833d5 |
|
MD5 | d21e6bc2b26374a61948f1051ecef841 |
|
BLAKE2b-256 | dfb228861932e9a4f1d0ccb6e06786a1714855689dc70d23bad6c9451de33b93 |
Hashes for datasketches-3.2.0-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e5b234f07f42f23fab401930c0e8c6bbce293c51adf7eba64789fa2e39a9d081 |
|
MD5 | 8ea8d40ba0f0b4237d4abc5af9048d40 |
|
BLAKE2b-256 | 506dd8def6640f8b92f5eb1dd80a944ee18eb750f1bebab83227c1121138f756 |
Hashes for datasketches-3.2.0-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2852f1762e896041829bcb734da17c9202006689f12152854c25fbb1b7a44686 |
|
MD5 | 9a571a58794e762fc9ae33f48fea9019 |
|
BLAKE2b-256 | a3d2db4247e95618aa6dfb7fdf9a7d49623208f0f4af8a5891bce77f81929da0 |
Hashes for datasketches-3.2.0-cp36-cp36m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12c3a0f35b3f59822afb26eabd7fc545c6fab293a2f351512fff71a1edb9e5ef |
|
MD5 | 6efd047dd721c18418c8c6ba425bce4a |
|
BLAKE2b-256 | 51e6f97b725de837af8fcf3f57aea47ec1582b9bf29a35038dc8829dff302208 |