Sparkhistogram contains helper functions for generating data histograms with the Spark DataFrame API.
Project description
Use this package, sparkhistogram, together with PySpark for generating data histograms using the Spark DataFrame API. Currently, the package contains only two functions covering some of the most common and low-complexity use cases.
Use:
from sparkhistogram import computeHistogram
-> computeHistogram is a function to compute the count/frequency histogram of a given DataFrame columnfrom sparkhistogram import computeWeightedHistogram
-> computeWeightedHistogram is a function to compute the weighted histogram of a given DataFrame column
def computeHistogram(df: "DataFrame", value_col: str, min_val: float, max_val: float, bins: int) -> "DataFrame"
Parameters
----------
df: the dataframe with the data to compute
value_col: column name on which to compute the histogram
min_val: minimum value in the histogram
max_val: maximum value in the histogram
bins: number of histogram buckets to compute
Output DataFrame
----------------
bucket: the bucket number, range from 1 to bins (included)
value: midpoint value of the given bucket
count: number of values in the bucket
Run this example in the PySpark shell
Note: requires PySpark version 3.1 or higher.
bin/pyspark
# import the helper function to generate the histogram using Spark DataFrame operations
from sparkhistogram import computeHistogram
# generate some toy data
num_events = 100
scale = 100
seed = 4242
df = spark.sql(f"select random({seed}) * {scale} as random_value from range({num_events})")
# define the DataFrame transformation to compute the histogram
histogram = computeHistogram(df, "random_value", -20, 90, 11)
# with Spark 3.3.0 and higher you can also use df.transform
# histogram = df.transform(computeHistogram, "random_value", -20, 90, 11)
# fetch and display the (toy) data
histogram.show()
# expected output:
+------+-----+-----+
|bucket|value|count|
+------+-----+-----+
| 1|-15.0| 0|
| 2| -5.0| 0|
| 3| 5.0| 6|
| 4| 15.0| 10|
| 5| 25.0| 15|
| 6| 35.0| 12|
| 7| 45.0| 9|
| 8| 55.0| 7|
| 9| 65.0| 10|
| 10| 75.0| 16|
| 11| 85.0| 7|
+------+-----+-----+
More details and notebooks with matplotlib visualization of the histograms at:
https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_DataFrame_Histograms.md
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file sparkhistogram-0.4.tar.gz
.
File metadata
- Download URL: sparkhistogram-0.4.tar.gz
- Upload date:
- Size: 3.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.11.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 022338b978f0cea6bde2933c24122328fcc8214c13422fe919d0b467e7e75fe1 |
|
MD5 | 014dfd1be747734efa9d957f8574aa26 |
|
BLAKE2b-256 | 790f30a59ff6c9d2175ada826b6ad3c095bedc394335c3c839aa858d54bdd711 |
File details
Details for the file sparkhistogram-0.4-py2.py3-none-any.whl
.
File metadata
- Download URL: sparkhistogram-0.4-py2.py3-none-any.whl
- Upload date:
- Size: 3.7 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.11.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | aad590cba577a0fa83781c14446e245e929a4b6578ffac12f347750684615202 |
|
MD5 | 073a4251b914000b880fb7c0e942dd83 |
|
BLAKE2b-256 | b6a1e8032f77bd96c72c63ecdd7666a10bbb440ba932b9c6be7d18984081a2ee |