Skip to main content

Sparkhistogram contains helper functions for generating data histograms with the Spark DataFrame API.

Project description

Use this package, sparkhistogram, together with PySpark for generating data histograms using the Spark DataFrame API. Currently, the package contains only two functions covering some of the most common and low-complexity use cases.

Use:

  • from sparkhistogram import computeHistogram -> computeHistogram is a function to compute the count/frequency histogram of a given DataFrame column
  • from sparkhistogram import computeWeightedHistogram -> computeWeightedHistogram is a function to compute the weighted histogram of a given DataFrame column
def computeHistogram(df: "DataFrame", value_col: str, min: int, max: int, bins: int) -> "DataFrame"

Parameters
----------
df: the dataframe with the data to compute
value_col: column name on which to compute the histogram
min: minimum value in the histogram
max: maximum value in the histogram
bins: number of histogram buckets to compute

Output DataFrame
----------------
bucket: the bucket number, range from 1 to bins (included)
value: midpoint value of the given bucket
count: number of values in the bucket

Run this example in the PySpark shell

Note: requires PySpark version 3.1 or higher.

bin/pyspark

# import the helper function to generate the histogram using Spark DataFrame operations
from sparkhistogram import computeHistogram

# generate some toy data
num_events = 100
scale = 100
seed = 4242
df = spark.sql(f"select random({seed}) * {scale} as random_value from range({num_events})")

# define the DataFrame transformation to compute the histogram 
histogram = computeHistogram(df, "random_value", -20, 90, 11) 

# with Spark 3.3.0 and higher you can also use df.transform
# histogram = df.transform(computeHistogram, "random_value", -20, 90, 11)

# fetch and display the (toy) data
histogram.show()

# expected output:
+------+-----+-----+
|bucket|value|count|
+------+-----+-----+
|     1|-15.0|    0|
|     2| -5.0|    0|
|     3|  5.0|    6|
|     4| 15.0|   10|
|     5| 25.0|   15|
|     6| 35.0|   12|
|     7| 45.0|    9|
|     8| 55.0|    7|
|     9| 65.0|   10|
|    10| 75.0|   16|
|    11| 85.0|    7|
+------+-----+-----+

More details and notebooks with matplotlib visualization of the histograms at:
https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_DataFrame_Histograms.md

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sparkhistogram-0.1.tar.gz (3.1 kB view hashes)

Uploaded Source

Built Distribution

sparkhistogram-0.1-py2.py3-none-any.whl (3.5 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page