Skip to main content

Sparkhistogram contains helper functions for generating data histograms with the Spark DataFrame API.

Project description

Use this package, sparkhistogram, together with PySpark for generating data histograms using the Spark DataFrame API. Currently, the package contains only two functions covering some of the most common and low-complexity use cases.

Use:

  • from sparkhistogram import computeHistogram -> computeHistogram is a function to compute the count/frequency histogram of a given DataFrame column
  • from sparkhistogram import computeWeightedHistogram -> computeWeightedHistogram is a function to compute the weighted histogram of a given DataFrame column
def computeHistogram(df: "DataFrame", value_col: str, min_val: float, max_val: float, bins: int) -> "DataFrame"

Parameters
----------
df: the dataframe with the data to compute
value_col: column name on which to compute the histogram
min_val: minimum value in the histogram
max_val: maximum value in the histogram
bins: number of histogram buckets to compute

Output DataFrame
----------------
bucket: the bucket number, range from 1 to bins (included)
value: midpoint value of the given bucket
count: number of values in the bucket

Run this example in the PySpark shell

Note: requires PySpark version 3.1 or higher.

bin/pyspark

# import the helper function to generate the histogram using Spark DataFrame operations
from sparkhistogram import computeHistogram

# generate some toy data
num_events = 100
scale = 100
seed = 4242
df = spark.sql(f"select random({seed}) * {scale} as random_value from range({num_events})")

# define the DataFrame transformation to compute the histogram 
histogram = computeHistogram(df, "random_value", -20, 90, 11) 

# with Spark 3.3.0 and higher you can also use df.transform
# histogram = df.transform(computeHistogram, "random_value", -20, 90, 11)

# fetch and display the (toy) data
histogram.show()

# expected output:
+------+-----+-----+
|bucket|value|count|
+------+-----+-----+
|     1|-15.0|    0|
|     2| -5.0|    0|
|     3|  5.0|    6|
|     4| 15.0|   10|
|     5| 25.0|   15|
|     6| 35.0|   12|
|     7| 45.0|    9|
|     8| 55.0|    7|
|     9| 65.0|   10|
|    10| 75.0|   16|
|    11| 85.0|    7|
+------+-----+-----+

More details and notebooks with matplotlib visualization of the histograms at:
https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_DataFrame_Histograms.md

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sparkhistogram-0.4.tar.gz (3.2 kB view details)

Uploaded Source

Built Distribution

sparkhistogram-0.4-py2.py3-none-any.whl (3.7 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file sparkhistogram-0.4.tar.gz.

File metadata

  • Download URL: sparkhistogram-0.4.tar.gz
  • Upload date:
  • Size: 3.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.11.7

File hashes

Hashes for sparkhistogram-0.4.tar.gz
Algorithm Hash digest
SHA256 022338b978f0cea6bde2933c24122328fcc8214c13422fe919d0b467e7e75fe1
MD5 014dfd1be747734efa9d957f8574aa26
BLAKE2b-256 790f30a59ff6c9d2175ada826b6ad3c095bedc394335c3c839aa858d54bdd711

See more details on using hashes here.

File details

Details for the file sparkhistogram-0.4-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for sparkhistogram-0.4-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 aad590cba577a0fa83781c14446e245e929a4b6578ffac12f347750684615202
MD5 073a4251b914000b880fb7c0e942dd83
BLAKE2b-256 b6a1e8032f77bd96c72c63ecdd7666a10bbb440ba932b9c6be7d18984081a2ee

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page