This project is meant to help pyspark user (Databricks) to get some utility functions.
Project description
pyspark-utility
Provide utility functions for pyspark
Example - How to use?
from pyspark_utility.size_calculator import SizeCalculator
calc = SizeCalculator(spark)
calc.get_size_for_human(numbers)
Listing here are the modules that can be used highlighting some useful functions also (version - 0.4.0)
| Module | Function | Description |
|---|---|---|
| SizeCalculator | get_size_for_machine(obj) |
get size in bytes |
| SizeCalculator | get_size_for_human(obj) |
get size in KB, MB, GB, and so on |
| FakeData | fake_user(), fake_users(spark), and so on. |
static functions to return fake data |
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pyspark_utility-0.4.0.tar.gz
(3.1 kB
view details)
Built Distribution
File details
Details for the file pyspark_utility-0.4.0.tar.gz.
File metadata
- Download URL: pyspark_utility-0.4.0.tar.gz
- Upload date:
- Size: 3.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.9.18
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 72e1f39eb0a424989c1368ff954f83bc00a891c1278def2480321d4cca7e1719 |
|
| MD5 | 4051eab6ae619f8eeecec76174463116 |
|
| BLAKE2b-256 | 1643fcb7741c40305c9840dc7eac81e7248e6c910e3817b09ff9311f1ad13090 |
File details
Details for the file pyspark_utility-0.4.0-py2.py3-none-any.whl.
File metadata
- Download URL: pyspark_utility-0.4.0-py2.py3-none-any.whl
- Upload date:
- Size: 3.8 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.9.18
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f381fab14f151ea7f8cc7508670a8345ba3149d6472f9d54b0a60c44919adf46 |
|
| MD5 | 88a7632e74025e598def341c1fe48027 |
|
| BLAKE2b-256 | 3ec70599337fea7cc3d73744bde1e3f30a429bcfd270e208c9bab053fbafeec2 |
