IOMETE's PySpark library that contains useful utilities for working with PySpark
Project description
Pyspark IOMETE Library
This library is providing a set of utility functions to speed up the development of pyspark applications.
Installation
pip install pyspark-iomete
Utility functions
get_spark_logger
This function is returning a spark logger instance.
As you may know, spark is using log4j as a logging framework. This function is returning a spark logger instance that is using the log4j logger. Standard python logging is not working with pyspark. The following function get the spark logger instance and returns it.
Usage:
from pyspark_iomete.utils import get_spark_logger
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
# spark session and name will be used to create the logger
# both are optional
logger = get_spark_logger(spark=spark, name="my_custom_logger")
# spark session will be retrieved using SparkSession.getActiveSession() and name will be set to the current file name
logger = get_spark_logger()
Test utility functions
table_name_with_random_suffix
This function is returning a table name with a random suffix. This is useful for testing purposes.
Usage:
from pyspark_iomete.test_utils import table_name_with_random_suffix
table_name = table_name_with_random_suffix("my_table")
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file pyspark_iomete-0.0.3.tar.gz
.
File metadata
- Download URL: pyspark_iomete-0.0.3.tar.gz
- Upload date:
- Size: 2.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab58af5d41bea82a97219c0ff8578dd6c84581fd4a8a97a4eff2ab2129a57b00 |
|
MD5 | 0346cc2626193b07d9cfe5a3f30eeead |
|
BLAKE2b-256 | b8251857521401a105e744100e7417fb79f1d6592b63f7cdfec9c8e06e50f0c5 |
File details
Details for the file pyspark_iomete-0.0.3-py3-none-any.whl
.
File metadata
- Download URL: pyspark_iomete-0.0.3-py3-none-any.whl
- Upload date:
- Size: 2.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 410c11953bab712ce696d8f78df80cfedcee048277a4a1769b13c196526d7e14 |
|
MD5 | 65ec9ffede954826088fa1e7f8cb8568 |
|
BLAKE2b-256 | e00f532da655af96a602449754e6778fc9b72055ca6365894250233696e0a561 |