Skip to main content

IOMETE's PySpark library that contains useful utilities for working with PySpark

Project description

Pyspark IOMETE Library

This library is providing a set of utility functions to speed up the development of pyspark applications.

Installation

pip install pyspark-iomete

Utility functions

get_spark_logger

This function is returning a spark logger instance.

As you may know, spark is using log4j as a logging framework. This function is returning a spark logger instance that is using the log4j logger. Standard python logging is not working with pyspark. The following function get the spark logger instance and returns it.

Usage:

from pyspark_iomete.utils import get_spark_logger
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# spark session and name will be used to create the logger
# both are optional
logger = get_spark_logger(spark=spark, name="my_custom_logger")

# spark session will be retrieved using SparkSession.getActiveSession() and name will be set to the current file name
logger = get_spark_logger()

Test utility functions

table_name_with_random_suffix

This function is returning a table name with a random suffix. This is useful for testing purposes.

Usage:

from pyspark_iomete.test_utils import table_name_with_random_suffix

table_name = table_name_with_random_suffix("my_table")

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyspark_iomete-0.0.3.tar.gz (2.5 kB view details)

Uploaded Source

Built Distribution

pyspark_iomete-0.0.3-py3-none-any.whl (2.8 kB view details)

Uploaded Python 3

File details

Details for the file pyspark_iomete-0.0.3.tar.gz.

File metadata

  • Download URL: pyspark_iomete-0.0.3.tar.gz
  • Upload date:
  • Size: 2.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.8.9

File hashes

Hashes for pyspark_iomete-0.0.3.tar.gz
Algorithm Hash digest
SHA256 ab58af5d41bea82a97219c0ff8578dd6c84581fd4a8a97a4eff2ab2129a57b00
MD5 0346cc2626193b07d9cfe5a3f30eeead
BLAKE2b-256 b8251857521401a105e744100e7417fb79f1d6592b63f7cdfec9c8e06e50f0c5

See more details on using hashes here.

File details

Details for the file pyspark_iomete-0.0.3-py3-none-any.whl.

File metadata

File hashes

Hashes for pyspark_iomete-0.0.3-py3-none-any.whl
Algorithm Hash digest
SHA256 410c11953bab712ce696d8f78df80cfedcee048277a4a1769b13c196526d7e14
MD5 65ec9ffede954826088fa1e7f8cb8568
BLAKE2b-256 e00f532da655af96a602449754e6778fc9b72055ca6365894250233696e0a561

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page