pyspark-json-model 0.0.3

JSON to Relational data model through Pyspark using Databricks

Easily Parse your JSON & convert to relational data model format. This is currenlty build to support on Databricks/Pyspark

Description

This package if for converting any json formatted data to relational data model in dataframe format for databricks

• jsontools: tools to parse json & converts to relational data model format

Installation

Normal installation

pip install pyspark_json_model

Development installation

git clone https://github.com/rohitpawar95/pyspark_json_model.git
cd pyspark_json_model
pip install --editable .

Usage

from pyspark_json_model import *

json2model_obj = jsontools.JSONParserPyspark('filepath_of_json', 
			<identifier_for_json_document:str>, 
		 	<is_looking_for_normalized_dataframe:bool>)

normalized_json_model = json2model_obj.process(sc, spark, ['drop_columns',...])

display(normalized_json_model[key])

Note: 
	is_looking_for_normalized_dataframe --> if True : The returned normalized_json_model 
	will hold extra key for root_node+'_normalized' which will have normalized json 
	format (This will be same as json_normalize function of pandas)

	is_looking_for_normalized_dataframe --> if False : The returned normalized_json_model 
	will just contain multiple keys which will hold independent tables of data model 
		
	normalized_json_model : This will be dictionary of data model key will be table name 
	and value will be dataframe corresponding to this key
	
	Pass sparkContext (sc) & sparkSession (spark) to process JSON along with 
	['drop_columns'] : The parameter to process() function is optional if need to remove 
	any columns while making this dataframe specify column list here

Change log

[0.0.3] - 2021-10-17

Added

Changed

Updated process function to handle UTF-8 BOM json format

Removed

[0.0.2] - 2021-10-17

Added

Changed

Updated process function to handle Array & Dict data types

Removed

[0.0.1] - 2021-10-14

Added

• JSONParserPyspark.array_identification()
• JSONParserPyspark.check_all_done()
• JSONParserPyspark.get_nested_field()
• JSONParserPyspark.explode_struct_field()
• JSONParserPyspark.explode_array_df()
• JSONParserPyspark.add_random_id()
• JSONParserPyspark.main()
• JSONParserPyspark.drop_columns()
• JSONParserPyspark.drop_all_duplicates()
• JSONParserPyspark.remove_df()
• JSONParserPyspark.check_relationship()
• JSONParserPyspark.join_multiple_df()
• JSONParserPyspark.normalized_json()
• JSONParserPyspark.process()

Changed

Removed