Effulge 0.0.1

A small package used to find data variances

Effulge

Use Case

• When we have two pyspark dataframes with valid Primary Key
• and we need to find the attributes that are mismatching between the two dataframes.

---

Example

Lets consider two dataframes "expectation" and "reality".

Inputs

If Primary Key is (ProductID, Colour) and if the contents of "expectation" are -

ProductID	ProductName	Colour	UnitPrice	Quantity	Fragile	Gift
1001	GelPen	Blue	10	2	0	0
1001	GelPen	Black	10	1	0	0
1001	GelPen	Red	10	1	0	0
1002	InkPen	Blue	50	1	0	1
1003	InkBottle	Blue	35	1	1	1
1004	Pencil	Grey	3	5	0	0
1005	Eraser	White	2	2	0	0
1006	Sharpner	Orange	3	1	0	0
1006	Sharpner	Steel	5	1	0	0
1007	GeometryBox	Green	40	0	0

And if contents of "reality" are -

ProductID	ProductName	Colour	UnitPrice	Quantity	Fragile	Gift
1001	GelPen	Blue	10	2	0	0
1001	GelPen	Black	10	7	0	0
1001	GelPen	Red	10	1	0	0
1002	InkPen	Blue	50	1	0	1
1003	InkBottle	Blue	3	1	1	0
1003	WaterBottle	Blue	20	2	0	0
1004	Pencil	Grey	3	5	0	0
1005	Eraser	Whiteee	2	2	0	0
1006	Sharpner	Orange	3	1	0	0
1006	Sharpner	Steel	5	1	0	0
1007	GeometryBox	Green	40	0	1

Output

Then, Effulge will produce an output dataframe with following contents -

productid	colour	EFFULGE_VARIANCE_PROVOKER
1007	Green	[fragile]
1001	Black	[quantity]
1003	Blue	[fragile, gift, productname, quantity, unitprice]
1003	Blue	[gift, unitprice]
1005	White	[MISSING_PRIMARY_KEY]
1003	Blue	[DUPLICATE_PRIMARY_KEY]

---

Usage:

from effulge import spot_variance

# Initialize SparkSession
# Load data into dataframes, let's say they are called "df_expectation" and "df_reality"
# Declare a tuple with valid primary key, let's say it is called "primary_key"

output = spot_variance(df_expectation, df_reality, primary_key)
output.show()


# to generate variance report
from effulge import save_variance_report
save_variance_report(
    variance_df=output, source_df=df_expectation, target_df=df_reality,
    super_key=primary_key, file_name="effulge_variance_report",
    src_prefix='SRC', tgt_prefix='TGT'
)

---