This project utilizes faker-pyspark to generate random schema and dataframes to mimic data table snapshots. Using these snapshots to process and apply SCD2 pattern into delta table as the destination.
Project description
Demo PySpark Delta Table SCD2 implementation
This project utilizes faker-pyspark
to generate random schema and dataframes to mimic data table snapshots.
Using these snapshots to process and apply SCD2 pattern into delta table as the destination.
Installation
Install with pip:
pip install pyspark-delta-scd2
Generate incremental updates to dataframe and apply scd2
>>> from pyspark_delta_scd2 import get_spark, PySparkDeltaScd2
>>> spark = get_spark()
>>> demo = PySparkDeltaScd2(spark=spark)
>>> # initial load
>>> df1 = demo.process()
>>> # incremental update
>>> df2 = demo.process()
>>> # df2 should have some deletes, updates and inserts
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for pyspark_delta_scd2-0.3.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fedbd5f8f68de2945f0b558009ca20932088392c015c12903f916340035a7784 |
|
MD5 | 59533ce154026a055a1cd73af3e7247c |
|
BLAKE2b-256 | 4e1dcf1409dfd2616c3d1f2c0f218751e62f8b6e527485adfec518b372314c28 |