spark-gaps-date-rorc-tools 0.0.7

spark_gaps_date_rorc_tools

spark_gaps_date_rorc_tools

spark_gaps_date_rorc_tools is a Python library that implements get gaps dates

Installation

The code is packaged for PyPI, so that the installation consists in running:

pip install spark-gaps-date-rorc-tools 

Usage

wrapper take gaps dates

config.yaml
===========
  conf-rorc:
    t_psan_test:
      table_path: "/data/master/psan/data/t_psan_test/"
      supplies : [
          "/data/master/psan/data/t_ksag_test/",
          "/data/master/psan/data/t_psan_test/"
      ]
    t_kctk_cust_rating_atrb:
      table_path: ""
       supplies : []



example1: file.py
=================
from spark_gaps_date_rorc_tools import show_gaps_date
df_detail, df_pivot = show_gaps_date(spark=spark,
                                     config_path_name="config.yaml",
                                     hdfs_uri="hdfs://pedaaslive.scmx2p100.isi",
                                     filter_date_initial="202101",
                                     filter_date_final="202112")
df_pivot.head()



example2: file.py
=================
from spark_gaps_date_rorc_tools import show_gaps_date
df_detail, df_pivot = show_gaps_date(spark=spark,
                                     config_path_name="config.yaml",
                                     table_rorc=["t_psan_xxx"]
                                     hdfs_uri="hdfs://pedaaslive.scmx2p100.isi",
                                     filter_date_initial="202101",
                                     filter_date_final="202112")

df_pivot.head()

License

Apache License 2.0.

New features v1.0

BugFix

• choco install visualcpp-build-tools

Reference

• Jonathan Quiza github.
• Jonathan Quiza RumiMLSpark.
• Jonathan Quiza linkedin.