spark-gaps-date-rorc-tools 0.0.9

spark_gaps_date_rorc_tools

spark_gaps_date_rorc_tools

spark_gaps_date_rorc_tools is a Python library that implements get gaps dates

Installation

The code is packaged for PyPI, so that the installation consists in running:

pip install spark-gaps-date-rorc-tools 

Usage

wrapper take gaps dates

config.yaml
===========
  conf-rorc:
    t_psan_test:
      table_path: "/data/master/psan/data/t_psan_test/"
      supplies : [
          "/data/master/psan/data/t_ksag_test/",
          "/data/master/psan/data/t_psan_test/"
      ]
    t_kctk_cust_rating_atrb:
      table_path: ""
       supplies : []



example1: file.py
=================
from spark_gaps_date_rorc_tools import show_gaps_date
df_pivot = show_gaps_date(spark=spark,
                          config_path_name="config.yaml",
                          hdfs_uri="hdfs://pedaaslive.scmx2p100.isi",
                          filter_date_initial="202101",
                          filter_date_final="202112")
df_pivot.head()



example2: file.py
=================
from spark_gaps_date_rorc_tools import show_gaps_date
df_pivot = show_gaps_date(spark=spark,
                          config_path_name="config.yaml",
                          table_rorc=["t_psan_xxx"]
                          hdfs_uri="hdfs://pedaaslive.scmx2p100.isi",
                          filter_date_initial="202101",
                          filter_date_final="202112")

df_pivot.head()


Style Dataframe (Spark): file.py
=========================
import pyspark
from spark_gaps_date_rorc_tools import show_spark_df
pyspark.sql.dataframe.DataFrame.show2 = show_spark_df

df_pivot = show_gaps_date(spark=spark,
                          config_path_name="config.yaml",
                          table_rorc=["t_psan_xxx"]
                          hdfs_uri="hdfs://pedaaslive.scmx2p100.isi",
                          filter_date_initial="202101",
                          filter_date_final="202112")

df_pivot.show2()



Style Dataframe (Pandas): file.py
=========================
import pandas as pd
from spark_gaps_date_rorc_tools import show_pd_df
pd.DataFrame.show2 = show_pd_df

df_pivot = show_gaps_date(spark=spark,
                          config_path_name="config.yaml",
                          table_rorc=["t_psan_xxx"]
                          hdfs_uri="hdfs://pedaaslive.scmx2p100.isi",
                          filter_date_initial="202101",
                          filter_date_final="202112")

df_pivot2 = df_pivot.toPandas()                      
df_pivot2.show2()

License

Apache License 2.0.

New features v1.0

BugFix

• choco install visualcpp-build-tools

Reference

• Jonathan Quiza github.
• Jonathan Quiza RumiMLSpark.
• Jonathan Quiza linkedin.