spark_gaps_date_rorc_tools
Project description
spark_gaps_date_rorc_tools
spark_gaps_date_rorc_tools is a Python library that implements get gaps dates
Installation
The code is packaged for PyPI, so that the installation consists in running:
pip install spark-gaps-date-rorc-tools
Usage
wrapper take gaps dates
config.yaml
===========
conf-rorc:
t_psan_test:
table_path: "/data/master/psan/data/t_psan_test/"
supplies : [
"/data/master/psan/data/t_ksag_test/",
"/data/master/psan/data/t_psan_test/"
]
t_kctk_cust_rating_atrb:
table_path: ""
supplies : []
example1: file.py
=================
from spark_gaps_date_rorc_tools import show_gaps_date
df_pivot = show_gaps_date(spark=spark,
config_path_name="config.yaml",
hdfs_uri="hdfs://pedaaslive.scmx2p100.isi",
filter_date_initial="202101",
filter_date_final="202112")
df_pivot.head()
example2: file.py
=================
from spark_gaps_date_rorc_tools import show_gaps_date
df_pivot = show_gaps_date(spark=spark,
config_path_name="config.yaml",
table_rorc=["t_psan_xxx"]
hdfs_uri="hdfs://pedaaslive.scmx2p100.isi",
filter_date_initial="202101",
filter_date_final="202112")
df_pivot.head()
Style Dataframe (Spark): file.py
=========================
import pyspark
from spark_gaps_date_rorc_tools import show_spark_df
pyspark.sql.dataframe.DataFrame.show2 = show_spark_df
df_pivot = show_gaps_date(spark=spark,
config_path_name="config.yaml",
table_rorc=["t_psan_xxx"]
hdfs_uri="hdfs://pedaaslive.scmx2p100.isi",
filter_date_initial="202101",
filter_date_final="202112")
df_pivot.show2()
Style Dataframe (Pandas): file.py
=========================
import pandas as pd
from spark_gaps_date_rorc_tools import show_pd_df
pd.DataFrame.show2 = show_pd_df
df_pivot = show_gaps_date(spark=spark,
config_path_name="config.yaml",
table_rorc=["t_psan_xxx"]
hdfs_uri="hdfs://pedaaslive.scmx2p100.isi",
filter_date_initial="202101",
filter_date_final="202112")
df_pivot2 = df_pivot.toPandas()
df_pivot2.show2()
License
New features v1.0
BugFix
- choco install visualcpp-build-tools
Reference
- Jonathan Quiza github.
- Jonathan Quiza RumiMLSpark.
- Jonathan Quiza linkedin.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for spark_gaps_date_rorc_tools-0.0.11.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6fd04657c285497a8fbafdd5cbbfb642a4a6e3a2c56c046ac2eff8c8786153d4 |
|
MD5 | 62821af0aac4d2a0becff5d6199f648a |
|
BLAKE2b-256 | bad6671db6a99f3600f5007391ad449cd6097ad1480176c80b2b15a8b957b3ef |
Close
Hashes for spark_gaps_date_rorc_tools-0.0.11-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d3249676cc69c9c184f1f443fbec5d2b92c0371befdb6c703889d9aa4775b29a |
|
MD5 | 9dd1a32ea516cb671c93dc29aae22488 |
|
BLAKE2b-256 | 5f2b3f18a59c3232265a18c9579cb19f8e865ec828780ac30f63171a77b5f7c8 |