sparktool for hccn
Project description
sparktool
built to simplified the process of creating sparksession(hccn).
What's New
- parser impala/hive view automatically
- use hive cmd to find real kudu.table_name
- batch return
features
- change kudu table to spark view automatically
- split and execute sql scripts
- can parser impala/hive view automatically
install
Python 2/3
pip install sparktool
functions
reset keytab
[in]
# encoding=utf-8
import sparktool as st
st.switch_keytab('admin@EXAMPLE.COM', keytabpath)
execute sql
[in]
# encoding=utf-8
import sparktool as st
ss = st.SparkCreator(appname='ryan_ttt', param={"spark.task.maxFailures":"10"})
sql = '''
select 1;
select
cc.skp_client
from
owner_ogg.ft_ccase_2_ccase_ad cc
join owner_ogg.clt_ccase_2_ccase_relation re
on cc.skp_ccase_2_ccase_relation = re.skp_ccase_2_ccase_relation
and re.code_ccase_relation = 'FIRST_POS'
limit 1
;
'''
ss.batch_excutesql(sql)
[out]
Tranform Table:
+--------------------------------------+--------------------------------------+--------------+
| Origin Table | Temporary View | If Transform |
+--------------------------------------+--------------------------------------+--------------+
| OWNER_OGG.FT_CCASE_2_CCASE_AD | owner_ogg_ft_ccase_2_ccase_ad | New |
| OWNER_OGG.CLT_CCASE_2_CCASE_RELATION | owner_ogg_clt_ccase_2_ccase_relation | New |
+--------------------------------------+--------------------------------------+--------------+
Excute Progress: 2/2
DataFrame[skp_client: decimal(38,0)]
[in]
# encoding=utf-8
import sparktool as st
ss = st.SparkCreator()
sql = '''
select * from winnie_wangtjww.hcp_crm_offer limit 1;
drop table if exists ttttttttt;
select 1 as tt;
select * from ap_crm.ft_sas_segment_client limit 1;
select * from owner_ogg.ft_mobile_app_user_ad limit 1;
'''
b,c,d,e,f = ss.batch_excutesql(sql, ifview=True, ifbatchre=True)
[out]
Tranform Table:
+-----------------------------------------+-----------------------------------------------+--------------+
| Origin Table | Kudu Table | If Transform |
+-----------------------------------------+-----------------------------------------------+--------------+
| hadoop_dl.dct_sas_campaign | impala::AP_ITBD.dct_sas_campaign | Added |
| hadoop_dl.dct_sas_communication | impala::ap_itbd.dct_sas_communication | Added |
| hadoop_dl.dct_sas_segment | impala::ap_itbd.dct_sas_segment | Added |
| owner_ogg.dct_capp_message_template | impala::OWNER_OGG.DCT_CAPP_MESSAGE_TEMPLATE | Added |
| owner_ogg.dct_sms_template | impala::OWNER_OGG.DCT_SMS_TEMPLATE | Added |
| hadoop_dl.clt_sas_contact_status | impala::ap_itbd.clt_sas_contact_status | Added |
| hadoop_dl.ft_sas_camp_segment_client_at | impala::AP_ITBD.ft_sas_camp_segment_client_at | Added |
| hadoop_dl.ft_sas_segment_treatment_tt | impala::AP_ITBD.ft_sas_segment_treatment_tt | Added |
| winnie_wangtjww.hcp_crm_offer | None | No |
| owner_ogg.dct_wechat_template | impala::OWNER_OGG.DCT_WECHAT_TEMPLATE | Added |
| hadoop_dl.dct_sas_cell | impala::AP_ITBD.dct_sas_cell | Added |
| owner_ogg.ft_mobile_app_user_ad | impala::OWNER_OGG.FT_MOBILE_APP_USER_AD | Added |
+-----------------------------------------+-----------------------------------------------+--------------+
Excute Progress: 5/5
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
sparktool-2.0.1.tar.gz
(5.7 kB
view hashes)