Skip to main content

sparktool for hccn

Project description

sparktool

built to simplified the process of creating sparksession(hccn).

  • can get saved query from hue
  • batch execute multiple sqls which are from same sql scripts
  • can parse impala/hive view automatically
  • can parse kudu table automatically

what's new

  • get saved query from hue

install

Python 2/3

  • pip install --user --upgrade sparktool
  • pip2 install --user --upgrade sparktool

functions

switch_keytab (first run)

import sparktool as st st.switch_keytab('admin@EXAMPLE.COM', keytabpath)


switch_keytab (first run)
--------
import sparktool as st
st.switch_huetab('admin@EXAMPLE.COM', keytabpath)

batch_excutesql

[in]

# encoding=utf-8
import sparktool as st

ss = st.SparkCreator()
sql = '''
select 1;
select
    cc.skp_client
from
    owner_ogg.ft_ccase_2_ccase_ad cc
    join owner_ogg.clt_ccase_2_ccase_relation re
      on cc.skp_ccase_2_ccase_relation = re.skp_ccase_2_ccase_relation
     and re.code_ccase_relation = 'FIRST_POS' 
limit 1
;
'''
ss.batch_excutesql(sql)

[out]

Tranform Table:
+--------------------------------------+--------------------------------------+--------------+
|             Origin Table             |            Temporary View            | If Transform |
+--------------------------------------+--------------------------------------+--------------+
|    OWNER_OGG.FT_CCASE_2_CCASE_AD     |    owner_ogg_ft_ccase_2_ccase_ad     |     New      |
| OWNER_OGG.CLT_CCASE_2_CCASE_RELATION | owner_ogg_clt_ccase_2_ccase_relation |     New      |
+--------------------------------------+--------------------------------------+--------------+
Excute Progress: 2/2
DataFrame[skp_client: decimal(38,0)]

[in]

# encoding=utf-8
import sparktool as st

ss = st.SparkCreator()
sql = '''
select * from winnie_wangtjww.hcp_crm_offer limit 1;
drop table if exists ttttttttt;
select 1 as tt;
select * from ap_crm.ft_sas_segment_client limit 1;
select * from owner_ogg.ft_mobile_app_user_ad limit 1;
'''
b,c,d,e,f = ss.batch_excutesql(sql, ifview=True, ifbatchre=True)

[out]

Tranform Table:
+-----------------------------------------+-----------------------------------------------+--------------+
|               Origin Table              |                   Kudu Table                  | If Transform |
+-----------------------------------------+-----------------------------------------------+--------------+
|        hadoop_dl.dct_sas_campaign       |        impala::AP_ITBD.dct_sas_campaign       |    Added     |
|     hadoop_dl.dct_sas_communication     |     impala::ap_itbd.dct_sas_communication     |    Added     |
|        hadoop_dl.dct_sas_segment        |        impala::ap_itbd.dct_sas_segment        |    Added     |
|   owner_ogg.dct_capp_message_template   |  impala::OWNER_OGG.DCT_CAPP_MESSAGE_TEMPLATE  |    Added     |
|        owner_ogg.dct_sms_template       |       impala::OWNER_OGG.DCT_SMS_TEMPLATE      |    Added     |
|     hadoop_dl.clt_sas_contact_status    |     impala::ap_itbd.clt_sas_contact_status    |    Added     |
| hadoop_dl.ft_sas_camp_segment_client_at | impala::AP_ITBD.ft_sas_camp_segment_client_at |    Added     |
|  hadoop_dl.ft_sas_segment_treatment_tt  |  impala::AP_ITBD.ft_sas_segment_treatment_tt  |    Added     |
|      winnie_wangtjww.hcp_crm_offer      |                      None                     |      No      |
|      owner_ogg.dct_wechat_template      |     impala::OWNER_OGG.DCT_WECHAT_TEMPLATE     |    Added     |
|          hadoop_dl.dct_sas_cell         |          impala::AP_ITBD.dct_sas_cell         |    Added     |
|     owner_ogg.ft_mobile_app_user_ad     |    impala::OWNER_OGG.FT_MOBILE_APP_USER_AD    |    Added     |
+-----------------------------------------+-----------------------------------------------+--------------+
Excute Progress: 5/5

batch_excutesql

[in]

# encoding=utf-8
from sparktool import HueCreator
aa  = HueCreator()
aa.query_printlist()

[out]

+--------+-------------------------+----------------------------+-------------------+
|   id   |           name          |        description         |   last_modified   |
+--------+-------------------------+----------------------------+-------------------+
| 691124 |     aaaaaaaaaaaaaaa     |                            | 2020-01-17T17:41Z |
| 686849 |           hcp           |                            | 2020-01-17T10:32Z |
| 675390 |           aaaa          |                            | 2020-01-16T11:05Z |
| 681235 |          ttttt          |                            | 2020-01-15T09:41Z |
| 676699 |           aaaa          |                            | 2020-01-14T09:53Z | |
+--------+-------------------------+----------------------------+-------------------+

[in]

# encoding=utf-8
aa.query_getscript('hcp')

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for sparktool, version 3.0.1
Filename, size File type Python version Upload date Hashes
Filename, size sparktool-3.0.1.tar.gz (7.6 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page