Client-less data retrieval from Hive.
Project description
hivehoney
Extract data from remote Hive to local Windows OS (without Hadoop client).
The most difficult part was figuring out expect+pbrun.
Because there are 2 interactive questions I had to pause after password.
Mode expect+pbrun details are here: https://github.com/hive-scripts/hivehoney/blob/master/expect_pbrun_howto.md
Data access path.
Windows desktop->
SSH->
Linux login->
pbrun service login->
kinit
beeline->
SQL->
save echo on Windows
Run it like this:
set PROXY_HOST=your_bastion_host
set SERVICE_USER=you_func_user
set LINUX_USER=your_SOID
set LINUX_PWD=your_pwd
python hh.py --query_file=query.sql
query.sql
select * from gfocnnsg_work.pytest LIMIT 1000000;
Result:
TOTAL BYTES: 60000127
Elaplsed: 79.637 s
exit status: 0
0
[]
TOTAL Elaplsed: 99.060 s
data_dump.csv
c:\tmp>dir data_dump.csv
Directory of c:\tmp
09/04/2018 12:53 PM 60,000,075 data_dump.csv
1 File(s) 60,000,075 bytes
0 Dir(s) 321,822,720 bytes free
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
hivehoney-1.0.4.tar.gz
(7.0 kB
view hashes)
Built Distribution
Close
Hashes for hivehoney-1.0.4-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a3770b5578cd29840d4dcd67fbef05ad47738e284056adeea1c8f992316ccca3 |
|
MD5 | 66e830ee01c3171081bd8e703d88214b |
|
BLAKE2b-256 | f084133b07be4efb30ce124c6aaf6b1bd9cb1b7160a38b696b22ff72549dedd6 |