Common utility functions for data engineering usecases
Project description
hip-data-tools
© Hipages Group Pty Ltd 2019
Common Python tools and utilities for data engineering, ETL, Exploration, etc. The package is uploaded to PyPi for easy drop and use in various environmnets, such as (but not limited to):
- Running production workloads
- ML Training in Jupyter like notebooks
- Local machine for dev and exploration
Installation
Install from PyPi repo:
pip3 install hip-data-tools
Install from source
pip3 install .
Connect to aws
You will need to instantiate an AWS Connection:
from hip_data_tools.authenticate import AwsConnection
conn = AwsConnection(mode="assume_role", settings={"profile_name": "default"})
# OR if you want to connect using Env Vars:
conn = AwsConnection(mode="standard_env_var", settings={})
# OR if you want custom set of env vars to connect
conn = AwsConnection(mode="custom_env_var", settings={
"aws_access_key_id_env_var": "aws_access_key_id",
"aws_secret_access_key_env_var": "aws_secret_access_key"
})
Using this connection to object you can use the aws utilities, for example aws Athena:
from hip_data_tools.aws.athena import AthenaUtil
au = AthenaUtil(database="default", conn=conn, output_bucket="example", output_key="tmp/scratch/")
result = au.run_query("SELECT * FROM temp limit 10", return_result=True)
print(result)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
hip_data_tools-1.7.1.tar.gz
(22.9 kB
view hashes)
Built Distribution
Close
Hashes for hip_data_tools-1.7.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 58458324f974e97d1487b045177b8c32b2a83cc7152db8914190d657398de8a1 |
|
MD5 | 0ec9a25b9bf5be5c18decaed66ec8f03 |
|
BLAKE2b-256 | cc383120f6d4ed6f71dc2917a85460a0f1dc991934e3a268ebc7cc759df08604 |