Python interface to iomete (Hive)
Project description
py-hive-iomete is a collection of Python DB-API and SQLAlchemy interfaces for iomete hive.
Usage
DB-API
from pyhive import hive
connection = hive.connect(
host="<data_plane_host>",
port=<data_plane_port>,
scheme="http", # or "https"
lakehouse="<lakehouse_cluster_name>",
data_plane=None # or data_plane (namespace)
database="default",
username="<username>",
password="<password>"
)
cursor = connection.cursor()
cursor.execute("SELECT * FROM my_awesome_data LIMIT 10")
print(cursor.fetchone())
print(cursor.fetchall())
DB-API (asynchronous)
from pyhive import hive
from TCLIService.ttypes import TOperationState
connection = hive.connect(
host="<data_plane_host>",
port=<data_plane_port>,
scheme="http", # or "https"
lakehouse="<lakehouse_cluster_name>",
data_plane=None # or data_plane (namespace)
database="default",
username="<username>",
password="<password>"
)
cursor = connection.cursor()
cursor.execute("SELECT * FROM my_awesome_data LIMIT 10", async_=True)
status = cursor.poll().operationState
while status in (TOperationState.INITIALIZED_STATE, TOperationState.RUNNING_STATE):
logs = cursor.fetch_logs()
for message in logs:
print(message)
# If needed, an asynchronous query can be cancelled at any time with:
# cursor.cancel()
status = cursor.poll().operationState
print(cursor.fetchall())
SQLAlchemy
First install this package to register it with SQLAlchemy (see setup.py).
from sqlalchemy.engine import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.schema import *
# Possible dialects (hive and iomete are both operate identically):
# hive+http
# hive+https
# iomete+http
# iomete+https
engine = create_engine(
'iomete+https://<username>:<password>@<data_plane_host>:<data_plane_port>/<database>?lakehouse=<lakehouse_cluster_name>')
# or with data_plane specified
# engine = create_engine(
# 'iomete+https://<username>:<password>@<data_plane_host>:<data_plane_port>/<database>?lakehouse=<lakehouse_cluster_name>&data_plane=<data_plane>')
# Alternatively, "hive" driver could be used as well
# engine = create_engine(
# 'hive+https://<username>:<password>@<data_plane_host>:<data_plane_port>/<database>?lakehouse=<lakehouse_cluster_name>')
session = sessionmaker(bind=engine)()
records = session.query(Table('my_awesome_data', MetaData(bind=engine), autoload=True)) \
.limit(10) \
.all()
print(records)
Note: query generation functionality is not exhaustive or fully tested, but there should be no problem with raw SQL.
Requirements
Install using
pip install 'py-hive-iomete' for the DB-API interface
pip install 'py-hive-iomete[sqlalchemy]' for the SQLAlchemy interface
py-hive-iomete works with
Python 2.7 / Python 3
Changelog
Contributing
Changes must come with tests, with the exception of trivial things like fixing comments. See .travis.yml for the test environment setup.
Notes on project scope:
This project is intended to be a minimal iomete (hive) client that does that one thing and nothing else. Features that can be implemented on top of py-hive-iomete, such integration with your favorite data analysis library, are likely out of scope.
We prefer having a small number of generic features over a large number of specialized, inflexible features.
Updating TCLIService
The TCLIService module is autogenerated using a TCLIService.thrift file. To update it, the generate.py file can be used: python generate.py <TCLIServiceURL>. When left blank, the version for Hive 2.3 will be downloaded.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file py_hive_iomete-2.1.1.tar.gz
.
File metadata
- Download URL: py_hive_iomete-2.1.1.tar.gz
- Upload date:
- Size: 44.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | fc27f047beee986a5c1bea0270252f5527d24cafd73d5a0d97fd09536727ccbb |
|
MD5 | b2ad8172dac01061063f73c1aeaf6f27 |
|
BLAKE2b-256 | 09a0460af7a673903f730560570b7d61eee5accaada6b82992e350f2ef8c58d0 |
File details
Details for the file py_hive_iomete-2.1.1-py3-none-any.whl
.
File metadata
- Download URL: py_hive_iomete-2.1.1-py3-none-any.whl
- Upload date:
- Size: 50.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5ce91f1f9152793f538e2a2abc8ad14000b4623f2e19a85f250e015a7fe7d9d6 |
|
MD5 | 6b08a3fc50ce8b20fd88bfcf384386ee |
|
BLAKE2b-256 | ac028e855316e97e01b981bd8dca0bfd274f41c1e4f74a301741263f3ff22b05 |