Simple and easy to use Qubole connector
Project description
PyQubole
A watered down version of Qubole's Python connector providing a much simpler API to interact with for running streaming queries or submitting a job and rerieving its output at a later time (sync or async). Allowing for easy use in notebooks or integration in projects without much overhead. Based on Qubole QDS-SDK-Py https://github.com/qubole/qds-sdk-py
Installtion
From PyPI
The library is available on PyPI - PyQubole.
$ pip install pyqubole
From Source
•Get source code: SSH git@github.com:achilleasatha/PyQubole.git
or HTTPS https://github.com/achilleasatha/PyQubole.git
•Install by running python setup.py install
API
You can find an example application in example/main.py
An example application needs to do:
- Set the api_token and instantiate the connection
con = QuboleConnector(api_token='api_token')
- Use the query data method to run a job, specifying the input query, engine and cluster (or just job_id):
data = con.query_data(sql_query=query, job_id=None, engine='Hive', cluster='Hive_cluster_name', verbose=False)
Note:
a) Query can be passed as a raw string query = r"""select * from table"""
or from a file: query = open('query.sql').read()
b) If job_id = None
the query will be executed on the engine specified ('Hive' or 'Presto')
c) If job_id = '123456'
then the results of the job will be retrieved (if job status is done)
d) You can use the optional method verbose = True / False
to get streaming output or only status updates
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for qubolepystream-0.3.win-amd64.zip
Algorithm | Hash digest | |
---|---|---|
SHA256 | a863dad378745c9528a324e937f2d9c4b0dc354a5afbaf99897b8edbe8dceb88 |
|
MD5 | bc28a9e515e90d1620c59d48700e7b8b |
|
BLAKE2b-256 | 4c16621d85518b2f8d7a310b1a89878c92a9ef728ce12afbfd580680af97fd85 |