Skip to main content

Simple and easy to use Qubole connector

Project description

PyQubole

A watered down version of Qubole's Python connector providing a much simpler API to interact with for running streaming queries or submitting a job and rerieving its output at a later time (sync or async). Allowing for easy use in notebooks or integration in projects without much overhead. Based on Qubole QDS-SDK-Py https://github.com/qubole/qds-sdk-py

Installtion

From PyPI

The library is available on PyPI - PyQubole.

$ pip install qubolepystream

From Source

•Get source code: SSH git@github.com:achilleasatha/PyQubole.git or HTTPS https://github.com/achilleasatha/PyQubole.git

•Install by running python setup.py install

API

You can find an example application in example/main.py

An example application needs to do:

  1. Set the api_token and instantiate the connection con = QuboleConnector(api_token='api_token')
  2. Use the query data method to run a job, specifying the input query, engine and cluster (or just job_id): data = con.query_data(sql_query=query, job_id=None, engine='Hive', cluster='Hive_cluster_name', verbose=False)

Note:

a) Query can be passed as a raw string query = r"""select * from table""" or from a file: query = open('query.sql').read()

b) If job_id = None the query will be executed on the engine specified ('Hive' or 'Presto')

c) If job_id = '123456' then the results of the job will be retrieved (if job status is done)

d) You can use the optional method verbose = True / False to get streaming output or only status updates

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

qubolepystream-0.7.tar.gz (3.4 kB view hashes)

Uploaded Source

Built Distribution

qubolepystream-0.7-py3-none-any.whl (3.5 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page