Skip to main content

Simple and easy to use Qubole connector

Project description

PyQubole

A watered down version of Qubole's Python connector providing a much simpler API to interact with for running streaming queries or submitting a job and rerieving its output at a later time (sync or async). Allowing for easy use in notebooks or integration in projects without much overhead. Based on Qubole QDS-SDK-Py https://github.com/qubole/qds-sdk-py

Installtion

From PyPI

The library is available on PyPI - PyQubole.

$ pip install qubolepystream

From Source

•Get source code: SSH git@github.com:achilleasatha/PyQubole.git or HTTPS https://github.com/achilleasatha/PyQubole.git

•Install by running python setup.py install

API

You can find an example application in example/main.py including a Spark command example example/spark_example.py.

More info on running Spark commands: Qubole docs - Submit a Spark command

An example application needs to do:

  1. Import the lib from qubolepystream.connector import QuboleConnector
  2. Set the api_token and instantiate the connection con = QuboleConnector(api_token='api_token')
  3. Use the query data method to run a job, specifying the input query, engine and cluster (or just job_id): data = con.query_data(sql_query=query, job_id=None, engine='Hive', cluster='Hive_cluster_name', verbose=False)

Note:

a) Query can be passed as a raw string query = r"""select * from table""" or from a file: query = open('query.sql').read()

b) If job_id = None the query will be executed on the engine specified ('Hive', 'Presto' or 'Spark'')

c) If job_id = '123456' then the results of the job will be retrieved (if job status is done)

d) You can use the optional method verbose = True / False to get streaming logs output or only status updates

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for qubolepystream, version 0.8.0
Filename, size File type Python version Upload date Hashes
Filename, size qubolepystream-0.8.0-py3-none-any.whl (3.8 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page