Skip to main content

Simple and easy to use Qubole connector

Project description

PyQubole

A watered down version of Qubole's Python connector providing a much simpler API to interact with for running streaming queries or submitting a job and rerieving its output at a later time (sync or async). Allowing for easy use in notebooks or integration in projects without much overhead. Based on Qubole QDS-SDK-Py https://github.com/qubole/qds-sdk-py

Installtion

From PyPI

The library is available on PyPI - PyQubole.

$ pip install qubolepystream

From Source

•Get source code: SSH git@github.com:achilleasatha/PyQubole.git or HTTPS https://github.com/achilleasatha/PyQubole.git

•Install by running python setup.py install

API

You can find an example application in example/main.py including a Spark command example example/spark_example.py.

More info on running Spark commands: Qubole docs - Submit a Spark command

An example application needs to do:

  1. Import the lib from qubolepystream.connector import QuboleConnector
  2. Set the api_token and instantiate the connection con = QuboleConnector(api_token='api_token')
  3. Use the query data method to run a job, specifying the input query, engine and cluster (or just job_id): data = con.query_data(sql_query=query, job_id=None, engine='Hive', cluster='Hive_cluster_name', verbose=False)

Note:

a) Query can be passed as a raw string query = r"""select * from table""" or from a file: query = open('query.sql').read()

b) If job_id = None the query will be executed on the engine specified ('Hive', 'Presto' or 'Spark'')

c) If job_id = '123456' then the results of the job will be retrieved (if job status is done)

d) You can use the optional method verbose = True / False to get streaming logs output or only status updates

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

qubolepystream-0.8.0-py3-none-any.whl (3.8 kB view details)

Uploaded Python 3

File details

Details for the file qubolepystream-0.8.0-py3-none-any.whl.

File metadata

  • Download URL: qubolepystream-0.8.0-py3-none-any.whl
  • Upload date:
  • Size: 3.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3

File hashes

Hashes for qubolepystream-0.8.0-py3-none-any.whl
Algorithm Hash digest
SHA256 e7e0e522410f5b1753e4b9fdf790f35b00889c46913ffada55c37c04d3e822b5
MD5 3cf5e8eff599b5c22a69a70498503fd8
BLAKE2b-256 355fa80b57b7aa9d5c99df671a8f5c1234cf2b77743998888848958c8dd167a0

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page