Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Python SDK for coding to the Qubole Data Service API

Project Description

A Python module that provides the tools you need to authenticate with, and use the Qubole Data Service API.

Installation

From PyPI

The SDK is available on PyPI.

$ pip install qds-sdk

From source

  • Download the source code:

  • Run the following command (may need to do this as root):

    $ python setup.py install
    
  • Alternatively, if you use virtualenv, you can do this:

    $ cd qds-sdk-py
    $ virtualenv venv
    $ source venv/bin/activate
    $ python setup.py install
    

This should place a command line utility qds.py somewhere in your path

$ which qds.py
/usr/bin/qds.py

CLI

qds.py allows running Hive, Hadoop, Pig, Presto and Shell commands against QDS. Users can run commands synchronously - or submit a command and check its status.

$ qds.py -h  # will print detailed usage

Examples:

  1. run a hive query and print the results

    $ qds.py --token 'xxyyzz' hivecmd run --query "show tables"
    $ qds.py --token 'xxyyzz' hivecmd run --script_location /tmp/myquery
    $ qds.py --token 'xxyyzz' hivecmd run --script_location s3://my-qubole-location/myquery
    
  2. pass in api token from bash environment variable

    $ export QDS_API_TOKEN=xxyyzz
    
  3. run the example hadoop command

    $ qds.py hadoopcmd run streaming -files 's3n://paid-qubole/HadoopAPIExamples/WordCountPython/mapper.py,s3n://paid-qubole/HadoopAPIExamples/WordCountPython/reducer.py' -mapper mapper.py -reducer reducer.py -numReduceTasks 1 -input 's3n://paid-qubole/default-datasets/gutenberg' -output 's3n://example.bucket.com/wcout'
    
  4. check the status of command # 12345678

    $ qds.py hivecmd check 12345678
    {"status": "done", ... }
    

SDK API

An example Python application needs to do the following:

  1. Set the api_token:

    from qds_sdk.qubole import Qubole
    
    Qubole.configure(api_token='ksbdvcwdkjn123423')
    
  2. Use the Command classes defined in commands.py to execute commands. To run Hive Command:

    from qds_sdk.commands import *
    
    hc=HiveCommand.create(query='show tables')
    print "Id: %s, Status: %s" % (str(hc.id), hc.status)
    

example/mr_1.py contains a Hadoop Streaming example

Release History

Release History

This version
History Node

1.9.8

History Node

1.9.7

History Node

1.9.6

History Node

1.9.5

History Node

1.9.4

History Node

1.9.3

History Node

1.9.2

History Node

1.9.1

History Node

1.9.0

History Node

1.8.0

History Node

1.7.0

History Node

1.6.2

History Node

1.6.1

History Node

1.6.0

History Node

1.5.1

History Node

1.5.0

History Node

1.4.3

History Node

1.4.2

History Node

1.4.1

History Node

1.4.0

History Node

1.3.4

History Node

1.3.3

History Node

1.3.2

History Node

1.3.1

History Node

1.3.0

History Node

1.3.0b2

History Node

1.3.0b1

History Node

1.2.2

History Node

1.2.1

History Node

1.2.0

History Node

1.1.3

History Node

1.1.2

History Node

1.1.2rc2

History Node

1.1.2rc1

History Node

1.1.1

History Node

1.1.0

History Node

1.0.18

History Node

1.0.17b1

History Node

1.0.16

History Node

1.0.15b1

History Node

1.0.14b1

History Node

1.0.13b1

History Node

1.0.12b1

History Node

1.0.11b1

History Node

1.0.10-beta

History Node

1.0.9-beta

History Node

1.0.8-beta

History Node

1.0.7-beta

History Node

1.0.6-beta

History Node

1.0.5-beta

History Node

1.0.3-beta

History Node

1.0.2-beta

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
qds_sdk-1.9.8.tar.gz (55.9 kB) Copy SHA256 Checksum SHA256 Source Aug 18, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting