This release is a pre-release and may not be stable for production use.
[Please visit the project page at https://github.com/qubole/qds-sdk-py]
Qubole Data Service Python SDK
A Python module that provides the tools you need to authenticate with, and use the Qubole Data Service API.
Installation
Run the following command (may need to do this as root):
$ python setup.py install
This should place a command line utility ‘qds.py’ somewhere in your path
$ which qds.py /usr/bin/qds.py
CLI
qds.py allows running Hive, Hadoop and Pig commands against QDS. Users can run commands synchronously - or submit a command and check it’s status.
$ qds.py -h # will print detailed usage
Examples:
run a hive query and print the results
$ qds.py –token ‘xxyyzz’ hivecmd run –query “show tables”
pass in api token from bash environment variable
$ export QDS_API_TOKEN=xxyyzz
run the example hadoop command
$ qds.py hadoopcmd run jar ‘s3://paid-qubole/HadoopAPIExamples/jars/hadoop-0.20.1-dev-streaming.jar’ -files ‘s3n://paid-qubole/HadoopAPIExamples/WordCountPython/mapper.py,s3n://paid-qubole/HadoopAPIExamples/WordCountPython/reducer.py’ -mapper mapper.py -reducer reducer.py -numReduceTasks 1 -input ‘s3n://paid-qubole/default-datasets/gutenberg’ -output ‘s3n://example.bucket.com/wcout’
check the status of command # 12345678
$ qds.py hivecmd check 12345678 {“status”: “done”, … }
SDK API
An example Python application needs to do the following:
Set the api_token:
from qubole import Qubole
Qubole.configure(api_token=’ksbdvcwdkjn123423’)
Use the Command classes defined in commands.py to execute commands. To run Hive Command:
from commands import *
hc=HiveCommand.create(query=’show tables’) print “Id: %s, Status: %s” % (str(hc.id), hc.status)
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file qds_sdk-1.0.2-beta.tar.gz.
File metadata
- Download URL: qds_sdk-1.0.2-beta.tar.gz
- Upload date:
- Size: 9.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c3c8acd01b20772f6628e24d9d5b018d0ddb96ad7abe6ea9674df4d1c6f9bd1b
|
|
| MD5 |
f82454f5689f5ca2f30d04b6e34cf882
|
|
| BLAKE2b-256 |
e886e83c6c05b14e7b320bd2611ea00845c149cac7f07014858adddeb98313ac
|
