Skip to main content
https://travis-ci.org/dropbox/PyHive.svg?branch=master https://img.shields.io/codecov/c/github/dropbox/PyHive.svg

PyHive

PyHive is a collection of Python DB-API and SQLAlchemy interfaces for Presto and Hive.

Usage

DB-API

from pyhive import presto  # or import hive or import trino
cursor = presto.connect('localhost').cursor()
cursor.execute('SELECT * FROM my_awesome_data LIMIT 10')
print cursor.fetchone()
print cursor.fetchall()

DB-API (asynchronous)

from pyhive import hive
from TCLIService.ttypes import TOperationState
cursor = hive.connect('localhost').cursor()
cursor.execute('SELECT * FROM my_awesome_data LIMIT 10', async=True)

status = cursor.poll().operationState
while status in (TOperationState.INITIALIZED_STATE, TOperationState.RUNNING_STATE):
    logs = cursor.fetch_logs()
    for message in logs:
        print message

    # If needed, an asynchronous query can be cancelled at any time with:
    # cursor.cancel()

    status = cursor.poll().operationState

print cursor.fetchall()

In Python 3.7 async became a keyword; you can use async_ instead:

cursor.execute('SELECT * FROM my_awesome_data LIMIT 10', async_=True)

SQLAlchemy

First install this package to register it with SQLAlchemy (see setup.py).

from sqlalchemy import *
from sqlalchemy.engine import create_engine
from sqlalchemy.schema import *
# Presto
engine = create_engine('presto://localhost:8080/hive/default')
# Trino
engine = create_engine('trino://localhost:8080/hive/default')
# Hive
engine = create_engine('hive://localhost:10000/default')
logs = Table('my_awesome_data', MetaData(bind=engine), autoload=True)
print select([func.count('*')], from_obj=logs).scalar()

# Hive + HTTPS + LDAP or basic Auth
engine = create_engine('hive+https://username:password@localhost:10000/')
logs = Table('my_awesome_data', MetaData(bind=engine), autoload=True)
print select([func.count('*')], from_obj=logs).scalar()

Note: query generation functionality is not exhaustive or fully tested, but there should be no problem with raw SQL.

Passing session configuration

# DB-API
hive.connect('localhost', configuration={'hive.exec.reducers.max': '123'})
presto.connect('localhost', session_props={'query_max_run_time': '1234m'})
trino.connect('localhost',  session_props={'query_max_run_time': '1234m'})
# SQLAlchemy
create_engine(
    'presto://user@host:443/hive',
    connect_args={'protocol': 'https',
                  'session_props': {'query_max_run_time': '1234m'}}
)
create_engine(
    'trino://user@host:443/hive',
    connect_args={'protocol': 'https',
                  'session_props': {'query_max_run_time': '1234m'}}
)
create_engine(
    'hive://user@host:10000/database',
    connect_args={'configuration': {'hive.exec.reducers.max': '123'}},
)
# SQLAlchemy with LDAP
create_engine(
    'hive://user:password@host:10000/database',
    connect_args={'auth': 'LDAP'},
)

Requirements

Install using

  • pip install 'pyhive[hive]' for the Hive interface and

  • pip install 'pyhive[presto]' for the Presto interface.

  • pip install 'pyhive[trino]' for the Trino interface

PyHive works with

  • Python 2.7 / Python 3

  • For Presto: Presto install

  • For Trino: Trino install

  • For Hive: HiveServer2 daemon

Changelog

See https://github.com/dropbox/PyHive/releases.

Contributing

  • Please fill out the Dropbox Contributor License Agreement at https://opensource.dropbox.com/cla/ and note this in your pull request.

  • Changes must come with tests, with the exception of trivial things like fixing comments. See .travis.yml for the test environment setup.

  • Notes on project scope:

    • This project is intended to be a minimal Hive/Presto client that does that one thing and nothing else. Features that can be implemented on top of PyHive, such integration with your favorite data analysis library, are likely out of scope.

    • We prefer having a small number of generic features over a large number of specialized, inflexible features. For example, the Presto code takes an arbitrary requests_session argument for customizing HTTP calls, as opposed to having a separate parameter/branch for each requests option.

Testing

https://travis-ci.org/dropbox/PyHive.svg http://codecov.io/github/dropbox/PyHive/coverage.svg?branch=master

Run the following in an environment with Hive/Presto:

./scripts/make_test_tables.sh
virtualenv --no-site-packages env
source env/bin/activate
pip install -e .
pip install -r dev_requirements.txt
py.test

WARNING: This drops/creates tables named one_row, one_row_complex, and many_rows, plus a database called pyhive_test_database.

Updating TCLIService

The TCLIService module is autogenerated using a TCLIService.thrift file. To update it, the generate.py file can be used: python generate.py <TCLIServiceURL>. When left blank, the version for Hive 2.3 will be downloaded.

Metadata

Release files for acryl-PyHive 0.6.18

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for acryl-PyHive 0.6.18
File Size Uploaded Provenance
acryl_pyhive-0.6.18.tar.gz 50.2 kB Without provenance Details

Built distribution (wheel)

Table of built distributions (wheels) for acryl-PyHive 0.6.18
File Interpreter ABI Platform
acryl_pyhive-0.6.18-py3-none-any.whl Python 3 none any Details

Total release size: 105.9 kB

Release files / acryl_pyhive-0.6.18.tar.gz

Download URL acryl_pyhive-0.6.18.tar.gz
Size 50.2 kB
Tags Source
SHA-256 checksum
How to use checksums
a7e0cb17b029f9447a17dd832c2a4b5c3a5050038d6c686b821b6945c17ce677
BLAKE2b-256 checksum
How to use checksums
67569e08303eb5214f3086046a1b2bbcddc8fb04ed5cbdf8f8e016e1604a10f8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.13

Release files / acryl_pyhive-0.6.18-py3-none-any.whl

Download URL acryl_pyhive-0.6.18-py3-none-any.whl
Size 55.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
02f5044fa547e643bf4ecf408307c1cbb1eeacb4bb025f08a2e86d973bc8572b
BLAKE2b-256 checksum
How to use checksums
aa5b734c35a49ce7919b258d1f4c6919adff99e169d5cb67d9b0451ac2d0f1a9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.13

Security for acryl-PyHive 0.6.18

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page