Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

Project is currently Supported by 6sense

https://travis-ci.org/dropbox/PyHive.svg?branch=master https://img.shields.io/codecov/c/github/dropbox/PyHive.svg

PyHive

PyHive is a collection of Python DB-API and SQLAlchemy interfaces for Presto , Hive and Trino.

Extra from Dropbox/Pyhive:

  • Connection Timeout (hive.connect(host=host, timeout=connection_timeout))

  • Query Timeout (hive.connect(host=host, query_timeout=query_timeout))

Usage

DB-API

from pyhive import presto  # or import hive or import trino
cursor = presto.connect('localhost').cursor()  # or use hive.connect or use trino.connect
cursor.execute('SELECT * FROM my_awesome_data LIMIT 10')
print cursor.fetchone()
print cursor.fetchall()

DB-API (asynchronous)

from pyhive import hive
from TCLIService.ttypes import TOperationState
cursor = hive.connect('localhost').cursor()
cursor.execute('SELECT * FROM my_awesome_data LIMIT 10', async=True)

status = cursor.poll().operationState
while status in (TOperationState.INITIALIZED_STATE, TOperationState.RUNNING_STATE):
    logs = cursor.fetch_logs()
    for message in logs:
        print message

    # If needed, an asynchronous query can be cancelled at any time with:
    # cursor.cancel()

    status = cursor.poll().operationState

print cursor.fetchall()

In Python 3.7 async became a keyword; you can use async_ instead:

cursor.execute('SELECT * FROM my_awesome_data LIMIT 10', async_=True)

SQLAlchemy

First install this package to register it with SQLAlchemy, see entry_points in setup.py.

from sqlalchemy import *
from sqlalchemy.engine import create_engine
from sqlalchemy.schema import *
# Presto
engine = create_engine('presto://localhost:8080/hive/default')
# Trino
engine = create_engine('trino+pyhive://localhost:8080/hive/default')
# Hive
engine = create_engine('hive://localhost:10000/default')

# SQLAlchemy < 2.0
logs = Table('my_awesome_data', MetaData(bind=engine), autoload=True)
print select([func.count('*')], from_obj=logs).scalar()

# Hive + HTTPS + LDAP or basic Auth
engine = create_engine('hive+https://username:password@localhost:10000/')
logs = Table('my_awesome_data', MetaData(bind=engine), autoload=True)
print select([func.count('*')], from_obj=logs).scalar()

# SQLAlchemy >= 2.0
metadata_obj = MetaData()
books = Table("books", metadata_obj, Column("id", Integer), Column("title", String), Column("primary_author", String))
metadata_obj.create_all(engine)
inspector = inspect(engine)
inspector.get_columns('books')

with engine.connect() as con:
    data = [{ "id": 1, "title": "The Hobbit", "primary_author": "Tolkien" },
            { "id": 2, "title": "The Silmarillion", "primary_author": "Tolkien" }]
    con.execute(books.insert(), data[0])
    result = con.execute(text("select * from books"))
    print(result.fetchall())

Note: query generation functionality is not exhaustive or fully tested, but there should be no problem with raw SQL.

Passing session configuration

# DB-API
hive.connect('localhost', configuration={'hive.exec.reducers.max': '123'})
presto.connect('localhost', session_props={'query_max_run_time': '1234m'})
trino.connect('localhost',  session_props={'query_max_run_time': '1234m'})
# SQLAlchemy
create_engine(
    'presto://user@host:443/hive',
    connect_args={'protocol': 'https',
                  'session_props': {'query_max_run_time': '1234m'}}
)
create_engine(
    'trino+pyhive://user@host:443/hive',
    connect_args={'protocol': 'https',
                  'session_props': {'query_max_run_time': '1234m'}}
)
create_engine(
    'hive://user@host:10000/database',
    connect_args={'configuration': {'hive.exec.reducers.max': '123'}},
)
# SQLAlchemy with LDAP
create_engine(
    'hive://user:password@host:10000/database',
    connect_args={'auth': 'LDAP'},
)

Requirements

Install using

  • pip install 'pyhive[hive]' or pip install 'pyhive[hive_pure_sasl]' for the Hive interface

  • pip install 'pyhive[presto]' for the Presto interface

  • pip install 'pyhive[trino]' for the Trino interface

Note: 'pyhive[hive]' extras uses sasl that doesn’t support Python 3.11, See github issue. Hence PyHive also supports pure-sasl via additional extras 'pyhive[hive_pure_sasl]' which support Python 3.11.

PyHive works with

Changelog

See https://github.com/dropbox/PyHive/releases.

Contributing

  • Please fill out the Dropbox Contributor License Agreement at https://opensource.dropbox.com/cla/ and note this in your pull request.

  • Changes must come with tests, with the exception of trivial things like fixing comments. See .travis.yml for the test environment setup.

  • Notes on project scope:

    • This project is intended to be a minimal Hive/Presto client that does that one thing and nothing else. Features that can be implemented on top of PyHive, such integration with your favorite data analysis library, are likely out of scope.

    • We prefer having a small number of generic features over a large number of specialized, inflexible features. For example, the Presto code takes an arbitrary requests_session argument for customizing HTTP calls, as opposed to having a separate parameter/branch for each requests option.

Tips for test environment setup

You can setup test environment by following .travis.yaml in this repository. It uses Cloudera’s CDH 5 which requires username and password for download. It may not be feasible for everyone to get those credentials. Hence below are alternative instructions to setup test environment.

You can clone this repository which has Docker Compose setup for Presto and Hive. You can add below lines to its docker-compose.yaml to start Trino in same environment:

trino:
    image: trinodb/trino:351
    ports:
        - "18080:18080"
    volumes:
        - ./trino:/etc/trino

Note: ./trino for docker volume defined above is trino config from PyHive repository

Then run::

docker-compose up -d

Testing

https://travis-ci.org/dropbox/PyHive.svg http://codecov.io/github/dropbox/PyHive/coverage.svg?branch=master

Run the following in an environment with Hive/Presto:

./scripts/make_test_tables.sh
virtualenv --no-site-packages env
source env/bin/activate
pip install -e .
pip install -r dev_requirements.txt
py.test

WARNING: This drops/creates tables named one_row, one_row_complex, and many_rows, plus a database called pyhive_test_database.

Updating TCLIService

The TCLIService module is autogenerated using a TCLIService.thrift file. To update it, the generate.py file can be used: python generate.py <TCLIServiceURL>. When left blank, the version for Hive 2.3 will be downloaded.

Metadata

Release files for PyHive6si 0.0.5.dev0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for PyHive6si 0.0.5.dev0
File Size Uploaded
PyHive6si-0.0.5.dev0.tar.gz 45.9 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for PyHive6si 0.0.5.dev0
File Interpreter ABI Platform
PyHive6si-0.0.5.dev0-py3-none-any.whl Python 3 none any Details

Total release size: 100.0 kB

Release files / PyHive6si-0.0.5.dev0.tar.gz

Download URL PyHive6si-0.0.5.dev0.tar.gz
Size 45.9 kB
Tags Source
SHA-256 checksum
How to use checksums
fce827b0bcdf07bd04042e17ef2060b417e91349a977d0d9f17766cae710a7d9
BLAKE2b-256 checksum
How to use checksums
28e818f9e349b3065a074258815dc4f21a4297433aa27aec3b32e65b7c6e4dcf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.4

Release files / PyHive6si-0.0.5.dev0-py3-none-any.whl

Download URL PyHive6si-0.0.5.dev0-py3-none-any.whl
Size 54.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
040f36290942dbfa951acf98cd886e07e3fb646116652e98227cbc6b947f07c3
BLAKE2b-256 checksum
How to use checksums
54879f69d3cb142fc404f3ba525a97b75b30b09ccc2faa10d904677766e6b48b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.4

Release history Release notifications | RSS feed

This release

0.0.5.dev0 This release

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page