Skip to main content

Python DB API 2.0 (PEP 249) compliant wrapper for Amazon Athena JDBC driver

Project description

https://img.shields.io/pypi/pyversions/PyAthenaJDBC.svg https://circleci.com/gh/laughingman7743/PyAthenaJDBC.svg?style=shield https://codecov.io/gh/laughingman7743/PyAthenaJDBC/branch/master/graph/badge.svg https://img.shields.io/pypi/l/PyAthenaJDBC.svg

PyAthenaJDBC

PyAthenaJDBC is a python DB API 2.0 (PEP 249) compliant wrapper for Amazon Athena JDBC driver.

Requirements

  • Python

    • CPython 2.6, 2,7, 3,4, 3.5

  • Java

    • Java 7+

Installation

$ pip install PyAthenaJDBC

Usage

Basic usage:

from pyathenajdbc import connect

conn = connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/')
try:
    with conn.cursor() as cursor:
        cursor.execute("""
        SELECT * FROM test_pyathena_jdbc.one_row
        """)
        print(cursor.description)
        print(cursor.fetchall())
finally:
    conn.close()

Cursor iteration:

from pyathenajdbc import connect

conn = connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/')
try:
    with conn.cursor() as cursor:
        cursor.execute("""
        SELECT * FROM test_pyathena_jdbc.many_rows LIMIT 10
        """)
        for row in cursor:
            print(row)
finally:
    conn.close()

Query with parameter:

from pyathenajdbc import connect

conn = connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/')
try:
    with conn.cursor() as cursor:
        cursor.execute("""
        SELECT col_int FROM test_pyathena_jdbc.one_row_complex where col_int = {0}
        """, 2147483647)
        print(cursor.fetchall())

        cursor.execute("""
        SELECT col_string FROM test_pyathena_jdbc.one_row_complex where col_string = {param}
        """, param='a string')
        print(cursor.fetchall())
finally:
    conn.close()

Minimal example for Pandas DataFrame:

from pyathenajdbc import connect
import pandas as pd

conn = connect(access_key=<access key>,
               secret_key=<secret key>,
               s3_staging_dir=<staging dir>,
               region_name=<region name>,
               jvm_path=<jvm path>) # optional, as used by jpype
df = pd.read_sql("SELECT * FROM <table name> LIMIT 10", conn)

As Pandas DataFrame:

import contextlib
from pyathenajdbc import connect
from pyathenajdbc.util import as_pandas

with contextlib.closing(
        connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/')) as conn:
    with conn.cursor() as cursor:
        cursor.execute("""
        SELECT * FROM test_pyathena_jdbc.many_rows
        """)
        df = as_pandas(cursor)
print(df.describe())

Credential

Support AWS CLI credentials configuration.

Credential Files

~/.aws/credentials

[default]
aws_access_key_id=YOUR_ACCESS_KEY_ID
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY

~/.aws/config

[default]
region=us-west-2
output=json

Environment variables

$ export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID
$ export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
$ export AWS_DEFAULT_REGION=us-west-2

Additional environment variable:

$ export AWS_ATHENA_S3_STAGING_DIR=s3://YOUR_S3_BUCKET/path/to/

Testing

Depends on the following environment variables:

$ export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID
$ export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
$ export AWS_DEFAULT_REGION=us-west-2
$ export AWS_ATHENA_S3_STAGING_DIR=s3://YOUR_S3_BUCKET/path/to/

Run test:

$ pip install pytest
$ py.test

Run test multiple Python versions:

$ pip install tox
$ pyenv local 2.6.9 2.7.12 3.4.5 3.5.2
$ tox

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

PyAthenaJDBC-1.0.3.tar.gz (10.5 kB view hashes)

Uploaded Source

Built Distribution

PyAthenaJDBC-1.0.3-py2.py3-none-any.whl (8.5 MB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page