Utilities for Pandas and Apache Spark on AWS.
Project description
AWS Data Wrangler (BETA)
Utilities for Pandas and Apache Spark on AWS
AWS Data Wrangler aims to fill a gap between AWS Analytics Services (Glue, Athena, EMR, Redshift, S3) and the most popular Python data libraries (Pandas, Apache Spark).
Contents: Use Cases | Installation | Examples | License
Use Cases
- Pandas Dataframe -> Parquet (S3)
- Pandas Dataframe -> CSV (S3)
- Pandas Dataframe -> Glue Catalog
- Pandas Dataframe -> Redshift
- Pandas Dataframe -> Athena
- CSV (S3) -> Pandas Dataframe
- Athena -> Pandas Dataframe
- Spark Dataframe -> Redshift
Installation
pip install awswrangler
AWS Data Wrangler runs only Python 3.6 and beyond. And runs on AWS Lambda, AWS Glue, EC2, on-premises, local, etc.
Examples
Writing Pandas Dataframe to Data Lake
session = awswrangler.Session()
session.pandas.to_parquet(
dataframe=dataframe,
database="database",
path="s3://...",
partition_cols=["col_name"],
)
If a Glue Database name is passed, all the metadata will be created in the Glue Catalog. If not, only the s3 data write will be done.
Reading from Data Lake to Pandas Dataframe
session = awswrangler.Session()
dataframe = session.pandas.read_sql_athena(
sql="select * from table",
database="database"
)
Reading from S3 file to Pandas Dataframe
session = awswrangler.Session()
dataframe = session.pandas.read_csv(path="s3://...")
Typical Pandas ETL
import pandas
import awswrangler
df = pandas.read_... # Read from anywhere
# Typical Pandas, Numpy or Pyarrow transformation HERE!
session = awswrangler.Session()
session.pandas.to_parquet( # Storing the data and metadata to Data Lake
dataframe=dataframe,
database="database",
path="s3://...",
partition_cols=["col_name"],
)
Loading Spark Dataframe to Redshift
session = awswrangler.Session(spark_session=spark)
session.spark.to_redshift(
dataframe=df,
path="s3://...",
connection=conn,
schema="public",
table="table",
iam_role="IAM_ROLE_ARN",
mode="append",
)
License
This library is licensed under the Apache 2.0 License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for awswrangler-0.0b15-py36,py37-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3dc74b6a341603e048427faa0d15e6d0677a99d1450ddb55e4f657fd50b34437 |
|
MD5 | df9e8e2b488ea2a5ae32f0df4acd4f67 |
|
BLAKE2b-256 | 5b92dff40a6d8226fed881838f9ab42c5990154df1c27a81a62795921da68eed |