Utility belt to handle data on AWS.
Project description
AWS Data Wrangler (beta)
Utility belt to handle data on AWS.
Contents: Use Cases | Installation | Examples
Use Cases
- Pandas -> Parquet (S3)
- Pandas -> CSV (S3)
- Pandas -> Glue Catalog
- Pandas -> Athena
- Pandas -> Redshift
- CSV (S3) -> Pandas
- Athena -> Pandas
- PySpark -> Redshift
Installation
pip install awswrangler
Runs only with Python 3.6 and beyond.
Runs anywhere (AWS Lambda, AWS Glue, EMR, EC2, on-premises, local, etc).
P.S. Lambda Layer bundle and Glue egg are available to download. It's just upload to your account and run! :rocket:
Examples
Writing Pandas Dataframe to S3 + Glue Catalog
session = awswrangler.Session()
session.pandas.to_parquet(
dataframe=dataframe,
database="database",
path="s3://...",
partition_cols=["col_name"],
)
If a Glue Database name is passed, all the metadata will be created in the Glue Catalog. If not, only the s3 data write will be done.
Reading from AWS Athena to Pandas
session = awswrangler.Session()
dataframe = session.pandas.read_sql_athena(
sql="select * from table",
database="database"
)
Reading from S3 (CSV) to Pandas
session = awswrangler.Session()
dataframe = session.pandas.read_csv(path="s3://...")
Typical Pandas ETL
import pandas
import awswrangler
df = pandas.read_... # Read from anywhere
# Typical Pandas, Numpy or Pyarrow transformation HERE!
session = awswrangler.Session()
session.pandas.to_parquet( # Storing the data and metadata to Data Lake
dataframe=dataframe,
database="database",
path="s3://...",
partition_cols=["col_name"],
)
Loading Pyspark Dataframe to Redshift
session = awswrangler.Session(spark_session=spark)
session.spark.to_redshift(
dataframe=df,
path="s3://...",
connection=conn,
schema="public",
table="table",
iam_role="IAM_ROLE_ARN",
mode="append",
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
awswrangler-0.0b19.tar.gz
(19.9 kB
view hashes)
Built Distribution
Close
Hashes for awswrangler-0.0b19-py36,py37-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a179048f0406fa0ba14fe413b5030f4ab6208583b1e2cc158daa872d46494a1b |
|
MD5 | 900390fc6fd9fd223f4f6ed7aa3a254d |
|
BLAKE2b-256 | 174ceaafa3431dacf607b2313d11461b95a12e9944522e94460e7cd045a1bacc |