awswrangler 1.6.1

Pandas on AWS.

AWS Data Wrangler

Pandas on AWS

Source	Downloads	Page	Installation Command
PyPi		Link	pip install awswrangler
Conda		Link	conda install -c conda-forge awswrangler

Quick Start

Install the Wrangler with: pip install awswrangler

import awswrangler as wr
import pandas as pd

df = pd.DataFrame({"id": [1, 2], "value": ["foo", "boo"]})

# Storing data on Data Lake
wr.s3.to_parquet(
    df=df,
    path="s3://bucket/dataset/",
    dataset=True,
    database="my_db",
    table="my_table"
)

# Retrieving the data directly from Amazon S3
df = wr.s3.read_parquet("s3://bucket/dataset/", dataset=True)

# Retrieving the data from Amazon Athena
df = wr.athena.read_sql_query("SELECT * FROM my_table", database="my_db")

# Get Redshift connection (SQLAlchemy) from Glue and retrieving data from Redshift Spectrum
engine = wr.catalog.get_engine("my-redshift-connection")
df = wr.db.read_sql_query("SELECT * FROM external_schema.my_table", con=engine)

# Creating QuickSight Data Source and Dataset to reflect our new table
wr.quicksight.create_athena_data_source("athena-source", allowed_to_manage=["username"])
wr.quicksight.create_athena_dataset(
    name="my-dataset",
    database="my_db",
    table="my_table",
    data_source_name="athena-source",
    allowed_to_manage=["username"]
)

# Get MySQL connection (SQLAlchemy) from Glue Catalog and LOAD the data into MySQL
engine = wr.catalog.get_engine("my-mysql-connection")
wr.db.to_sql(df, engine, schema="test", name="my_table")

# Get PostgreSQL connection (SQLAlchemy) from Glue Catalog and LOAD the data into PostgreSQL
engine = wr.catalog.get_engine("my-postgresql-connection")
wr.db.to_sql(df, engine, schema="test", name="my_table")

Read The Docs

• What is AWS Data Wrangler?

• Install

  • PyPi (pip)
  • Conda
  • AWS Lambda Layer
  • AWS Glue Wheel
  • Amazon SageMaker Notebook
  • Amazon SageMaker Notebook Lifecycle
  • EMR
  • From source

• Tutorials

  • 001 - Introduction
  • 002 - Sessions
  • 003 - Amazon S3
  • 004 - Parquet Datasets
  • 005 - Glue Catalog
  • 006 - Amazon Athena
  • 007 - Databases (Redshift, MySQL and PostgreSQL)
  • 008 - Redshift - Copy & Unload.ipynb
  • 009 - Redshift - Append, Overwrite and Upsert
  • 010 - Parquet Crawler
  • 011 - CSV Datasets
  • 012 - CSV Crawler
  • 013 - Merging Datasets on S3
  • 014 - Schema Evolution
  • 015 - EMR
  • 016 - EMR & Docker
  • 017 - Partition Projection
  • 018 - QuickSight
  • 019 - Athena Cache

• API Reference

  • Amazon S3
  • AWS Glue Catalog
  • Amazon Athena
  • Databases (Amazon Redshift, PostgreSQL, MySQL)
  • Amazon EMR
  • Amazon CloudWatch Logs
  • Amazon QuickSight
  • AWS STS

• License

• Contributing

• Legacy Docs (pre-1.0.0)