AWS Data Wrangler
Pandas on AWS
Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
An AWS Professional Service open source initiative | aws-proserve-opensource@amazon.com
| Source | Downloads | Installation Command |
|---|---|---|
| PyPi |  |
pip install awswrangler |
| Conda |  |
conda install -c conda-forge awswrangler |
⚠️ For platforms without PyArrow 3 support (e.g. EMR, Glue PySpark Job, MWAA):
➡️pip install pyarrow==2 awswrangler
Powered By
Table of contents
- Quick Start
- Read The Docs
- Getting Help
- Community Resources
- Logging
- Who uses AWS Data Wrangler?
- What is Amazon Sagemaker Data Wrangler?
Quick Start
Installation command: pip install awswrangler
⚠️ For platforms without PyArrow 3 support (e.g. EMR, Glue PySpark Job, MWAA):
➡️pip install pyarrow==2 awswrangler
import awswrangler as wr
import pandas as pd
from datetime import datetime
df = pd.DataFrame({"id": [1, 2], "value": ["foo", "boo"]})
# Storing data on Data Lake
wr.s3.to_parquet(
df=df,
path="s3://bucket/dataset/",
dataset=True,
database="my_db",
table="my_table"
)
# Retrieving the data directly from Amazon S3
df = wr.s3.read_parquet("s3://bucket/dataset/", dataset=True)
# Retrieving the data from Amazon Athena
df = wr.athena.read_sql_query("SELECT * FROM my_table", database="my_db")
# Get a Redshift connection from Glue Catalog and retrieving data from Redshift Spectrum
con = wr.redshift.connect("my-glue-connection")
df = wr.redshift.read_sql_query("SELECT * FROM external_schema.my_table", con=con)
con.close()
# Amazon Timestream Write
df = pd.DataFrame({
"time": [datetime.now(), datetime.now()],
"my_dimension": ["foo", "boo"],
"measure": [1.0, 1.1],
})
rejected_records = wr.timestream.write(df,
database="sampleDB",
table="sampleTable",
time_col="time",
measure_col="measure",
dimensions_cols=["my_dimension"],
)
# Amazon Timestream Query
wr.timestream.query("""
SELECT time, measure_value::double, my_dimension
FROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3
""")
Read The Docs
- What is AWS Data Wrangler?
- Install
- Tutorials
- 001 - Introduction
- 002 - Sessions
- 003 - Amazon S3
- 004 - Parquet Datasets
- 005 - Glue Catalog
- 006 - Amazon Athena
- 007 - Databases (Redshift, MySQL, PostgreSQL and SQL Server)
- 008 - Redshift - Copy & Unload.ipynb
- 009 - Redshift - Append, Overwrite and Upsert
- 010 - Parquet Crawler
- 011 - CSV Datasets
- 012 - CSV Crawler
- 013 - Merging Datasets on S3
- 014 - Schema Evolution
- 015 - EMR
- 016 - EMR & Docker
- 017 - Partition Projection
- 018 - QuickSight
- 019 - Athena Cache
- 020 - Spark Table Interoperability
- 021 - Global Configurations
- 022 - Writing Partitions Concurrently
- 023 - Flexible Partitions Filter
- 024 - Athena Query Metadata
- 025 - Redshift - Loading Parquet files with Spectrum
- 026 - Amazon Timestream
- 027 - Amazon Timestream 2
- 028 - Amazon DynamoDB
- API Reference
- License
- Contributing
- Legacy Docs (pre-1.0.0)
Getting Help
The best way to interact with our team is through GitHub. You can open an issue and choose from one of our templates for bug reports, feature requests... You may also find help on these community resources:
- The #aws-data-wrangler Slack channel
- Ask a question on Stack Overflow
and tag it with
awswrangler
Community Resources
Please send a Pull Request with your resource reference and @githubhandle.
- Optimize Python ETL by extending Pandas with AWS Data Wrangler [@igorborgest]
- Reading Parquet Files With AWS Lambda [@anand086]
- Transform AWS CloudTrail data using AWS Data Wrangler [@anand086]
- Rename Glue Tables using AWS Data Wrangler [@anand086]
- Getting started on AWS Data Wrangler and Athena [@dheerajsharma21]
- Simplifying Pandas integration with AWS data related services [@bvsubhash]
- Build an ETL pipeline using AWS S3, Glue and Athena [@taupirho]
Logging
Enabling internal logging examples:
import logging
logging.basicConfig(level=logging.INFO, format="[%(name)s][%(funcName)s] %(message)s")
logging.getLogger("awswrangler").setLevel(logging.DEBUG)
logging.getLogger("botocore.credentials").setLevel(logging.CRITICAL)
Into AWS lambda:
import logging
logging.getLogger("awswrangler").setLevel(logging.DEBUG)
Who uses AWS Data Wrangler?
Knowing which companies are using this library is important to help prioritize the project internally.
Please send a Pull Request with your company name and @githubhandle if you may.
- Amazon
- AWS
- Cepsa [@alvaropc]
- Cognitivo [@msantino]
- Digio [@afonsomy]
- DNX [@DNXLabs]
- Funcional Health Tech [@webysther]
- Informa Markets [@mateusmorato]
- LINE TV [@bryanyang0528]
- Magnataur [@brianmingus2]
- M4U [@Thiago-Dantas]
- NBCUniversal [@vibe]
- nrd.io [@mrtns]
- OKRA Technologies [@JPFrancoia, @schot]
- Pier [@flaviomax]
- Pismo [@msantino]
- ringDNA [@msropp]
- Serasa Experian [@andre-marcos-perez]
- Shipwell [@zacharycarter]
- strongDM [@mrtns]
- Thinkbumblebee [@dheerajsharma21]
- Zillow [@nicholas-miles]
What is Amazon SageMaker Data Wrangler?
Amazon SageMaker Data Wrangler is a new SageMaker Studio feature that has a similar name but has a different purpose than the AWS Data Wrangler open source project.
-
AWS Data Wrangler is open source, runs anywhere, and is focused on code.
-
Amazon SageMaker Data Wrangler is specific for the SageMaker Studio environment and is focused on a visual interface.
3 maintainers
Release files for awswrangler 2.10.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| awswrangler-2.10.0.tar.gz | 148.7 kB | Details |
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| awswrangler-2.10.0-py3.6.egg | Legacy Egg format | - | - | Details |
| awswrangler-2.10.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 736.7 kB
Release files / awswrangler-2.10.0.tar.gz
| Download URL | awswrangler-2.10.0.tar.gz |
|---|---|
| Size | 148.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e2737cdf291a26a8647753bbe99f1c91f643c009b6ca108a384f796f788e014f
|
|
BLAKE2b-256 checksum How to use checksums |
d70055befc2e2be3b04286379d878e16fdd0c22db0028d6f06ba76cfb6e3cccb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.6.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.1 CPython/3.9.0
|
Release files / awswrangler-2.10.0-py3.6.egg
| Download URL | awswrangler-2.10.0-py3.6.egg |
|---|---|
| Size | 403.1 kB |
| Tags | Egg |
|
SHA-256 checksum How to use checksums |
35283f15411df50210361f7c4c612099d8bc7750ff2dc673584af7e55d6ce9ea
|
|
BLAKE2b-256 checksum How to use checksums |
6649ed23e66451a628303cef05a79af50def28f068edc5616c487cf01ecee925
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.6.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.1 CPython/3.9.0
|
Release files / awswrangler-2.10.0-py3-none-any.whl
| Download URL | awswrangler-2.10.0-py3-none-any.whl |
|---|---|
| Size | 184.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d88295ec6a4c6d6e9b5519d65a9ccdd883aa590482dc76a7912eb62dcde7cad5
|
|
BLAKE2b-256 checksum How to use checksums |
ce3ff1647ec73b94a1ea9efeff6cec50e590cf0a33e0978fb5a606816029360f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.6.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.1 CPython/3.9.0
|
