Skip to main content

Pandas on AWS.

Project description

AWS Data Wrangler

Pandas on AWS

Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

AWS Data Wrangler

An AWS Professional Service open source initiative | aws-proserve-opensource@amazon.com

Release Python Version Code style: black License

Checked with mypy Coverage Static Checking Documentation Status

Source Downloads Installation Command
PyPi PyPI Downloads pip install awswrangler
Conda Conda Downloads conda install -c conda-forge awswrangler

⚠️ For platforms without PyArrow 3 support (e.g. EMR, Glue PySpark Job, MWAA):
➡️ pip install pyarrow==2 awswrangler

Powered By

Table of contents

Quick Start

Installation command: pip install awswrangler

⚠️ For platforms without PyArrow 3 support (e.g. EMR, Glue PySpark Job, MWAA):
➡️pip install pyarrow==2 awswrangler

import awswrangler as wr
import pandas as pd
from datetime import datetime

df = pd.DataFrame({"id": [1, 2], "value": ["foo", "boo"]})

# Storing data on Data Lake
wr.s3.to_parquet(
    df=df,
    path="s3://bucket/dataset/",
    dataset=True,
    database="my_db",
    table="my_table"
)

# Retrieving the data directly from Amazon S3
df = wr.s3.read_parquet("s3://bucket/dataset/", dataset=True)

# Retrieving the data from Amazon Athena
df = wr.athena.read_sql_query("SELECT * FROM my_table", database="my_db")

# Get a Redshift connection from Glue Catalog and retrieving data from Redshift Spectrum
con = wr.redshift.connect("my-glue-connection")
df = wr.redshift.read_sql_query("SELECT * FROM external_schema.my_table", con=con)
con.close()

# Amazon Timestream Write
df = pd.DataFrame({
    "time": [datetime.now(), datetime.now()],   
    "my_dimension": ["foo", "boo"],
    "measure": [1.0, 1.1],
})
rejected_records = wr.timestream.write(df,
    database="sampleDB",
    table="sampleTable",
    time_col="time",
    measure_col="measure",
    dimensions_cols=["my_dimension"],
)

# Amazon Timestream Query
wr.timestream.query("""
SELECT time, measure_value::double, my_dimension
FROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3
""")

Read The Docs

Getting Help

The best way to interact with our team is through GitHub. You can open an issue and choose from one of our templates for bug reports, feature requests... You may also find help on these community resources:

Community Resources

Please send a Pull Request with your resource reference and @githubhandle.

Logging

Enabling internal logging examples:

import logging
logging.basicConfig(level=logging.INFO, format="[%(name)s][%(funcName)s] %(message)s")
logging.getLogger("awswrangler").setLevel(logging.DEBUG)
logging.getLogger("botocore.credentials").setLevel(logging.CRITICAL)

Into AWS lambda:

import logging
logging.getLogger("awswrangler").setLevel(logging.DEBUG)

Who uses AWS Data Wrangler?

Knowing which companies are using this library is important to help prioritize the project internally.

Please send a Pull Request with your company name and @githubhandle if you may.

What is Amazon SageMaker Data Wrangler?

Amazon SageMaker Data Wrangler is a new SageMaker Studio feature that has a similar name but has a different purpose than the AWS Data Wrangler open source project.

  • AWS Data Wrangler is open source, runs anywhere, and is focused on code.

  • Amazon SageMaker Data Wrangler is specific for the SageMaker Studio environment and is focused on a visual interface.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

awswrangler-2.10.0.tar.gz (148.7 kB view details)

Uploaded Source

Built Distributions

awswrangler-2.10.0-py3.6.egg (403.1 kB view details)

Uploaded Egg

awswrangler-2.10.0-py3-none-any.whl (184.8 kB view details)

Uploaded Python 3

File details

Details for the file awswrangler-2.10.0.tar.gz.

File metadata

  • Download URL: awswrangler-2.10.0.tar.gz
  • Upload date:
  • Size: 148.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.1 CPython/3.9.0

File hashes

Hashes for awswrangler-2.10.0.tar.gz
Algorithm Hash digest
SHA256 e2737cdf291a26a8647753bbe99f1c91f643c009b6ca108a384f796f788e014f
MD5 8d4cde1a1a9d83cd452b45714fbf5d33
BLAKE2b-256 d70055befc2e2be3b04286379d878e16fdd0c22db0028d6f06ba76cfb6e3cccb

See more details on using hashes here.

File details

Details for the file awswrangler-2.10.0-py3.6.egg.

File metadata

  • Download URL: awswrangler-2.10.0-py3.6.egg
  • Upload date:
  • Size: 403.1 kB
  • Tags: Egg
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.1 CPython/3.9.0

File hashes

Hashes for awswrangler-2.10.0-py3.6.egg
Algorithm Hash digest
SHA256 35283f15411df50210361f7c4c612099d8bc7750ff2dc673584af7e55d6ce9ea
MD5 04b25cc465cf0513e4a69462dfe8d4d2
BLAKE2b-256 6649ed23e66451a628303cef05a79af50def28f068edc5616c487cf01ecee925

See more details on using hashes here.

File details

Details for the file awswrangler-2.10.0-py3-none-any.whl.

File metadata

  • Download URL: awswrangler-2.10.0-py3-none-any.whl
  • Upload date:
  • Size: 184.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.1 CPython/3.9.0

File hashes

Hashes for awswrangler-2.10.0-py3-none-any.whl
Algorithm Hash digest
SHA256 d88295ec6a4c6d6e9b5519d65a9ccdd883aa590482dc76a7912eb62dcde7cad5
MD5 f8f9e71f868eb470eec22d1aa1e574a2
BLAKE2b-256 ce3ff1647ec73b94a1ea9efeff6cec50e590cf0a33e0978fb5a606816029360f

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page