Skip to main content
Documentation Status https://travis-ci.org/MacHu-GWU/s3iotools-project.svg?branch=master https://codecov.io/gh/MacHu-GWU/s3iotools-project/branch/master/graph/badge.svg https://img.shields.io/pypi/v/s3iotools.svg https://img.shields.io/pypi/l/s3iotools.svg https://img.shields.io/pypi/pyversions/s3iotools.svg https://img.shields.io/badge/STAR_Me_on_GitHub!--None.svg?style=social
https://img.shields.io/badge/Link-Document-blue.svg https://img.shields.io/badge/Link-API-blue.svg https://img.shields.io/badge/Link-Source_Code-blue.svg https://img.shields.io/badge/Link-Install-blue.svg https://img.shields.io/badge/Link-GitHub-blue.svg https://img.shields.io/badge/Link-Submit_Issue-blue.svg https://img.shields.io/badge/Link-Request_Feature-blue.svg https://img.shields.io/badge/Link-Download-blue.svg

Welcome to s3iotools Documentation

Usage

Copy local file to s3 and download file object from s3 to local is easy:

from s3iotools import S3FileObject

s3obj = S3FileObject(bucket="my-bucket", key="hello.txt", path="hello.txt")

# get started, now we don't have file either on local or on s3
if s3obj.path_obj.exists():
    s3obj.path_obj.remove()
assert s3obj.exists_on_local() is False
assert s3obj.exists_on_s3() is False

s3obj.path_obj.write_text("hello world", encoding="utf-8)
assert s3obj.exists_on_local() is True

s3obj.copy_to_s3()
assert s3obj.exists_on_s3() is True

s3obj.path_obj.remove()
assert s3obj.exists_on_local() is False

s3obj.copy_to_local()
assert s3obj.exists_on_local() is True

You can manipulate s3 backed pandas.DataFrame easily:

import boto3
import pandas as pd
from s3iotools import S3Dataframe

session = boto3.Session(profile_name="xxx")
s3 = session.resource("s3")
bucket_name = "my-bucket"
s3df = S3Dataframe(s3_resource=s3, bucket_name=bucket_name)
s3df.df = pd.DataFrame(...)

s3df.to_csv(key="data.csv")
s3df.to_csv(key="data.csv.gz", gzip_compressed=True)

s3df_new = S3Dataframe(s3_resource=s3, bucket_name=bucket_name, key="data.csv")
s3df_new.read_csv()
s3df_new.df # access data

s3df_new = S3Dataframe(s3_resource=s3, bucket_name=bucket_name, key="data.csv.gz")
s3df_new.read_csv(gzip_compressed=True)
s3df_new.df # access data

json IO is similar.

s3df = S3Dataframe(s3_resource=s3, bucket_name=bucket_name)
s3df.df = pd.DataFrame(...)
s3df.to_json(key="data.json.gz", gzip_compressed=True)
s3df_new = S3Dataframe(s3_resource=s3, bucket_name=bucket_name, key="data.json.gz")
s3df_new.read_json(gzip_compressed=True)
s3df_new.df # access data

parquet is a columnar storage format, which is very efficient for OLAP query. You can just put data on S3, then use AWS Athena to query parquet files. parquet IO in s3iotools is easy:

s3df = S3Dataframe(s3_resource=s3, bucket_name=bucket_name)
s3df.df = pd.DataFrame(...)
s3df.to_parquet(key="data.parquet", compression="gzip")
s3df_new = S3Dataframe(s3_resource=s3, bucket_name=bucket_name, key="data.parquet")
s3df_new.read_parquet()
s3df_new.df # access data

s3iotools doesn’t automatically install pyarrow, you can install it with pip install pyarrow.

Install

s3iotools is released on PyPI, so all you need is:

$ pip install s3iotools

To upgrade to latest version:

$ pip install --upgrade s3iotools

Metadata

Release files for s3iotools 0.0.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for s3iotools 0.0.3
File Size Uploaded Provenance Actions
s3iotools-0.0.3.tar.gz 27.0 kB Without provenance Details for s3iotools-0.0.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for s3iotools 0.0.3
File Interpreter ABI Platform Actions
s3iotools-0.0.3-py2.py3-none-any.whl Python 3, Python 2 none any Details for s3iotools-0.0.3-py2.py3-none-any.whl

Total release size: 61.6 kB

Release files / s3iotools-0.0.3.tar.gz

Download URL s3iotools-0.0.3.tar.gz
Size 27.0 kB
Tags Source
SHA-256 checksum
How to use checksums
689ba157b39fadc44e1884344474a57662d80e005238274d184f10b56af6019b
BLAKE2b-256 checksum
How to use checksums
f414f954cae8a80ce3708e53ffce76af826f2651aba55ef3c48e7947055152a6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.1 CPython/2.7.13

Release files / s3iotools-0.0.3-py2.py3-none-any.whl

Download URL s3iotools-0.0.3-py2.py3-none-any.whl
Size 34.5 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
35f606a3284835dc1f13353d582d287be6a1fe6f322dc869cd074a0d4236e62d
BLAKE2b-256 checksum
How to use checksums
e24756a8722cfb95009ff6b33cf10eb064005bedf898c49a32edd983ea1ebdb9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.1 CPython/2.7.13

Security for s3iotools 0.0.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page