Skip to main content

Simplify use of AWS resources in your code with aws-python-utils

Project description


Simplify use of AWS resources in your code with aws-python-utils


pip install aws-python-util

SecretManager Util

  • Easy secret retrieval

Use ENVIRONMENT variable "AWS_SECRET_MANAGER_SECRET" or pass in secret_key name.

Example Usage

Via ENVIRONMENT variable

from aws_python_utils import secretmanager
import os

os.environ["AWS_SECRET_MANAGER_SECRET"] = "mysecret"
secretmanager = secretmanager.AwsSecretManager()
mysecret = secretmanager.get_secret()

Or pass in secret_key name ..

from aws_python_utils import secretmanager
import os

secretmanager = secretmanager.AwsSecretManager()
mysecret = secretmanager.get_secret(secret_key="mysecret")

S3 Util

  • Streams an s3 object directly into a pandas DataFrame to avoid writing to disk and then loading from disk
  • Uploads a DataFrame directly to s3
Example Usage
from aws_python_utils import s3
from io import BytesIO
import pandas as pd
import numpy as np

bucket,key = s3.AwsS3.get_bucket_and_key_from_s3_path("s3://my-bucket/mypath/to/object")

print("bucket = " + bucket)  # my-bucket
print("key = " + key)        # mypath/to/object

# download a tab separated file schema: id    val1  val2
df = s3.AwsS3.download_s3_file(s3_path, header=0, sep='\t', index='id')

df2 = pd.DataFrame(np.random.randint(low=0, high=10, size=(5, 5)), columns=['a', 'b', 'c', 'd', 'e'])
io_buffer = BytesIO()
df2.to_csv(io_buffer, columns=['a', 'c', 'e'], sep='\t', index=False)

s3.AwsS3.upload_to_s3("s3://your-bucket/path/to/object.tsv", io_buffer)

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
aws_python_utils-0.0.6.tar.gz (5.2 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page