Add S3 support to dtool
Project description
Free software: MIT License
Features
Copy datasets to and from S3 object storage
List all the datasets in a S3 bucket
Create datasets directly in S3
Installation
To install the dtool-S3 package:
pip install dtool-s3
Configuration
Install the aws client, for details see https://docs.aws.amazon.com/cli/latest/userguide/installing.html. In short:
pip install awscli --upgrade --user
Configure the credentials using:
aws configure
These are needed for the boto3 library, for more details see https://boto3.readthedocs.io/en/latest/guide/quickstart.html.
Usage
To copy a dataset from local disk (my-dataset) to a S3 bucket (/data_raw) one can use the command below:
dtool copy ./my-dataset s3://data_raw
To list all the datasets in a S3 bucket one can use the command below:
dtool ls s3://data_raw
See the dtool documentation for more detail.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
