Sample python script to work with Amazon S3.
Project description
s3-client - Example python script to work with S3
Installation
pip install s3-client
Authorization
Authorization is performed using environment variables:
-
AWS_ACCESS_KEY_ID - Specifies an AWS access key
-
AWS_SECRET_ACCESS_KEY - Specifies the secret key associated with the access key. This is essentially the "password" for the access key.
Usage:
$ s3-client
usage: s3-client [-h] [-d] [-e ENDPOINT] [-r REGION_NAME] {listbuckets,listobj,deleteobj,metadataobj,upload,download} ...
S3 Client sample script
optional arguments:
-h, --help show this help message and exit
-d, --debug debug flag
-e ENDPOINT, --endpoint ENDPOINT
S3 endpoint URL
-r REGION_NAME, --region REGION_NAME
S3 Region Name
Commands:
{listbuckets,listobj,deleteobj,metadataobj,upload,download}
listbuckets List all buckets
listobj List objects in a bucket
deleteobj Delete object in a bucket
metadataobj List object metadata
upload Upload files to bucket
download Download files from bucket
Example of use:
s3-client listbuckets
s3-client -r us-east-1 listbuckets
s3-client -e https://s3.amazonaws.com listobj my_bucket -t
s3-client -e https://s3.amazonaws.com upload my_bucket -f file1
s3-client -e https://s3.amazonaws.com upload my_bucket -d mydir
Example:
List buckets
$ s3-client -e https://s3.amazonaws.com listbuckets
Bucket_Name: test-script1 Creation_Date: 2019-03-22 19:40:36.379000+00:00 versioning_status: None
Bucket_Name: test-script2 Creation_Date: 2019-03-22 19:50:35.706000+00:00 versioning_status: Enabled
Upload file(s)
$ s3-client -e https://s3.amazonaws.com upload my_bucket -f my_file.csv
Uploading file my_file.csv with object name my_file.csv
data transferred: 100%|█████████████████████████████████████████████████████████████| 8.39M/8.39M [00:16<00:00, 520kB/s]
- Elapsed time 16.1471 seconds
- Upload completed successfully
$ s3-client -e https://s3.amazonaws.com upload my_bucket -d mydir/ --nopbar
Uploading file mydir/test2 with object name mydir/test2
- Elapsed time 0.1007 seconds
- Upload completed successfully
Uploading file mydir/test1 with object name mydir/test1
- Elapsed time 0.1183 seconds
- Upload completed successfully
Uploading file mydir/internal/test3 with object name mydir/internal/test3
- Elapsed time 0.1277 seconds
- Upload completed successfully
Uploading file mydir/internal/deep/test4 with object name mydir/internal/deep/test4
- Elapsed time 0.1114 seconds
- Upload completed successfully
Uploading file mydir/internal/deep/test5 with object name mydir/internal/deep/test5
- Elapsed time 0.0683 seconds
- Upload completed successfully
List objects in a bucket
$ s3-client listobj -h
usage: s3-client listobj [-h] [-l LIMIT] [-t] [-p PREFIX] [-v] bucket
positional arguments:
bucket Bucket Name
optional arguments:
-h, --help show this help message and exit
-l LIMIT, --limit LIMIT
Limit the number of objects returned
-t, --table Show output as table
-p PREFIX, --prefix PREFIX
Only objects with specific prefix
-v, --versions Show all object versions
$ s3-client -e https://s3.amazonaws.com listobj my_bucket
key: mydir/internal/deep/test4 size: 10 storage_class: STANDARD e_tag: "d41d8cd98f00b204e9800998ecf8427e" last_modified: 2020-08-21 16:40:42.790000+00:00
key: mydir/internal/deep/test5 size: 203 storage_class: STANDARD e_tag: "7c41d8cd98f00b204e9800998ecf8427" last_modified: 2020-08-21 16:40:42.894000+00:00
key: mydir/internal/test3 size: 42 storage_class: STANDARD e_tag: "9acbdfd98f00b204e9100998ecf8423a" last_modified: 2020-08-21 16:40:42.655000+00:00
key: mydir/test1 size: 1031 storage_class: STANDARD e_tag: "8acffca98f00b204e98a0821ecf8447e" last_modified: 2020-08-21 16:40:42.538000+00:00
key: mydir/test2 size: 121 storage_class: STANDARD e_tag: "d3ab64d98f00b20401800998ecf8438b" last_modified: 2020-08-21 16:40:42.429000+00:00
$ s3-client -e https://s3.amazonaws.com listobj my_bucket -t
| key | size | storage_class | e_tag | last_modified |
|---------------------------|--------|-----------------|------------------------------------|----------------------------------|
| mydir/internal/deep/test4 | 10 | STANDARD | "d41d8cd98f00b204e9800998ecf8427e" | 2020-08-21 16:40:42.790000+00:00 |
| mydir/internal/deep/test5 | 203 | STANDARD | "7c41d8cd98f00b204e9800998ecf8427" | 2020-08-21 16:40:42.894000+00:00 |
| mydir/internal/test3 | 42 | STANDARD | "9acbdfd98f00b204e9100998ecf8423a" | 2020-08-21 16:40:42.655000+00:00 |
| mydir/test1 | 1031 | STANDARD | "8acffca98f00b204e98a0821ecf8447e" | 2020-08-21 16:40:42.538000+00:00 |
| mydir/test2 | 121 | STANDARD | "d3ab64d98f00b20401800998ecf8438b" | 2020-08-21 16:40:42.429000+00:00 |
Download objects
$ s3-client -e https://s3.amazonaws.com download -h
usage: s3-client download [-h] [--nopbar] [-l LOCALDIR] [-o] [-v VERSIONID] (-f FILENAME | -p PREFIX) bucket
positional arguments:
bucket Bucket Name
optional arguments:
-h, --help show this help message and exit
--nopbar Disable progress bar
-l LOCALDIR, --localdir LOCALDIR
Local directory to save downloaded file. Default current directory
-o, --overwrite Overwrite local destination file if it exists. Default false
-v VERSIONID, --versionid VERSIONID
Object version id
-f FILENAME, --file FILENAME
Download a specific file
-p PREFIX, --prefix PREFIX
Download recursively all files with a prefix.
$ s3-client -e https://s3.amazonaws.com download my_bucket -f mydir/test1
Error: File ./mydir/test1 exist. Remove it from local drive to download.
$ s3-client -e https://s3.amazonaws.com download my_bucket -f mydir/test1 --overwrite
Downloading object mydir/test1 to path ./mydir/test1
data transferred: 100%|███████████████████████████████████████████████| 5.24M/5.24M [00:11<00:00, 468kB/s]
- Elapsed time 11.3103 seconds
- Download completed successfully
$ s3-client -e https://s3.amazonaws.com download my_bucket -f mydir/test1 -l /tmp/
Downloading object mydir/test1 to path /tmp/mydir/test1
data transferred: 100%|███████████████████████████████████████████████| 5.24M/5.24M [00:11<00:00, 468kB/s]
- Elapsed time 11.3103 seconds
- Download completed successfully
$ ls /tmp/mydir/test1
/tmp/mydir/test1
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
s3_client-0.1.2.tar.gz
(13.0 kB
view hashes)
Built Distribution
s3_client-0.1.2-py3-none-any.whl
(11.9 kB
view hashes)
Close
Hashes for s3_client-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 11d3cf9f8264ef81cbc2481a1f8cda5fb0edf5c7a89acda9096e3cf9d227b194 |
|
MD5 | 86107aa8865c04c3b226a2b491f500c2 |
|
BLAKE2b-256 | 0a4f2a92dc6b37dd29dcf4cc30c47d64b3364ec42d6ec5a3005c698d3d0894a9 |