Amazon EFS (amazon-efs) allows programmatically manipulate EFS data (create, read, delete, list files) from any machine.
Project description
Amazon EFS (amazon-efs)
Amazon EFS (amazon-efs) allows programmatically manipulate EFS data (create, read, delete, list files) from any machine.
Prerequisites
- python
- pip
- boto3
- AWS Account
- AWS Credentials
Install
pip install amazon-efs
Warning
EFS should have at least one mount target in a Private subnet
Limits
Lambda compute env
list_files, upload, download, delete actions are limited by 15 minutes execution time (AWS Lambda works under the hood)
Batch compute env
list_files, upload, download actions are not implemented yet
Basics
Supported compute environments:
- Lambda (Default)
- Batch
Lambda compute environment (by default):
efs = Efs('<file_system_id>')
Lambda compute environment:
efs = Efs('<file_system_id>', compute_env_name='lambda')
Batch compute environment:
The "batch_queue" option is required |
---|
efs = Efs('<file_system_id>', {
'batch_queue': '<batch_queue>',
}, compute_env_name='batch')
Lambda compute environment
This computing environment is used for lightweight operations (lasting no more than 15 minutes).
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
# Deploying required underlying resources
efs.init()
# Actions (e.g. list_files, upload, download, delete)
files_list = efs.list_files()
# Don't forget to destroy underlying resources at the end of the session
efs.destroy()
Actions
List files
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
efs.init()
files_list = efs.list_files()
print(files_list)
files_list = efs.list_files('dir1')
print(files_list)
files_list = efs.list_files('dir1/dir2')
print(files_list)
efs.destroy()
Upload
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
efs.init()
efs.upload('file.txt')
efs.upload('file.txt', 'dir1/new_file.txt')
efs.upload('file.txt', 'dir1/dir2/new_file.txt')
efs.upload('file.txt', 'dir1/dir3/new_file.txt')
efs.upload('file.txt', 'dir2/dir3/new_file.txt')
efs.upload('file.txt', 'dir2/dir4/new_file.txt')
efs.destroy()
Download
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
efs.init()
efs.download('dir1/dir3/new_file.txt', 'file1.txt')
efs.destroy()
Delete
Delete file
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
efs.init()
efs.delete('dir2/dir3/new_file.txt')
efs.destroy()
Delete folder
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
efs.init()
efs.delete('dir1/dir2/*')
efs.delete('dir1/*')
efs.destroy()
Async delete
Use it if you want to schedule deletion and monitor progress yourself.
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
state = efs.init()
http_response_status_code = efs.delete('dir2/dir3/new_file.txt')
Then, after the job is completed, destroy the infrastructure.
efs = Efs(efs_id, { 'state': state })
efs.destroy()
Batch compute environment
This computing environment is used for heavy operations (lasting more than 15 minutes).
Actions
Delete
The "batch_queue" option is required |
---|
Delete file
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
batch_queue = '<batch_queue>'
efs = Efs(efs_id, {
'batch_queue': batch_queue,
}, compute_env_name='batch')
efs.init()
efs.delete('dir2/dir3/new_file.txt')
efs.destroy()
Delete folder
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
batch_queue = '<batch_queue>'
efs = Efs(efs_id, {
'batch_queue': batch_queue,
}, compute_env_name='batch')
efs.init()
efs.delete('dir1/dir2/*')
efs.delete('dir1/*')
efs.destroy()
Async delete
Use it if you want to schedule deletion and monitor progress yourself.
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
batch_queue = '<batch_queue>'
efs = Efs(efs_id, {
'batch_queue': batch_queue,
}, compute_env_name='batch')
state = efs.init()
batch_job_arn = efs.delete('dir2/dir3/new_file.txt')
Then, after the job is completed, destroy the infrastructure.
efs = Efs(efs_id, { 'state': state }, compute_env_name='batch')
efs.destroy()
State
You can destroy underlying infrastructure even after destroying EFS object from RAM if you saved the state
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
state = efs.init()
# Destroy object
del efs
efs = Efs(efs_id, { 'state': state })
files_list = efs.list_files()
print(files_list)
efs.destroy()
Tags
You can add custom tags to underlying resources
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id, {
'tags': {
'k1': 'v1',
'k2': 'v2'
}
})
efs.init()
files_list = efs.list_files()
print(files_list)
efs.destroy()
Logging
from amazon_efs import Efs
import logging
fs_id = 'fs-0d74736bfc*******'
logger = logging.getLogger()
logging.basicConfig(level=logging.ERROR, format='%(asctime)s: %(levelname)s: %(message)s')
efs = Efs(fs_id, logger=logger)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for amazon_efs-0.4.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3afde11da85e40f7dfd297d09df8f3aa9f5c4c9d6146bafff257087f30281acc |
|
MD5 | 2d6feec836e647c0337bab838a90db38 |
|
BLAKE2b-256 | b38b25e8458f5ff34559e7cb943c474204276de48e0cff4024bb7e291638f876 |