Skip to main content

No project description provided

Project description

S3Path

Latest version S3Path CI

S3Path provide a Python convenient File-System/Path like interface for AWS S3 Service using boto3 S3 resource as a driver.

Like pathlib, but for S3 Buckets

AWS S3 is among the most popular cloud storage solutions. It’s object storage, is built to store and retrieve various amounts of data from anywhere.

Currently, Python developers use Boto3 as the default API to connect / put / get / list / delete files from S3.

S3Path blends Boto3’s ease of use and the familiarity of pathlib api.

Install:

From PyPI:

$ pip install s3path

From Conda:

$ conda install -c conda-forge s3path

Basic use:

The following example assumes an s3 bucket setup as specified bellow:

$ aws s3 ls s3://pypi-proxy/

2018-04-24 22:59:59        186 requests/index.html
2018-04-24 22:59:57     485015 requests/requests-2.9.1.tar.gz
2018-04-24 22:35:01      89112 boto3/boto3-1.4.1.tar.gz
2018-04-24 22:35:02        180 boto3/index.html
2018-04-24 22:35:19    3308919 botocore/botocore-1.4.93.tar.gz
2018-04-24 22:35:36        188 botocore/index.html

Importing the main class:

>>> from s3path import S3Path

Listing “subdirectories” - s3 keys can be split like file-system with a / in s3path we:

>>> bucket_path = S3Path('/pypi-proxy/')
>>> [path for path in bucket_path.iterdir() if path.is_dir()]
[S3Path('/pypi-proxy/requests/'),
 S3Path('/pypi-proxy/boto3/'),
 S3Path('/pypi-proxy/botocore/')]

Listing html source files in this “directory” tree:

>>> bucket_path = S3Path('/pypi-proxy/')
>>> list(bucket_path.glob('**/*.html'))
[S3Path('/pypi-proxy/requests/index.html'),
 S3Path('/pypi-proxy/boto3/index.html'),
 S3Path('/pypi-proxy/botocore/index.html')]

Navigating inside a “directory” tree:

>>> bucket_path = S3Path('/pypi-proxy/')
>>> boto3_package_path = bucket_path / 'boto3' / 'boto3-1.4.1.tar.gz'
>>> boto3_package_path
S3Path('/pypi-proxy/boto3/boto3-1.4.1.tar.gz')

Querying path properties:

>>> boto3_package_path = S3Path('/pypi-proxy/boto3/boto3-1.4.1.tar.gz')
>>> boto3_package_path.exists()
True
>>> boto3_package_path.is_dir()
False
>>> boto3_package_path.is_file()
True

Opening a “file” (s3 key):

>>> botocore_index_path = S3Path('/pypi-proxy/botocore/index.html')
>>> with botocore_index_path.open() as f:
>>>     print(f.read())
"""
<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <title>Package Index</title>
</head>
<body>
    <a href="botocore-1.4.93.tar.gz">botocore-1.4.93.tar.gz</a><br>
</body>
</html>
"""

Or Simply reading:

>>> botocore_index_path = S3Path('/pypi-proxy/botocore/index.html')
>>> botocore_index_path.read_text()
"""
<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <title>Package Index</title>
</head>
<body>
    <a href="botocore-1.4.93.tar.gz">botocore-1.4.93.tar.gz</a><br>
</body>
</html>
"""

Versioned S3 Objects:

s3path supports versioned objects for S3 buckets that have versioning enabled. VersionedS3Path is a subclass of S3Path that supports all of its features. The main difference is an additional required version_id keyword parameter in each of its constructor methods.

>>> from s3path import VersionedS3Path
>>> bucket, key, version_id = 'my-bucket', 'my-key', 'my-version-id'
>>> VersionedS3Path(f'/{bucket}/{key}', version_id=version_id)
VersionedS3Path('/my-bucket/my-key', version_id='my-version-id')
>>> VersionedS3Path.from_uri(f's3://{bucket}/{key}', version_id=version_id)
VersionedS3Path('/my-bucket/my-key', version_id='my-version-id')
>>> VersionedS3Path.from_bucket_key(bucket=bucket, key=key, version_id=version_id)
VersionedS3Path('/my-bucket/my-key', version_id='my-version-id')

New in version 0.5.0

Requirements:

  • Python >= 3.4

  • boto3

  • smart-open

Further Documentation:

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

s3path-0.6.4.tar.gz (36.7 kB view details)

Uploaded Source

Built Distribution

s3path-0.6.4-py3-none-any.whl (29.7 kB view details)

Uploaded Python 3

File details

Details for the file s3path-0.6.4.tar.gz.

File metadata

  • Download URL: s3path-0.6.4.tar.gz
  • Upload date:
  • Size: 36.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.9.23

File hashes

Hashes for s3path-0.6.4.tar.gz
Algorithm Hash digest
SHA256 1006cf38e17c5af031ed8808f8aa9bb7ced06fc8ce3ea92a7e3639290711e5d2
MD5 95a5f1f41b4870d8e810c26d63e3fd37
BLAKE2b-256 305cd7987e7f89edf0410afdb970336bcc2ecc79ff5554870532adb5e44a1363

See more details on using hashes here.

File details

Details for the file s3path-0.6.4-py3-none-any.whl.

File metadata

  • Download URL: s3path-0.6.4-py3-none-any.whl
  • Upload date:
  • Size: 29.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.9.23

File hashes

Hashes for s3path-0.6.4-py3-none-any.whl
Algorithm Hash digest
SHA256 689232a9df906a7c90e2a00c347c34c4c3dad25cde89b050dd6cf5b6f00c723e
MD5 f2a4e60bf90f6e4b0cb91fc195a13d2c
BLAKE2b-256 3bfb46b60121009ad4be9f38c94e0796f06e8de8bf6a8b73f6c1d8e039304a5e

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page