Skip to main content

Smart dispatch between pathlib.Path and s3path.S3Path. It was borrowed from the project by EMTF: emtf@megvii.com

Project description

smart_path

Dispatch to pathlib.Path if a local file system path is given, or s3path.S3Path if the given path is prefixed with s3://

Example

Basic

from smart_path import smart_path
from pathlib import Path
from s3path import S3Path


# A local path on file system
path_local = smart_path("/tmp/t")
assert isinstance(path_local, Path)


# A s3 path on oss
path_brainpp_oss = smart_path("s3://public-datasets-contrib/", endpoint_url='s3.amazonaws.com')
assert isinstance(path_local, S3Path)

Stub Mode

from smart_path import smart_path


# Stub mode works like the symbolic link in Unix. The source path can be either
# on local file system or brain++ oss, so do the target path. Stub Mode is
# excepted to be helpful when we need to store a large file on oss and pretend 
# it is on local file system.
#
# In stub mode, smart_path writes to two destinations:
# 1. The string "hello-world" is written to `s3://somewhere/on/oss`
# 2. The path "s3://somewhere/on/oss" is written to `corresponding/local/path`
path = smart_path('s3://somewhere/on/oss', stub='corresponding/local/path')
path.write_text('hello-world')
# smart_path knows it is a stub file
stub = smart_path('corresponding/local/path')
print(stub.read_text())  # prints "hello-world"

Installation

pip install -e git://github.com/wanzysky/smart_path.git#egg=smart_path
# Or from PyPI
pip install smart_path

Notice:

The entire project is borrowed from EMTF: EMTF-at-megvii-dot-com.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

smart_path-1.0.1.tar.gz (3.4 kB view hashes)

Uploaded source

Built Distribution

smart_path-1.0.1-py3-none-any.whl (3.7 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page