Skip to main content

Smart dispatch between pathlib.Path and s3path.S3Path. It was borrowed from the project by EMTF: emtf@megvii.com

Project description

smart_path

Dispatch to pathlib.Path if a local file system path is given, or s3path.S3Path if the given path is prefixed with s3://

Example

Basic

from smart_path import smart_path
from pathlib import Path
from s3path import S3Path


# A local path on file system
path_local = smart_path("/tmp/t")
assert isinstance(path_local, Path)


# A s3 path on oss
path_brainpp_oss = smart_path("s3://public-datasets-contrib/", endpoint_url='s3.amazonaws.com')
assert isinstance(path_local, S3Path)

Stub Mode

from smart_path import smart_path


# Stub mode works like the symbolic link in Unix. The source path can be either
# on local file system or brain++ oss, so do the target path. Stub Mode is
# excepted to be helpful when we need to store a large file on oss and pretend 
# it is on local file system.
#
# In stub mode, smart_path writes to two destinations:
# 1. The string "hello-world" is written to `s3://somewhere/on/oss`
# 2. The path "s3://somewhere/on/oss" is written to `corresponding/local/path`
path = smart_path('s3://somewhere/on/oss', stub='corresponding/local/path')
path.write_text('hello-world')
# smart_path knows it is a stub file
stub = smart_path('corresponding/local/path')
print(stub.read_text())  # prints "hello-world"

Installation

pip install -e git://github.com/wanzysky/smart_path.git#egg=smart_path

Notice:

The entire project is borrowed from EMTF: EMTF-at-megvii-dot-com.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for smart-path, version 1.0.0
Filename, size File type Python version Upload date Hashes
Filename, size smart_path-1.0.0-py3-none-any.whl (3.7 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size smart_path-1.0.0.tar.gz (3.3 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page