A robust file-system mount to a remote storage.
Project description
RMount - Robust Remote Mount
This is a robust remote mount wrapper around the mount utility rclone. The python-side implementation makes it possible to integrate rclone into your python application with pre-packaged rclone binaries (version v1.62.2). This is a ready-to-go solution without external dependencies for mounting a local directory to a remote storage provider such as AWS S3. RMount is robust to time-outs, connection drops, while it abstracts the details of integrating directly with the remote provider. NOTE Currently only supports Linux.
Philosophy of the library:
- Monitor
- Restart when possible
- Exit Gracefully
- Fail Loudly
Storage Systems currently supported:
- Google Cloud Storage via S3.
- AWS via S3.
- Remote SSH, i.e. for your own private server with SSH access
- S3 remote file-systems are supported by several cloud providers listed below.
System Requirements
mountpoint
command should be in PATHfusermount
command should be in PATH- System support for FUSE file system
The above requirements are by default met on most Linux distributions such as Ubuntu.
Install
pip install rmount
mountpoint
command must be accessible and in Path. e.g. runningmountpoint .
should return. is not a mountpoint
or. is a mountpoint
Usage
You will first need to define your configuration object and then you can use RemoteMount
with a context manager i.e. with
or simply by calling .mount()
and .unmount()
. See below and example.
AWS S3 Config
from rmount import S3
config = S3(
provider="AWS",
region="us-east-1",
secret_access_key="xxx",
access_key_id="xxx",
)
GCS S3 Config
config = S3(
provider="GCS",
secret_access_key="xxx",
access_key_id="xxx",
endpoint="https://storage.googleapis.com",
)
SSH Remote Config
from pathlib import Path
from rmount import Remote
config = Remote(
host="localhost",
user="root",
port=22,
key_file=Path.home() / ".ssh" / "id_rsa",
)
Writing a file
# local directory
local_path = Path("/tmp/s3")
# remote directory e.g. s3://rmount gs://rmount or /rmount
remote_path = "rmount"
mount = RemoteMount(config, remote_path, local_path)
with mount:
local_path.joinpath("foo").write_text("bar")
Developer guide
Full details HERE
$ pip install -e .[dev]
$ make test
Currently there is no support for Mac or Windows as they require multiple additional steps to install mount. It is not viable for the main developer of this project to support these systems. If you are interested in writing code to support additional OS. You should find a way to replace the command line utility depedencies listed above.
mountpoint
command checks whether a directory is a mount point and fusermount
command unmounts a directory. As long as robust alternatives can be found and packaged in this repo or documentation provided for them it should be sufficient for the same library to work on any OS. Additionally, the OS must support FUSE filesystem, e.g. WinFsp or macFUSE.
S3 Remote Storage Providers
In theory, RMount support all the providers supported by rclone, but you will need to implement your own configuration object. We have only tested with AWS S3 but in theory it should work with all providers:
- AWS S3
- Alibaba Cloud (Aliyun) Object Storage System (OSS)
- Ceph
- China Mobile Ecloud Elastic Object Storage (EOS)
- Cloudflare R2
- Arvan Cloud Object Storage (AOS)
- DigitalOcean Spaces
- Dreamhost
- GCS
- Huawei OBS
- IBM COS S3
- IDrive e2
- IONOS Cloud
- Liara Object Storage
- Minio
- Petabox
- Qiniu Cloud Object Storage (Kodo)
- RackCorp Object Storage
- Scaleway
- Seagate Lyve Cloud
- SeaweedFS
- StackPath
- Storj
- Tencent Cloud Object Storage (COS)
- Wasabi
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for rmount-0.0.2-py3-none-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6f8020be90eba24a1d160164da61d268a06702ec925f648a5f1a01cd13d84750 |
|
MD5 | 5c718f87ea039ff28b342a12585ddf06 |
|
BLAKE2b-256 | 5138455306668bd26d5a28ae558afe2e6c6de3df41880e09242dd195db63c82b |