Skip to main content

A package for low- and high-level high-bandwidth asynchronous data transfer

Project description

pyRemoteData

pyRemoteData is a module developed for scientific computation using the remote storage platform ERDA (Electronic Research Data Archive) provided by Aarhus University IT, as part of my PhD at the Department of Ecoscience at Aarhus University.

It can be used with any passwordless SSH-enabled storage facility that supports SFTP and LFTP. But is only tested on a minimal SFTP server found at atmoz/sftp and on the live AU ERDA service which runs on MiG (Minimum intrusion Grid - SourceForge/GitHub) developed by SCIENCE HPC Centre at Copenhagen University.

If your facility requires a password, it should be very easy to modify the code to support this, in fact it is already implemented, but not exposed to the user. Merely change line 76 in src/remote_data/implicit_mount.py to fetch the password from the environment variable of your choice, or simply hardcode it. However, do this at your own risk, as I have not assessed the security implications.

Capabilities

In order to facility high-throughput computation in a cross-platform setting, pyRemoteData handles data transfer with multithreading and asynchronous data streaming using thread-safe buffers.

Use-cases

If your storage facility supports SFTP and LFTP, and you need high-bandwidth data streaming for analysis, data migration or other purposes such as model-training, then this module may be of use to you. Experience with SFTP or LFTP is not necessary, but you must be able to setup the required SSH configurations.

Setup

A more user-friendly setup process, which facilitates both automated as well as interactive setup is currently in development. (TODO: Finish and describe the setup process)

Installation

The package is available on PyPI, and can be installed using pip:

pip install pyremotedata

Interactive

Simply follow the popup instructions that appear once you load the package for the first time.

Automated

The automatic configuration setup relies on setting the correct environment variables BEFORE LOADING THE PACKAGE:

  • PYREMOTEDATA_REMOTE_USERNAME : Should be set to your username on your remote service.
  • PYREMOTEDATA_REMOTE_URI : Should be set to the URI of the endpoint for your remote service (e.g. for ERDA it is "io.erda.au.dk").
  • PYREMOTEDATA_REMOTE_DIRECTORY : If you would like to set a default working directory, that is not the root of your remote storage, then set this to that (e.g. "/MY_PROJECT/DATASETS") otherwise simply set this to "/".
  • PYREMOTEDATA_AUTO : Should be set to "yes" to disable interactive mode. If this is not set, or set to anything other than "yes" (not case-sensitive), while any of the prior environment variables are unset an error will be thrown.

Example

If you want to test against a mock server simply follow the instructions in tests/README.

If you have a remote storage facility that supports SFTP and LFTP, then you can use the following example to test the functionality of the module:

# Set the environment variables (only necessary in a non-interactive setting)
# If you are simply running this as a Python script, 
# you can omit these lines and you will be prompted to set them interactively
import os
os.environ["PYREMOTEDATA_REMOTE_USERNAME"] = "username"
os.environ["PYREMOTEDATA_REMOTE_URI"] = "storage.example.com"
os.environ["PYREMOTEDATA_REMOTE_DIRECTORY"] = "/MY_PROJECT/DATASETS"
os.environ["PYREMOTEDATA_AUTO"] = "yes"

from pyremotedata.implicit_mount import IOHandler

handler = IOHandler()

with handler as io:
    print(io.ls())

# The configuration is persistent, but can be removed using the following:
from pyremotedata.config import remove_config
remove_config()

Issues

This module is certainly not maximally efficient, and you may run into network- or OS-specific issues. Any and all feedback and contributions is highly appreciated.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyremotedata-0.0.36.tar.gz (933.0 kB view details)

Uploaded Source

Built Distribution

pyremotedata-0.0.36-py3-none-any.whl (27.8 kB view details)

Uploaded Python 3

File details

Details for the file pyremotedata-0.0.36.tar.gz.

File metadata

  • Download URL: pyremotedata-0.0.36.tar.gz
  • Upload date:
  • Size: 933.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.15

File hashes

Hashes for pyremotedata-0.0.36.tar.gz
Algorithm Hash digest
SHA256 3863c1b77c02a0206777e8e3490ee83f9c2d38fdb68b4a5d00ed02015ccc1d31
MD5 87c4749311f16325a6d5927edc10f73d
BLAKE2b-256 9dcdd79f39ab650118c04832d3811fcf6e74f4c9b3d3f634a903afcb837a5d69

See more details on using hashes here.

File details

Details for the file pyremotedata-0.0.36-py3-none-any.whl.

File metadata

File hashes

Hashes for pyremotedata-0.0.36-py3-none-any.whl
Algorithm Hash digest
SHA256 ed286e29e6e89899ae687be74c7f8549ebf2429dbe266e71996ae5d7aa56770e
MD5 3841f9e8a7d6c5aef7c2b8cf07ccabcc
BLAKE2b-256 ce8cde5e331e2d6cbc0cd9dafed57ab02862f8dd48143d7bbd2af735230a3157

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page