Skip to main content

Command Line Client for Apache Airavata MFT data transfer software

Project description

Airavata Managed File Transfers (MFT)

Apache Airavata MFT is a high-performance, multi-protocol data transfer engine to orchestrate data movement and operations across most cloud and On-premises storages. MFT aims to abstract the complexity of heterogenous storages by providing a unified and simple interface for users to seamlessly access and move data across any storage endpoint. To accomplish this goal, MFT provides simple but highly-performing tools to access most cloud and on-premise storages as seamlessly as they access local files in their workstations.

Apache Airavata MFT bundles easily deployable agents that auto determine optimum network path with additional multi-channel, parallel data paths to optimize the transfer performance to gain the maximum throughput between storage endpoints. MFT utilizes parallel Agents to transfer data between endpoints to gain the advantage of multiple network links.

Try Airavata MFT

MFT requires Java 11+ and python3.10+ to install Airavata MFT in your environment. MFT currently supports Linux and MacOS operating systems. Contributions to support Windows are welcome!!.

Download and Install

Following commands will download Airavata MFT into your machine and start the MFT service.

pip3 install airavata-mft-cli
mft init

If the installer failed for M1 and M2 Macs complaining about grpcio installation. Follow the solution mentioned in here. You might have to uninstall already installed grpcio and grpcio-tools distributions first. For other common installation issues, please refer to the troubleshooting section.

To stop MFT after using

mft stop

Registering Storages

First you need to register your storage endpoints into MFT in order to access them. Registering storage is an interactive process and you can easily register those without prior knowledge

mft storage add

This will ask the type of storage you need and credentials to access those storages. To list already added storages, you can run

mft storage list

Accessing Data in Storages

In Airavata MFT, we provide a unified interface to access the data in any storage. Users can access data in storages just as they access data in their computers. MFT converts user queries into storage specific data representations (POSIX, Block, Objects, ..) internally

mft ls <storage name>
mft ls <storage name>/<resource path>

Moving Data between Storages

Copying data between storages are simple as copying data between directories of local machine for users. MFT takes care of network path optimizations, parallel data path selections and selections or creations of suitable transfer agents.

mft cp <source storage name>/<path> <destination storage name>/<path> 

MFT is capable of auto detecting directory copying and file copying based on the path given.

Troubleshooting and Issue Reporting

This is our very first attempt release Airavata MFT for community usage and there might be lots of corner cases that we have not noticed. All the logs of MFT service are available in ~/.mft/Standalone-Service-0.01/logs/airavata.log. If you see any error while using MFT, please report that in our Github issue page and we will respond as soon as possible. We really appreciate your contribution as it will greatly help to improve the stability of the product.

Common issues

  • Following error can be noticed if you have a python version which is less than 3.10
  ERROR: Could not find a version that satisfies the requirement airavata-mft-cli (from versions: none)
  ERROR: No matching distribution found for airavata-mft-cli

If the Error still occurs after installing the right python version, try creating a virtual environemnt

python3.10 -m venv venv
source venv/bin/activate
pip install airavata-mft-cli

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airavata_mft_cli-0.1.24.tar.gz (16.4 kB view details)

Uploaded Source

Built Distribution

airavata_mft_cli-0.1.24-py3-none-any.whl (27.0 kB view details)

Uploaded Python 3

File details

Details for the file airavata_mft_cli-0.1.24.tar.gz.

File metadata

  • Download URL: airavata_mft_cli-0.1.24.tar.gz
  • Upload date:
  • Size: 16.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.6.1 CPython/3.10.8 Darwin/23.3.0

File hashes

Hashes for airavata_mft_cli-0.1.24.tar.gz
Algorithm Hash digest
SHA256 f6d979f1928ade8ac83ce010a36c135b154fefca03c193f03e9d848a7a3265e6
MD5 adfcf1bb5d0cda1266a4b7ac4b6e2304
BLAKE2b-256 4aed6a629e76bdcb33a50c5061c44d8950247bd61b3a6f32dde6742edf454491

See more details on using hashes here.

File details

Details for the file airavata_mft_cli-0.1.24-py3-none-any.whl.

File metadata

File hashes

Hashes for airavata_mft_cli-0.1.24-py3-none-any.whl
Algorithm Hash digest
SHA256 0700c639da1921b6cd1e3cb00470957d8aaba802d8e51a20668d80799ce7f9b7
MD5 049ce4f4a995565e98161d9dffbc94a3
BLAKE2b-256 cc9dac469e57f9efe206c95e6d7aae5d0e17f00123ecf02fe60da8c876e5727f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page