fsxpathlib 1.0.0

Objective Oriented Interface for AWS FSx, similar to boto3 APIs for s3.

What is it? fsxpathlib is the Python package that provides the Pythonic Objective Orirented programming (OOP) interface to allow developers to handle files and folders on Amazon FSx (for Windows File Server) storage services WITHOUT mounting Amazon FSx to local or on-line Windows machines. Amazon FSx for Windows File Server provides fully managed Microsoft Windows file servers, backed by a fully native Windows file system. With file storage on Amazon FSx and this Python package, the existing code, applications, and tools that Windows developers and administrators use today can be more easily migrated onto AWS. Also, the developers are more flexible to expand their code and applications for their business use cases with this Python package by removing the constraint of mounting Amazon FSx service.

Why do you need it? There is NO (OOP) Python library to allow developers to work on file level operations on Amazon FSx without mounting. The boto3 FSx client APIs only provides APIs to handle storage-volume-level operations (e.g., storage volume creation, volume deletion, etc.) but not file-level ones. This Python package is able to provide the APIs to do file or folder copy, read, delete, open, change, save, etc. on Amazon FSx service regardless those APIs are running in Linux, Mac or Windows environments. Moreover, it can handle file transfer between Amazon FSx and Amazon S3 storage services as well.

How does it work? This library requires smbprotocol library to be available as low-level APIs. The following shows highlighted attributes and methods avilable on this Python package:

Highlighted Attributes:

• abspath: return absolute path as a string for a given FSx path

• dirpath: return the directory info from a given FSx path

• dirname: return parent directory name for a given FSx path

• basename: return file name with extension (not included directories) for a given FSx path. If it is a directory, then return empty string.

• fname: return the file name without extension for a given FSx path

• ext: return the file extension for a given FSx path. If it is a directory, then return empty string.

• size_for_human: return human readble size info for a given FSx path

Highlighted Methonds:

• copy_to(): copy an existing FSx file or directory to a target location, which can be a FSx path, a local file or a S3 object or directory

• copy_from(): copy and save to a non-existing FSx file or directory from a source location, which can be a FSx path, a local file or a S3 objct or folder

• change(): change an existing FSx file or directory on path, name and extension

• exist(): return true if a FSx file or directory exists

• is_file(): return true if the given path is a file

• is_dir(): return true if the give path is a folder

• mkdir_if_not_exists(): create a new directory on FSx if it doesn’t exist

• read_bytes(): read content as bytes from an existing FSx file

• read_text(): read content as (utf-8) text from an existing FSx file

• remove_if_exists(): delete a file or an entire directory on FSx if it exists

• select_file(): return the list of of files for a given FSx path

• select_dir(): return the list of directories for a given FSx path

• select_by_ext(): return the list of files filtered by the extension for a given FSx path

• write_bytes(): save content as bytes into a FSx file

• write_text(): save content as (utf-8) text into a FSx file

Note: This library currently only supports operations within the same AWS region and doesn’t support any cross-region operations yet.

• Features

• Install

• Quick Start

• Dev Runbook

• Contributing

• Copyright

Features

APIs for manipulating files and folders on a FSx (for Windows File Server) storage volume.

Install

To install this library from a downloaded wheel file, you can do it as an example below:

$ pip install fsxpathlib-0.0.1-py2.py3-none-any.whl

Quick Start

Prerequisites

• Please follow the tutorial link to setup a sample FSx (for Windows File Server) storage volume if you don’t have any. Locate and remember the File System ID on the console like the screen shot below. Also, please remember user name, password and domain that you have established during FSx service setup.

• Prepare an AWS account and an IAM user with AdministratorAccess permission. For sign up to AWS, please refer to this link. For how to create an admin IAM user, please refer to this AWS on-line document.

• Install and setup AWS Command Line Interface (CLI) at your local environment. For how to install and setup AWS CLI, please refer to this on-line document.

• Install and setup Python 3 virtual environment.

Run Sample Code

First, try out the following examples for path manipulating functions:

# import
>>> from fsxpathlib.path import FsxPath

# Establish a FSx path. For example:
>>> p = FsxPath("server", "database", "table", "file.json")
# Then display the path info:
>>> print(f"Absolute path is: {p.abspath}")
Absolute path is: server\database\table\file.json
>>> print(p.dirpath)
server\database\table
>>> print(p.dirname)
table
>>> print(p.basename)
file.json
>>> print(p.fname)
file
>>> print(p.ext)
.json

# Let us now try some changes on the path:
>>> p1 = p.change(new_ext=".txt")
>>> print(p1.ext)
.txt
>>> print(p1.abspath)
server\database\table\file.txt
>>> p1 = p.change(new_fname="hello")
>>> print(p1.abspath)
server\database\table\hello.json

Second, establish a connection to the FSx storage service that you just setup by following the tutorial link. The codes below show examples to transfer data among FSx, S3 and local storage. When you try yours, please remember to replace the fsx_file_system_id, username, and password variables with your own values. Also, make sure you have pathlib_mate, s3pathlib and fsxpathlib libraries installed.

# import python libraries
>>> import getpass
>>> from s3pathlib import S3Path
>>> from fsxpathlib import FSxClient
>>> from fsxpathlib import FsxPath
>>> from pathlib_mate import Path
>>> import os

# Establish a connection session to the FSx service.
# Replace fsx_filesystemid, username and password variables with your own values.
>>> password = getpass.getpass(prompt='What is the password for accessing FSx?')
>>> fsx = FSxClient(fsx_file_system_id='fs-054a31b0ff86de2b0',
        ad_username='admin',
        ad_password=password)
What is the password for accessing FSx?········
>>> if fsx:
    print("Connect to the FSx server successfully")
Connect to the FSx server successfully
>>> fsx_sess = fsx.create_session()

Then, try out the following examples to copy a text file among FSx, S3 and local storage:

# create a FSx path including server, folders and file name
>>> p = FsxPath(fsx.server, "share", "test", "input.csv")
# display the absolute path
>>> print(p.abspath)
amznfsx3sh9aujr.corp.fsxvpc.com\share\test\input.csv
# remove the file if it exists on FSx.
>>> p.remove_if_exists()

# check if a file exists or not
>>> p.exists()
False
# create another FSx path
>>> p = FsxPath(fsx.server, "share", "test", "new.csv")

# upload a file from local to a FSx server
>>> p.copy_from(Path('./input.csv'))
copy from /Users/cheyaohu/WorkDocs/Open_Source/fsxpathlib-project/examples/input.csv to amznfsx3sh9aujr.corp.fsxvpc.com\share\test\new.csv
  done
True

# copy a file from FSx into a local folder
>>> p.copy_to(Path('./new.csv'))
copy from amznfsx3sh9aujr.corp.fsxvpc.com\share\test\new.csv to /Users/cheyaohu/WorkDocs/Open_Source/fsxpathlib-project/examples/new.csv
True
# check if the local file exists
>>> file_exists = os.path.exists('new.csv')
>>> print(file_exists)
True

# print out the content of a file in a FSx folder
>>> print(p.read_text())
name, id
ivan, 1
chen, 2

# copy a file from FSx into S3 bucket
>>> s3path_prefix = S3Path("fsx-cheyaohu", "demo", "input.csv")
>>> p.copy_to(s3path_prefix)
copy from amznfsx3sh9aujr.corp.fsxvpc.com\share\test\new.csv to s3://fsx-cheyaohu/demo/input.csv
  done
True

Dev Runbook

1. Setup Virtualenv:

# Create a Python virtual environment for dev / test
$ virtualenv -p python3.8 venv

# Enter virtualenv
$ source ./venv/bin/activate

# pip install this library and dependencies
$ pip install -e .

2. Run Tests:

# pip install test dependencies
# NOTE YOU MAY NEED TO RE-ENTER virtualenv
$ pip install -r requirements-test.txt

# run unit test and code coverage test
$ pytest tests -s --cov=fsxpathlib --cov-report term-missing --cov-report "annotate:fsxpathlib/.coverage.annotate"

3. Package and Publish:

# pip install development dependencies
# NOTE YOU MAY NEED TO RE-ENTER virtualenv
$ pip install -r requirements-dev.txt

# build artifacts locally
$ bash ./bin/build.sh

# publish to https://pypi.org
$ bash ./bin/publish.sh

1. Then create a release branch release/x.y.z that match the version.

2. Tag the repo from this branch using naming convention x.y.z.

3. Create a GitHub Release that name matching this version using naming convention x.y.z, and upload the .whl file to the Release.

Contributing

Please see the Contribution Guidelines.

Copyright

fsxpathlib is an open source project. See the license file for more information.