aioftps3 0.0.1

FTP in front of AWS S3, powered by asyncio, aioftp, and aiohttp

aioftps3

FTP in front of AWS S3, using asyncio, aioftp and aiohttp.

Features / Design / Limitations

• Can upload files bigger than 2G: uses multipart upload under the hood.

• Does not store uploading files in memory before uploading them to S3: i.e. it is effectively a streaming upload. However, it's not completely streaming: each part of multipart upload is stored in memory before it begins to transfer to S3, in order to be able to hash its content and determine its length.

• For uploading files, hashes are computed incrementally as data comes in in order to not block the event loop just before uploads to S3.

• As few dependencies as is reasonable: aioftp, aiohttp, and their dependencies. Boto 3 is not used.

• May not behave well if upload to the server is faster than its upload to S3.

• It is likely not robust if concurrent users change the same files or directories.

• Does nothing to deal with eventual consistency of S3, and so some operations may appear to not have an immediate effect.

Building and running locally

docker build -t ftps-s3 . && \
docker run --rm -p 8021-8042:8021-8042 \
  -e AWS_ACCESS_KEY_ID=ommitted \
  -e AWS_SECRET_ACCESS_KEY=ommitted \
  -e AWS_S3_BUCKET_REGION=eu-west-1 \
  -e AWS_S3_BUCKET_HOST=s3-eu-west-1.amazon.aws.com \
  -e AWS_S3_BUCKET_NAME=my-bucket-name
  ftps-s3

Building and pushing to Quay

docker build -t ftps-s3 . && \
docker tag ftps-s3:latest quay.io/uktrade/ftps-s3:latest && \
docker push quay.io/uktrade/ftps-s3:latest