FTP in front of AWS S3, powered by asyncio, aioftp, and aiohttp
Project description
aioftps3
FTP in front of AWS S3, using asyncio, aioftp and aiohttp.
Features / Design / Limitations
-
Can upload files bigger than 2G: uses multipart upload under the hood.
-
Does not store uploading files in memory before uploading them to S3: i.e. it is effectively a streaming upload. However, it's not completely streaming: each part of multipart upload is stored in memory before it begins to transfer to S3, in order to be able to hash its content and determine its length.
-
For uploading files, hashes are computed incrementally as data comes in in order to not block the event loop just before uploads to S3.
-
As few dependencies as is reasonable: aioftp, aiohttp, and their dependencies. Boto 3 is not used.
-
May not behave well if upload to the server is faster than its upload to S3.
-
It is likely not robust if concurrent users change the same files or directories.
-
Does nothing to deal with eventual consistency of S3, and so some operations may appear to not have an immediate effect.
Building and running locally
docker build -t ftps-s3 . && \
docker run --rm -p 8021-8042:8021-8042 \
-e AWS_ACCESS_KEY_ID=ommitted \
-e AWS_SECRET_ACCESS_KEY=ommitted \
-e AWS_S3_BUCKET_REGION=eu-west-1 \
-e AWS_S3_BUCKET_HOST=s3-eu-west-1.amazon.aws.com \
-e AWS_S3_BUCKET_NAME=my-bucket-name
ftps-s3
Building and pushing to Quay
docker build -t ftps-s3 . && \
docker tag ftps-s3:latest quay.io/uktrade/ftps-s3:latest && \
docker push quay.io/uktrade/ftps-s3:latest
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.