Skip to main content
Join the official 2020 Python Developers SurveyStart the survey!

FTP in front of AWS S3, powered by asyncio and aiohttp

Project description

aioftps3 CircleCI Maintainability Test Coverage

FTP in front of AWS S3, using asyncio, and aiohttp. Only a subset of the FTP protocol is supported, with implicit TLS and PASV mode; connections will fail otherwise.

Installation

pip install aioftps3

An SSL key and certificate must be present $HOME/ssl.key and $HOME/ssl.crt respectively. To create a self-signed certificate, you can use openssl.

openssl req -new -newkey rsa:2048 -days 3650 -nodes -x509 -subj /CN=selfsigned \
    -keyout $HOME/ssl.key \
    -out $HOME/ssl.crt

Running

python -m aioftps3.server_main

Configuration

Configuration is through environment variables

Varaiable Description Example
AWS_AUTH_MECHANISM How requests to AWS are authenticated. Can be secret_access_key or ecs_role. If ecs_role it is expected that the server runs in an ECS container. secret_access_key
AWS_ACCESS_KEY_ID The ID of the AWS access key, if AWS_AUTH_MECHANISM is secret_access_key. ommitted
AWS_SECRET_ACCESS_KEY The secret part of the AWS access key, if AWS_AUTH_MECHANISM is secret_access_key ommitted
AWS_S3_BUCKET_REGION The region of the S3 bucket that stores the files. eu-west-1
AWS_S3_BUCKET_HOST The hostname used to communicate with S3. s3-eu-west-1.amazonaws.com
AWS_S3_BUCKET_NAME The name of the bucket files are stored in. my-bucket-name
AWS_S3_BUCKET_DIR_SUFFIX The suffix of the keys created in order to simulate a directory. Must start with a forward slash, but does not need to be longer. /
FTP_USERS__i__LOGIN For i any integer, the username of an FTP user that can login. my-user
FTP_USERS__i__PASSWORD_HASHED For i any integer, the hash, as generated by create_password.py, of the password of an FTP user that can login, using the salt in FTP_USERS__i__PASSWORD_SALT ommitted
FTP_USERS__i__PASSWORD_SALT See FTP_USERS__i__PASSWORD_HASHED ommitted
FTP_COMMAND_PORT The port that the server listens on for command connections. 8021
FTP_DATA_PORTS_FIRST The first data port in the range for PASV mode data transfers. 4001
FTP_DATA_PORTS_COUNT The number of ports used after FTP_DATA_PORTS_FIRST. 30
FTP_DATA_CIDR_TO_DOMAINS__i__CIDR For i any integer, a CIDR range used to match the IP of incoming command connections. If a match is found, the IP of the corresponding domain or IP address in FTP_DATA_CIDR_TO_DOMAINS__i__DOMAIN is returned to the client in response to PASV mode requests. Some clients will respond to FTP_DATA_CIDR_TO_DOMAINS__i__DOMAIN being 0.0.0.0 by making PASV mode data connections to the same IP as the original command connection, but not all. 0.0.0.0/0
FTP_DATA_CIDR_TO_DOMAINS__i__DOMAIN See FTP_DATA_CIDR_TO_DOMAINS__i__CIDR. ftp.my-domain.com
HEALTHCHECK_PORT The port the server listens on for healthcheck requests, such as from an AWS network load balancer. 8022

Advanced usage

The code in aioftps3.server_main satisfies a very particular use case, which may not be useful to most. However, the bulk of the code can be used for other cases: you will have to write your own aioftps3.server_main-equivalent, using the functions aioftps3.server.on_client_connect and aioftps3.server_socket.server. For example, you could

  • Store credentials, appropriately hashed, differently, .e.g. in a database.
  • Have the credentials hashed differently.
  • Allow/deny PASV mode data connections based on some condition.

See the source of aioftps3.server_main for how these functions can be used.

Creating a password and salt

python ./create_password.py

Running tests

Certificates must be created, and Minio, which emulates S3 locally, must be started

./certificates-create.sh && ./minio-start.sh

and then to run the tests themselves.

./tests.sh

Features / Design / Limitations

  • Can upload files bigger than 2G: uses multipart upload under the hood.

  • Does not store uploading files in memory before uploading them to S3: i.e. it is effectively a streaming upload. However, it's not completely streaming: each part of multipart upload is stored in memory before it begins to transfer to S3, in order to be able to hash its content and determine its length.

  • For uploading files, hashes are computed incrementally as data comes in in order to not block the event loop just before uploads to S3.

  • As few dependencies as is reasonable: aiohttp and its dependencies. Boto 3 is not used.

  • May not behave well if upload to the server is faster than its upload to S3.

  • There is some locking to deal with the same files being operated on concurrently. However...

  • .... it does nothing to deal with eventual consistency of S3, and so some operations may appear to not have an immediate effect.

Building and running locally

docker build -t ftps-s3 . && \
docker run --rm -p 8021-8042:8021-8042 \
  -e AWS_AUTH_MECHANISM=secret_access_key \
  -e AWS_ACCESS_KEY_ID=ommitted \
  -e AWS_SECRET_ACCESS_KEY=ommitted \
  -e AWS_S3_BUCKET_REGION=eu-west-1 \
  -e AWS_S3_BUCKET_HOST=s3-eu-west-1.amazonaws.com \
  -e AWS_S3_BUCKET_NAME=my-bucket-name \
  -e AWS_S3_BUCKET_DIR_SUFFIX=/ \
  -e FTP_USERS__1__LOGIN=user \
  -e FTP_USERS__1__PASSWORD_HASHED=ommitted \
  -e FTP_USERS__1__PASSWORD_SALT=ommitted \
  -e FTP_COMMAND_PORT=8021 \
  -e FTP_DATA_PORTS_FIRST=4001 \
  -e FTP_DATA_PORTS_COUNT=2 \
  -e FTP_DATA_CIDR_TO_DOMAINS__1__CIDR=0.0.0.0/0 \
  -e FTP_DATA_CIDR_TO_DOMAINS__1__DOMAIN=0.0.0.0 \
  -e HEALTHCHECK_PORT=8022
  ftps-s3

Building and pushing to Quay

docker build -t ftps-s3 . && \
docker tag ftps-s3:latest quay.io/uktrade/ftps-s3:latest && \
docker push quay.io/uktrade/ftps-s3:latest

Building and pushing healthcheck application to Quay

docker build -t ftps-s3-healthcheck . -f Dockerfile-healthcheck && \
docker tag ftps-s3-healthcheck:latest quay.io/uktrade/ftps-s3-healthcheck:latest && \
docker push quay.io/uktrade/ftps-s3-healthcheck:latest

Building and pushing Minio, used for testing, to Quay

docker build -t ftps-s3-minio . -f Dockerfile-minio && \
docker tag ftps-s3-minio:latest quay.io/uktrade/ftps-s3-minio:latest && \
docker push quay.io/uktrade/ftps-s3-minio:latest

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for aioftps3, version 0.0.6
Filename, size File type Python version Upload date Hashes
Filename, size aioftps3-0.0.6-py3-none-any.whl (11.5 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size aioftps3-0.0.6.tar.gz (4.5 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page