Skip to main content

S3 to SFTP

Simple application that listens on a SQS Queue and transfers files from S3 to a SFTP Server.

The authentication can be done via username/password or username/private key. Private key should be stored “as-is” in AWS Secrets Manager.

The container must have an environment variable SFTP_TARGET, that follows the format below. You can set an enviroment variable for each of the below, using the property name as environment variable name.

The value in environment variables overrides the values otherwise set.

Build

You can build the docker image with the following commands

docker build . -t s3-to-sftp

Alternatively, using docker-compose

docker-compose build

Deploy with ECS Compose-X

We highly recommend that you store the value for SFTP_TARGET in AWS Secrets Manager.

Install compose-x

python3 -m venv compose-x
source compose-x/bin/activate
pip install pip -U; pip install ecs-composex

Deploy

You might need to update the settings in the aws.yaml file to meet your environment settings. On start, the connection to the SFTP server is not established, so the deployment won’t fail because of that. However, it requires to have the SFTP_TARGET details setup in before it starts listening on the SQS queue jobs.

ecs-compose-x up -d templates -f docker-compose.yaml -f aws.yaml -p s3-to-sftp-testing

Monitoring

In AWS CloudWatch, you will see new metrics in a Namespace called S3ToSFTP which is going to have some metrics, here for statistics.

These metrics are published using EMF.

SFTP_TARGET format

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "id": "sftp-secret-format",
    "type": "object",
    "additionalProperties": false,
    "properties": {
        "host": {
            "type": "string",
            "format": "idn-hostname"
        },
        "port": {
            "type": "number",
            "minimum": 1,
            "maximum": 65535
        },
        "username": {
            "type": "string"
        },
        "password": {
            "type": "string"
        },
        "default_path": {
            "type": "string"
        },
        "private_key": {
            "type": "string"
        },
        "private_key_pass": {
            "type": "string"
        }
    },
    "required": [
        "host",
        "port",
        "username"
    ]
}

Metadata

Release files for s3-to-sftp 1.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for s3-to-sftp 1.0.1
File Size Uploaded
s3-to-sftp-1.0.1.tar.gz 22.2 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for s3-to-sftp 1.0.1
File Interpreter ABI Platform
s3_to_sftp-1.0.1-py3-none-any.whl Python 3 none any Details

Total release size: 56.9 kB

Release files / s3-to-sftp-1.0.1.tar.gz

Download URL s3-to-sftp-1.0.1.tar.gz
Size 22.2 kB
Tags Source
SHA-256 checksum
How to use checksums
02e31cf9ebe644cde7a21e519c26f9f1a15f979640e67f49fd2d5b282ff9176c
BLAKE2b-256 checksum
How to use checksums
8b502b2008ee651328bde3c842c1f4bf185c6364e1dc3571aec91544fe89f9f4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.1.14 CPython/3.10.5 Linux/5.18.11-100.fc35.x86_64

Release files / s3_to_sftp-1.0.1-py3-none-any.whl

Download URL s3_to_sftp-1.0.1-py3-none-any.whl
Size 34.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f312324f89e1051a394af813fde3ef7867c89a6050f0a661e2621a028802d1a2
BLAKE2b-256 checksum
How to use checksums
be6f4d23399ac0447b178b60fff509e10969ceaaf02fce0c5432a9fe7f4aa16b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.1.14 CPython/3.10.5 Linux/5.18.11-100.fc35.x86_64

Release history Release notifications | RSS feed

This release

1.0.1 This release

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page