S3 to SFTP
Simple application that listens on a SQS Queue and transfers files from S3 to a SFTP Server.
The authentication can be done via username/password or username/private key. Private key should be stored “as-is” in AWS Secrets Manager.
The container must have an environment variable SFTP_TARGET, that follows the format below. You can set an enviroment variable for each of the below, using the property name as environment variable name.
The value in environment variables overrides the values otherwise set.
Build
You can build the docker image with the following commands
docker build . -t s3-to-sftp
Alternatively, using docker-compose
docker-compose build
Deploy with ECS Compose-X
We highly recommend that you store the value for SFTP_TARGET in AWS Secrets Manager.
Install compose-x
python3 -m venv compose-x
source compose-x/bin/activate
pip install pip -U; pip install ecs-composex
Deploy
You might need to update the settings in the aws.yaml file to meet your environment settings. On start, the connection to the SFTP server is not established, so the deployment won’t fail because of that. However, it requires to have the SFTP_TARGET details setup in before it starts listening on the SQS queue jobs.
ecs-compose-x up -d templates -f docker-compose.yaml -f aws.yaml -p s3-to-sftp-testing
Monitoring
In AWS CloudWatch, you will see new metrics in a Namespace called S3ToSFTP which is going to have some metrics, here for statistics.
These metrics are published using EMF.
SFTP_TARGET format
{
"$schema": "http://json-schema.org/draft-07/schema#",
"id": "sftp-secret-format",
"type": "object",
"additionalProperties": false,
"properties": {
"host": {
"type": "string",
"format": "idn-hostname"
},
"port": {
"type": "number",
"minimum": 1,
"maximum": 65535
},
"username": {
"type": "string"
},
"password": {
"type": "string"
},
"default_path": {
"type": "string"
},
"private_key": {
"type": "string"
},
"private_key_pass": {
"type": "string"
}
},
"required": [
"host",
"port",
"username"
]
}
Metadata
Release files for s3-to-sftp 1.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| s3-to-sftp-1.0.1.tar.gz | 22.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| s3_to_sftp-1.0.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 56.9 kB
Release files / s3-to-sftp-1.0.1.tar.gz
| Download URL | s3-to-sftp-1.0.1.tar.gz |
|---|---|
| Size | 22.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
02e31cf9ebe644cde7a21e519c26f9f1a15f979640e67f49fd2d5b282ff9176c
|
|
BLAKE2b-256 checksum How to use checksums |
8b502b2008ee651328bde3c842c1f4bf185c6364e1dc3571aec91544fe89f9f4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.14 CPython/3.10.5 Linux/5.18.11-100.fc35.x86_64
|
Release files / s3_to_sftp-1.0.1-py3-none-any.whl
| Download URL | s3_to_sftp-1.0.1-py3-none-any.whl |
|---|---|
| Size | 34.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f312324f89e1051a394af813fde3ef7867c89a6050f0a661e2621a028802d1a2
|
|
BLAKE2b-256 checksum How to use checksums |
be6f4d23399ac0447b178b60fff509e10969ceaaf02fce0c5432a9fe7f4aa16b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.14 CPython/3.10.5 Linux/5.18.11-100.fc35.x86_64
|