sftp-to-s3 0.1.1

A robust and scalable solution to ingest files from SFTP to AWS S3.

---

Welcome to sftp_to_s3 Documentation

sftp_to_s3 is an Open Source solution to ingest large amount of data from SFTP server to AWS S3.

How it Work

This solution assume that the SFTP server folder structure has to have partition strategy. For example, the SFTP server folder structure is like this:

/data/2022-01-01/many-folder-many-files...
/data/2022-01-02/many-folder-many-files...
/data/2022-01-03/many-folder-many-files...
/data/...

Once a partition of files are created on SFTP server, it should not be changed to frequently. Files in each partition are grouped into many batches to ensure that the total amount of files and total file size has a reasonable boundary. A coordinator will be responsible to detect the delta of partitions on SFTP and AWS S3, and then group files into batches. Then the batch data will be sent to a cluster of worker to download in parallel if it fits the SFTP server’s network bandwidth.

The coordinator and worker can be deployed as either AWS Lambda function or AWS Batch job. Also this solution uses DynamoDB to track the status of each partition and each batch to ensure the data integrity.

Please read the following links to learn more:

• Architecture design

• Sample project using this library

• Sample script to generate fake data for integration test

• Sample script to run the downloader

Partition Tracker Data:

{
    "n_batches": 2, # this partition has 2 batches to download
    "n_succeeded_batches": 2, # number of succeeded batch
    "sftp_dir": "/home/username/part0001", # SFTP directory of this partition
    "s3_dir": "s3://bucket/projects/sftp_to_s3/download/part1", # where you store the downloads
    "n_files": 6, # total number of files in this partition
    "total_size": 5430500 # total size of files in this partition
}

Request Tracker Data:

# this is the first batch
{
    "partition_key": "part1", # partition key
    "request_id": "request1", # request id / batch id
    "sftp_dir": "/home/username/part1", # SFTP directory of this partition
    "s3_dir": "s3://bucket/projects/sftp_to_s3/download/part1", # where you store the downloads
    "s3uri_batch": "s3://bucket/projects/sftp_to_s3/requests/part1/request1.json", # where you store the batch data
    "n_files": "3", # total number of files in this batch
    "total_size": 3442228 # 3.4 MB # total size of files in this batch
}

Batch data JSON file:

# this is the sample content of the batch data
# it is the "s3uri_batch" field in Request Tracker Data
{
    "dir_root": "/home/username/part1", # SFTP directory of this partition
    "s3uri_root": "s3://bucket/projects/sftp_to_s3/download/part1", # where you store the downloads
    "files": [ # list of files to download
        "/home/username/part1/file0001.txt",
        "/home/username/part1/file0002.txt",
        "/home/username/part1/file0003.txt"
    ],
    "n_files": "3", # total number of files in this batch
    "total_size": 3442228 # 3.4 MB # total size of files in this batch
}

Install

sftp_to_s3 is released on PyPI, so all you need is:

$ pip install sftp_to_s3

To upgrade to latest version:

$ pip install --upgrade sftp_to_s3