s3hop - Cross-account S3 bucket streaming copy

A command-line tool to efficiently copy files between S3 buckets across different AWS accounts.

Features

• Copy files between S3 buckets across different AWS accounts
• Smart file comparison to only copy new or updated files
• Progress tracking with ETA and transfer speed
• Detailed transfer summary with file type statistics
• Handles large files and directories efficiently using streaming
• Preserves file structure between source and destination

Quick Start

# Install
pip install s3hop

# Use
s3hop source-profile s3://source-bucket/prefix/ dest-profile s3://dest-bucket/prefix/

Installation

You can install this package directly from PyPI:

pip install s3hop

Or install from source:

git clone https://github.com/thehamsti/s3hop.git
cd s3hop
pip install .

Prerequisites

1. Python 3.6 or later
2. AWS credentials configured for both source and destination accounts
3. Appropriate S3 permissions in both accounts

AWS Credentials Setup

1. Configure your AWS credentials for both accounts in ~/.aws/credentials:

[source-profile]
aws_access_key_id = YOUR_SOURCE_ACCESS_KEY
aws_secret_access_key = YOUR_SOURCE_SECRET_KEY

[dest-profile]
aws_access_key_id = YOUR_DEST_ACCESS_KEY
aws_secret_access_key = YOUR_DEST_SECRET_KEY

2. Ensure you have the necessary S3 permissions:
   • Source bucket: s3:ListBucket, s3:GetObject
   • Destination bucket: s3:ListBucket, s3:PutObject

Usage

Basic usage:

s3hop source-profile s3://source-bucket/prefix/ dest-profile s3://dest-bucket/prefix/

Example with specific profiles and paths:

s3hop prod s3://prod-bucket/data/ staging s3://staging-bucket/backup/

Arguments

• source-profile: AWS profile for the source account
• source_url: Source S3 URL (s3://bucket-name/prefix/)
• dest-profile: AWS profile for the destination account
• dest_url: Destination S3 URL (s3://bucket-name/prefix/)

Features in Detail

1. Smart File Comparison

   • Compares files using ETags and last modified timestamps
   • Only copies new or updated files
   • Preserves existing files that haven't changed

2. Progress Tracking

   • Real-time transfer speed
   • Estimated time remaining
   • Progress bar with file counts
   • Total data transferred/remaining

3. Transfer Summary

   • Start and end times
   • Duration
   • Number of files transferred/skipped
   • File type statistics
   • Failed transfers (if any)

Development

For development setup and contributing guidelines, see CONTRIBUTING.md.

Changelog

See CHANGELOG.md for a list of changes and version history.

Security

For security issues, please email security@hamsti.co instead of using the issue tracker.

Support

• 📫 Email: john@hamsti.co
• 🐛 Issues: GitHub Issues

License

MIT License - see LICENSE for details.

Acknowledgments

• Built with boto3
• Progress bars powered by tqdm