S3 to Synapse Orchestration
Project description
S3 to Synapse Orchestration
- Utilities for provisioning EBS volumes and attaching them to an EC2 instance.
- Pull a S3 bucket to the EBS volume.
- Transfer the contents of the S3 bucket to Synapse.
- List files in S3.
- List local files.
- Compare local objects to S3 objects.
Dependencies
- Python3.7
- A Synapse account with a username/password. Authentication through a 3rd party (.e.g., Google) will not work, you must have a Synapse user/pass for the API to authenticate.
- synapse-uploader
- AWS CLI
Installation
pip install snowplow
or
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple --no-cache-dir snowplow
Usage
usage: snowplow [-h] {list-local,list-s3,compare-s3,compare-csv,new} ...
S3 to Synapse Orchestration
optional arguments:
-h, --help show this help message and exit
Commands:
{list-local,list-s3,compare-s3,compare-csv,new}
list-local List all the folders and files with their size and
MD5.
list-s3 List all the files in an S3 bucket with their file
size.
compare-s3 Compare a local directory against an S3 bucket.
compare-csv Compare a CSV file from the list-local command with a
local directory.
new Provision a new transfer instance. This will create an
EBS volume and attach it to the host system
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
snowplow-0.0b27.tar.gz
(14.7 kB
view hashes)
Built Distribution
Close
Hashes for snowplow-0.0b27-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5bb3d8d26ed9d2bb62aa5a0e1c1794b62459ad7b4c6a660e5147cdb93a8411b5 |
|
MD5 | a78740f8ce41c61c538fb260cea07c50 |
|
BLAKE2b-256 | 455bc71f8901791105d8884cb9d2b11403ce2232127a73ac6d97e36d15a17d6e |