S3 to Synapse Orchestration
Project description
S3 to Synapse Orchestration
- Utilities for provisioning EBS volumes and attaching them to an EC2 instance.
- Pull a S3 bucket to the EBS volume.
- Transfer the contents of the S3 bucket to Synapse.
- List files in S3.
- List local files.
- Compare local objects to S3 objects.
Dependencies
- Python3.7
- A Synapse account with a username/password. Authentication through a 3rd party (.e.g., Google) will not work, you must have a Synapse user/pass for the API to authenticate.
- synapse-uploader
- AWS CLI
Installation
pip install snowplow
or
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple --no-cache-dir snowplow
Usage
usage: snowplow [-h] {list-local,list-s3,compare-s3,compare-csv,new} ...
S3 to Synapse Orchestration
optional arguments:
-h, --help show this help message and exit
Commands:
{list-local,list-s3,compare-s3,compare-csv,new}
list-local List all the folders and files with their size and
MD5.
list-s3 List all the files in an S3 bucket with their file
size.
compare-s3 Compare a local directory against an S3 bucket.
compare-csv Compare a CSV file from the list-local command with a
local directory.
new Provision a new transfer instance. This will create an
EBS volume and attach it to the host system
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
snowplow-0.0b27.tar.gz
(14.7 kB
view details)
Built Distribution
File details
Details for the file snowplow-0.0b27.tar.gz
.
File metadata
- Download URL: snowplow-0.0b27.tar.gz
- Upload date:
- Size: 14.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 51d8d9f78a6ccec9e7b11599ba689c9a3ebdc7f3947a3b545bec276805f7e52b |
|
MD5 | 61cb1da563bc0944af61856f6c952681 |
|
BLAKE2b-256 | 69e6a6b74c97166b351c87506171d7581b6fa7658a48491a38f3895a5e2a2011 |
File details
Details for the file snowplow-0.0b27-py2.py3-none-any.whl
.
File metadata
- Download URL: snowplow-0.0b27-py2.py3-none-any.whl
- Upload date:
- Size: 21.8 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5bb3d8d26ed9d2bb62aa5a0e1c1794b62459ad7b4c6a660e5147cdb93a8411b5 |
|
MD5 | a78740f8ce41c61c538fb260cea07c50 |
|
BLAKE2b-256 | 455bc71f8901791105d8884cb9d2b11403ce2232127a73ac6d97e36d15a17d6e |