Skip to main content

S3 to Synapse Orchestration

Project description

S3 to Synapse Orchestration

  • Utilities for provisioning EBS volumes and attaching them to an EC2 instance.
  • Pull a S3 bucket to the EBS volume.
  • Transfer the contents of the S3 bucket to Synapse.
  • List files in S3.
  • List local files.
  • Compare local objects to S3 objects.

Dependencies

Installation

pip install snowplow

or

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple --no-cache-dir snowplow

Usage

usage: snowplow [-h] {list-local,list-s3,compare-s3,compare-csv,new} ...

S3 to Synapse Orchestration

optional arguments:
  -h, --help            show this help message and exit

Commands:
  {list-local,list-s3,compare-s3,compare-csv,new}
    list-local          List all the folders and files with their size and
                        MD5.
    list-s3             List all the files in an S3 bucket with their file
                        size.
    compare-s3          Compare a local directory against an S3 bucket.
    compare-csv         Compare a CSV file from the list-local command with a
                        local directory.
    new                 Provision a new transfer instance. This will create an
                        EBS volume and attach it to the host system

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

snowplow-0.0b27.tar.gz (14.7 kB view hashes)

Uploaded Source

Built Distribution

snowplow-0.0b27-py2.py3-none-any.whl (21.8 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page