Skip to main content

S3 to Synapse Orchestration

Project description

S3 to Synapse Orchestration

  • Utilities for provisioning EBS volumes and attaching them to an EC2 instance.
  • Pull a S3 bucket to the EBS volume.
  • Transfer the contents of the S3 bucket to Synapse.
  • List files in S3.
  • List local files.
  • Compare local objects to S3 objects.

Dependencies

Installation

pip install snowplow

or

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple --no-cache-dir snowplow

Usage

usage: snowplow [-h] {list-local,list-s3,compare-s3,compare-csv,new} ...

S3 to Synapse Orchestration

optional arguments:
  -h, --help            show this help message and exit

Commands:
  {list-local,list-s3,compare-s3,compare-csv,new}
    list-local          List all the folders and files with their size and
                        MD5.
    list-s3             List all the files in an S3 bucket with their file
                        size.
    compare-s3          Compare a local directory against an S3 bucket.
    compare-csv         Compare a CSV file from the list-local command with a
                        local directory.
    new                 Provision a new transfer instance. This will create an
                        EBS volume and attach it to the host system

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

snowplow-0.0b27.tar.gz (14.7 kB view details)

Uploaded Source

Built Distribution

snowplow-0.0b27-py2.py3-none-any.whl (21.8 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file snowplow-0.0b27.tar.gz.

File metadata

  • Download URL: snowplow-0.0b27.tar.gz
  • Upload date:
  • Size: 14.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.5

File hashes

Hashes for snowplow-0.0b27.tar.gz
Algorithm Hash digest
SHA256 51d8d9f78a6ccec9e7b11599ba689c9a3ebdc7f3947a3b545bec276805f7e52b
MD5 61cb1da563bc0944af61856f6c952681
BLAKE2b-256 69e6a6b74c97166b351c87506171d7581b6fa7658a48491a38f3895a5e2a2011

See more details on using hashes here.

File details

Details for the file snowplow-0.0b27-py2.py3-none-any.whl.

File metadata

  • Download URL: snowplow-0.0b27-py2.py3-none-any.whl
  • Upload date:
  • Size: 21.8 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.5

File hashes

Hashes for snowplow-0.0b27-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 5bb3d8d26ed9d2bb62aa5a0e1c1794b62459ad7b4c6a660e5147cdb93a8411b5
MD5 a78740f8ce41c61c538fb260cea07c50
BLAKE2b-256 455bc71f8901791105d8884cb9d2b11403ce2232127a73ac6d97e36d15a17d6e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page