Skip to main content

S3 data import utility

Project description

Introduction

s3import is a tool for importing data from S3 to an SX Cluster. For each source S3 bucket s3import creates a volume on the cluster and copies the keys from the bucket to the new volume.

s3import uses boto (https://pypi.python.org/pypi/boto) and python-sxclient (https://pypi.python.org/pypi/sxclient) and works in Python 2.7.

Installation

To install s3import, run:

$ pip install s3import

Alternatively, to install s3import from source, run:

$ pip install <path>

or

$ pip install -e <path>

replacing <path> with path to the source.

Configuration

S3

Configuration for connecting with an S3 service is obtained from boto config file. Your file should contain at least:

  • aws_access_key_id in [Credentials] section, with your S3 access key ID as a value;

  • aws_secret_access_key in [Credentials] section, with your S3 secret access key as a value.

Additionally, if you want to connect to an S3 service other than Amazon S3, your file should contain:

  • s3_host in [Credentials] section, with hostname of the S3 service you want to access;

  • s3_port in [Credentials] section, with port which the S3 service uses.

By default, connection with S3 is secured by SSL and the SSL certificates are verified. If you wish to change any of these behaviours, add is_secure = False and https_validate_certificates = False respectively to the [Boto] section of your config file.

In order for boto to read the config file automatically, you should name it .boto and place it in your home directory. You can also set the environment variable BOTO_CONFIG with the path to the configuration file you want to use prior to running s3import.

Be wary that boto tries to read configuration options from other locations than boto config file. For example, if a file ~/.aws/credentials exists, boto will read credential settings from there. For more information see http://docs.pythonboto.org/en/latest/boto_config_tut.html.

Below is an example content of a configuration file:

[Credentials]
s3_host = s3.example.com
s3_port = 8443
aws_access_key_id = s3_user
aws_secret_access_key = secret_key

[Boto]
is_secure = True
https_validate_certificates = True

SX and import configuration

Configuration for connecting with an SX Cluster and importing the buckets is obtained from s3import command-line arguments. You can display their descriptions by running s3import --help.

Usage

After installation, a command-line tool named s3import will become available. Run:

$ s3import --help

for a list of available options.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

s3import-0.6.3.tar.gz (11.5 kB view details)

Uploaded Source

Built Distribution

s3import-0.6.3-py2-none-any.whl (17.2 kB view details)

Uploaded Python 2

File details

Details for the file s3import-0.6.3.tar.gz.

File metadata

  • Download URL: s3import-0.6.3.tar.gz
  • Upload date:
  • Size: 11.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for s3import-0.6.3.tar.gz
Algorithm Hash digest
SHA256 31542558eb8de778806d4bff0c6fd77bc96aae3fd900587cf61874cf072599fa
MD5 04c8ae9632bed35dc90af948d70e64d7
BLAKE2b-256 ccc5f0b7a006f4897cefb1b6d2bdfc14d1813cd158c981b6654a5cd964bd64cb

See more details on using hashes here.

File details

Details for the file s3import-0.6.3-py2-none-any.whl.

File metadata

File hashes

Hashes for s3import-0.6.3-py2-none-any.whl
Algorithm Hash digest
SHA256 3327ae65caf61ed9c9cb85fd52fd97c63b0c78e47700a1f99897ea9fbba9f46a
MD5 c37f945eb7bc82a90139b20bb81eb2a7
BLAKE2b-256 e9e86667b3ed592e2a5e4f5c89a526de6b00f26ac95ed68d36142db2d0c96e37

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page