Skip to main content

Introduction

s3import is a tool for importing data from S3 to an SX Cluster. For each source S3 bucket s3import creates a volume on the cluster and copies the keys from the bucket to the new volume.

s3import uses boto (https://pypi.python.org/pypi/boto) and python-sxclient (https://pypi.python.org/pypi/sxclient) and works in Python 2.7.

Installation

To install s3import, run:

$ pip install s3import

Alternatively, to install s3import from source, run:

$ pip install <path>

or

$ pip install -e <path>

replacing <path> with path to the source.

Configuration

S3

Configuration for connecting with an S3 service is obtained from boto config file. Your file should contain at least:

  • aws_access_key_id in [Credentials] section, with your S3 access key ID as a value;

  • aws_secret_access_key in [Credentials] section, with your S3 secret access key as a value.

Additionally, if you want to connect to an S3 service other than Amazon S3, your file should contain:

  • s3_host in [Credentials] section, with hostname of the S3 service you want to access;

  • s3_port in [Credentials] section, with port which the S3 service uses.

By default, connection with S3 is secured by SSL and the SSL certificates are verified. If you wish to change any of these behaviours, add is_secure = False and https_validate_certificates = False respectively to the [Boto] section of your config file.

In order for boto to read the config file automatically, you should name it .boto and place it in your home directory. You can also set the environment variable BOTO_CONFIG with the path to the configuration file you want to use prior to running s3import.

Be wary that boto tries to read configuration options from other locations than boto config file. For example, if a file ~/.aws/credentials exists, boto will read credential settings from there. For more information see http://docs.pythonboto.org/en/latest/boto_config_tut.html.

Below is an example content of a configuration file:

[Credentials]
s3_host = s3.example.com
s3_port = 8443
aws_access_key_id = s3_user
aws_secret_access_key = secret_key

[Boto]
is_secure = True
https_validate_certificates = True

SX and import configuration

Configuration for connecting with an SX Cluster and importing the buckets is obtained from s3import command-line arguments. You can display their descriptions by running s3import --help.

Usage

After installation, a command-line tool named s3import will become available. Run:

$ s3import --help

for a list of available options.

Release files for s3import 0.6.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for s3import 0.6.4
File Size Uploaded
s3import-0.6.4.tar.gz 12.7 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for s3import 0.6.4
File Interpreter ABI Platform
s3import-0.6.4-py2-none-any.whl Python 2 none any Details

Total release size:29.9 kB

Release files / s3import-0.6.4.tar.gz

Download URL s3import-0.6.4.tar.gz
Size 12.7 kB
Tags Source
SHA-256 checksum
How to use checksums
54444cb9aa003d5f7eb42abaab01d405ed56ab8920bcfcca4d10775371d318f3
BLAKE2b-256 checksum
How to use checksums
d00e7720505610cd635cf6ffa85127e67c4eff79141c34ee9953d105f85c9061
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / s3import-0.6.4-py2-none-any.whl

Download URL s3import-0.6.4-py2-none-any.whl
Size 17.2 kB
Tags Python 2
SHA-256 checksum
How to use checksums
b4f875106896e3aa0b11a07807314769cec5dc9067c8a8d152d0c915bb10e848
BLAKE2b-256 checksum
How to use checksums
24aef5085c3b89f4d9ed8607f8f0c40a03ad160bd823c6cd3ecff25ff48e283e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release history Release notifications | RSS feed

This release

0.6.4 This release

2 release files

0.6.3

2 release files

0.6.2

2 release files

0.6.1

2 release files

0.6.0

2 release files

0.5.1

2 release files

0.5.0

2 release files

0.4.1

2 release files

0.4.0

2 release files

0.3.0

2 release files

0.2.1

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page