Skip to main content

UNKNOWN

Project description

sqoopy
======

Python CLI to generate custom [sqoop](http://sqoop.apache.org/) import statements.
Modified from [https://github.com/wikimedia/sqoopy/](https://github.com/wikimedia/sqoopy/).


## Installation

You can install `sqoopy` via `pip`:

```bash
$ pip install sqoopy
```

## Usage

`sqoopy` will generate custom [sqoop](http://sqoop.apache.org/) import statements given a few simple options:

```bash
usage: sqoopy [-h] [-c CONNECT] [-d TARGET_DIR] [-t TABLES]

Python CLI to generate custom sqoop import statements.

optional arguments:
-h, --help show this help message and exit
-c CONNECT, --connect CONNECT
A jdbc connection string.
-d TARGET_DIR, --target-dir TARGET_DIR
The directory to send output to. If sending to s3, use
"{table}" to insert the table name into the directory.
EG: s3://my-bucket/{table}/
-t TABLES, --tables TABLES
(Optional) comma-separated list of tables that need to
be inspected. If not supplied, all tables will be
imported.
```

For example, here is a simple command that will generate an import command given the [test database](tests/test.db):

```bash
$ sqoopy --connect=sqlite:///tests/test.db --target-dir=s3://foo-bar/
```

This will output:

```
sqoop import --connect=sqlite:///tests/test.db --table=test --target-dir=s3://foo-bar/
```

If you'd like to programmatically add the table name to the `target-dir`, use `{table}`:

```
$ sqoopy --connect=sqlite:///tests/test.db --target-dir=s3://foo-bar/{table}
```

This will output:

```
sqoop import --connect=sqlite:///tests/test.db --table=test --target-dir=s3://foo-bar/test/
```

If you'd like to use a custom list of tables to import, use the `--tables` argument, with each table name separated by a comma:

```
$ sqoopy --connect=sqlite:///tests/test.db --target-dir=s3://foo-bar/{table} --tables=test
```

Finally, you can also passthrough any other `sqoop import` arguments:

```bash
$ sqoopy --connect=sqlite:///tests/test.db --target-dir=s3://foo-bar/{table} --tables=test --split-by id --num-mappers 4
```

This will output:

```
sqoop import --connect=sqlite:///tests/test.db --table=test --target-dir=s3://foo-bar/test/ --split-by id --num-mappers 4
```

## Tests

You can run tests by first installing `nose`:

```
$ pip install nose
$ nosetests
```

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sqoopy-0.0.8.tar.gz (5.2 kB view hashes)

Uploaded Source

Built Distribution

sqoopy-0.0.8.macosx-10.10-intel.exe (108.7 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page