Skip to main content

UNKNOWN

Project description

sqoopy
======

Python CLI to generate custom [sqoop](http://sqoop.apache.org/) import statements.
Modified from [https://github.com/wikimedia/sqoopy/](https://github.com/wikimedia/sqoopy/).


## Installation

You can install `sqoopy` via `pip`:

```bash
$ pip install sqoopy
```

## Usage

`sqoopy` will generate custom [sqoop](http://sqoop.apache.org/) import statements given a few simple options:

```bash
usage: sqoopy [-h] [-c CONNECT] [-d TARGET_DIR] [-t TABLES]

Python CLI to generate custom sqoop import statements.

optional arguments:
-h, --help show this help message and exit
-c CONNECT, --connect CONNECT
A jdbc connection string.
-d TARGET_DIR, --target-dir TARGET_DIR
The directory to send output to. If sending to s3, use
"{table}" to insert the table name into the directory.
EG: s3://my-bucket/{table}/
-t TABLES, --tables TABLES
(Optional) comma-separated list of tables that need to
be inspected. If not supplied, all tables will be
imported.
```

For example, here is a simple command that will generate an import command given the [test database](tests/test.db):

```bash
$ sqoopy --connect=sqlite:///tests/test.db --target-dir=s3://foo-bar/
```

This will output:

```
sqoop import --connect=sqlite:///tests/test.db --table=test --target-dir=s3://foo-bar/
```

If you'd like to programmatically add the table name to the `target-dir`, use `{table}`:

```
$ sqoopy --connect=sqlite:///tests/test.db --target-dir=s3://foo-bar/{table}
```

This will output:

```
sqoop import --connect=sqlite:///tests/test.db --table=test --target-dir=s3://foo-bar/test/
```

If you'd like to use a custom list of tables to import, use the `--tables` argument, with each table name separated by a comma:

```
$ sqoopy --connect=sqlite:///tests/test.db --target-dir=s3://foo-bar/{table} --tables=test
```

Finally, you can also passthrough any other `sqoop import` arguments:

```bash
$ sqoopy --connect=sqlite:///tests/test.db --target-dir=s3://foo-bar/{table} --tables=test --split-by id --num-mappers 4
```

This will output:

```
sqoop import --connect=sqlite:///tests/test.db --table=test --target-dir=s3://foo-bar/test/ --split-by id --num-mappers 4
```

## Tests

You can run tests by first installing `nose`:

```
$ pip install nose
$ nosetests
```

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for sqoopy, version 0.0.75
Filename, size File type Python version Upload date Hashes
Filename, size sqoopy-0.0.75.macosx-10.10-intel.exe (108.8 kB) File type Windows Installer Python version any Upload date Hashes View
Filename, size sqoopy-0.0.75.tar.gz (5.2 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page