Skip to main content

UNKNOWN

Project description

sqoopy
======

Python CLI to generate custom [sqoop](http://sqoop.apache.org/) import statements.
Modified from [https://github.com/wikimedia/sqoopy/](https://github.com/wikimedia/sqoopy/).


## Installation

You can install `sqoopy` via `pip`:

```bash
$ pip install sqoopy
```

## Usage

`sqoopy` will generate custom [sqoop](http://sqoop.apache.org/) import statements given a few simple options:

```bash
usage: sqoopy [-h] [-c CONNECT] [-d TARGET_DIR] [-t TABLES] [-x EXCL_TABLES]
[--generate] [--pool-size POOL_SIZE]
[--max-pool-maps MAX_POOL_MAPS] [--min-mbs MIN_MBS]
[--max-mbs MAX_MBS]

Python CLI to generate custom sqoop import statements.

optional arguments:
-h, --help show this help message and exit
-c CONNECT, --connect CONNECT
A jdbc connection string.
-d TARGET_DIR, --target-dir TARGET_DIR
The directory to send output to. If sending to s3, use
"{table}" to insert the table name into the directory.
EG: s3://my-bucket/{table}/
-t TABLES, --tables TABLES
(Optional) comma-separated list of tables that need to
be inspected. If not supplied, all tables will be
imported.
-x EXCL_TABLES, --excl-tables EXCL_TABLES
(Optional) comma-separated list of tables to exclude.
If not supplied and --tables not specified, all tables
will be imported.
--generate Just generate the sqoop commands and print them to the
console.
--pool-size POOL_SIZE
The number of commands to execute concurrently
--max-pool-maps MAX_POOL_MAPS
The number of mappers at which the import of a table
will occur serially, after all other pooled imports
are complete
--min-mbs MIN_MBS The minimim chunk size (in MBs). Used to determine the
number of mappers needed for a given table
--max-mbs MAX_MBS The maximum chunk size (in MBs). Used to determine the
number of mappers needed for a given table
```

You can also passthrough any other `sqoop import` arguments:

```bash
$ sqoopy --connect=sqlite:///tests/test.db --target-dir=s3://foo-bar/{table} --tables=test --split-by id --num-mappers 4
```

This will output:

```
sqoop import --connect=sqlite:///tests/test.db --table=test --target-dir=s3://foo-bar/test/ --split-by id --num-mappers 4
```

## Tests

You can run tests by first installing `nose`:

```
$ pip install nose
$ nosetests
```

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sqoopy-0.0.8.8.tar.gz (5.8 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page