Skip to main content

sqoop-it

A python package that lets you sqoop into HDFS/Hive/HBase data from RDBMS using sqoop.

PyPI Python MIT license

To install the package via pip, run

pip install sqoopit

You can then use the package using

from sqoopit.SqoopImport import Sqoop 
sqoop = Sqoop(help=True)
code = sqoop.perform_import()

This will print the output of the command

sqoop --help

to your stoud; e.g.

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.3.0-235/accumulo/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/08/13 20:25:13 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.6.3.0-235
usage: sqoop import [GENERIC-ARGS] [TOOL-ARGS]

Common arguments:
   --connect <jdbc-uri>                                       Specify JDBC
                                                              connect
                                                              string
   --connection-manager <class-name>                          Specify
                                                              connection
                                                              manager
                                                              class name
   ...

Useful Resources

A more concrete example

The following code

sqoop = Sqoop(fs='hdfs://remote-cluster:8020', hive_drop_import_delims=True, fields_terminated_by='\;',
enclosed_by='\'"\'', escaped_by='\\\\', null_string='\'\'', null_non_string='\'\'',
table='sample_table', target_dir='hdfs://remote-cluster/user/hive/warehouse/db/sample_table',
delete_target_dir=True, connect='jdbc:oracle:thin:@//your_ip:your_port/your_schema',
username='user', password='pwd', num_mappers=2,
bindir='/path/to/bindir/folder')

sqoop.perform_import()

will execute the following command

sqoop import -fs hdfs://remote-cluster:8020 --hive-drop-import-delims --fields-terminated-by \; --enclosed-by \'\"\' --escaped-by \\\\ --null-string \'\' --null-non-string \'\' --table sample_table --target-dir hdfs://remote-cluster/user/hive/warehouse/db/sample_table --delete-target-dir --connect jdbc:oracle:thin:@//your_ip:your_port/your_schema --username user --password pwd --num-mappers 2 --bindir /path/to/bindir/folder

Conditional Building

Use the set_param, unset_param function to build conditioned sqoop imports.

sqoop = Sqoop(table="MyTable")

sqoop.set_param(param="--connect", value="jdbc:a_valid_string")

if taget_is_hbase :
   added_table = sqoop.set_param(param="--hbase-table", value="MyTable")
   added_key = sqoop.set_param(param="--hbase-row-key", value="Id_MyTable")
   if added_table and added_key:
      print("all params added :D")

sqoop.perform_import()

Doing

  • handle sqoop jobs
  • more tests coverage

TODOs

  • add missing parameters

Original Idea By Luca Fontanili

Metadata

Release files for sqoopit 0.0.12

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sqoopit 0.0.12
File Size Uploaded Provenance Actions
sqoopit-0.0.12.tar.gz 6.7 kB Without provenance Details for sqoopit-0.0.12.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sqoopit 0.0.12
File Interpreter ABI Platform Actions
sqoopit-0.0.12-py3-none-any.whl Python 3 none any Details for sqoopit-0.0.12-py3-none-any.whl

Total release size: 15.5 kB

Release files / sqoopit-0.0.12.tar.gz

Download URL sqoopit-0.0.12.tar.gz
Size 6.7 kB
Tags Source
SHA-256 checksum
How to use checksums
dd455760e9aae28c17eec9d082a738139e2987a5c0970c135509ab1298906b5a
BLAKE2b-256 checksum
How to use checksums
e9e43a7fba8f4d51dc69ad8499397be5325383ca6f5bdbd0ef775d9957da28e4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

Release files / sqoopit-0.0.12-py3-none-any.whl

Download URL sqoopit-0.0.12-py3-none-any.whl
Size 8.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e344eebf4706050086245166fb7e933d5c0790c54bc2e2b928860264124d5d86
BLAKE2b-256 checksum
How to use checksums
14feab350c9ef7dd5116d4b463c582985680b48373e020e7bcc849dcc416f503
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

Security for sqoopit 0.0.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.0.12 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page