sqoop-it
A python package that lets you sqoop into HDFS/Hive/HBase data from RDBMS using sqoop.
To install the package via pip, run
pip install sqoopit
You can then use the package using
from sqoopit.SqoopImport import Sqoop
sqoop = Sqoop(help=True)
code = sqoop.perform_import()
This will print the output of the command
sqoop --help
to your stoud; e.g.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.3.0-235/accumulo/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/08/13 20:25:13 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.6.3.0-235
usage: sqoop import [GENERIC-ARGS] [TOOL-ARGS]
Common arguments:
--connect <jdbc-uri> Specify JDBC
connect
string
--connection-manager <class-name> Specify
connection
manager
class name
...
Useful Resources
- HBase Client for Python : happybase
A more concrete example
The following code
sqoop = Sqoop(fs='hdfs://remote-cluster:8020', hive_drop_import_delims=True, fields_terminated_by='\;',
enclosed_by='\'"\'', escaped_by='\\\\', null_string='\'\'', null_non_string='\'\'',
table='sample_table', target_dir='hdfs://remote-cluster/user/hive/warehouse/db/sample_table',
delete_target_dir=True, connect='jdbc:oracle:thin:@//your_ip:your_port/your_schema',
username='user', password='pwd', num_mappers=2,
bindir='/path/to/bindir/folder')
sqoop.perform_import()
will execute the following command
sqoop import -fs hdfs://remote-cluster:8020 --hive-drop-import-delims --fields-terminated-by \; --enclosed-by \'\"\' --escaped-by \\\\ --null-string \'\' --null-non-string \'\' --table sample_table --target-dir hdfs://remote-cluster/user/hive/warehouse/db/sample_table --delete-target-dir --connect jdbc:oracle:thin:@//your_ip:your_port/your_schema --username user --password pwd --num-mappers 2 --bindir /path/to/bindir/folder
Conditional Building
Use the set_param, unset_param function to build conditioned sqoop imports.
sqoop = Sqoop(table="MyTable")
sqoop.set_param(param="--connect", value="jdbc:a_valid_string")
if taget_is_hbase :
added_table = sqoop.set_param(param="--hbase-table", value="MyTable")
added_key = sqoop.set_param(param="--hbase-row-key", value="Id_MyTable")
if added_table and added_key:
print("all params added :D")
sqoop.perform_import()
Doing
- handle sqoop jobs
- more tests coverage
TODOs
- add missing parameters
Original Idea By Luca Fontanili
Metadata
Release files for sqoopit 0.0.12
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| sqoopit-0.0.12.tar.gz | 6.7 kB | Without provenance | Details for sqoopit-0.0.12.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| sqoopit-0.0.12-py3-none-any.whl | Python 3 | none | any | Details for sqoopit-0.0.12-py3-none-any.whl |
Total release size: 15.5 kB
Release files / sqoopit-0.0.12.tar.gz
| Download URL | sqoopit-0.0.12.tar.gz |
|---|---|
| Size | 6.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dd455760e9aae28c17eec9d082a738139e2987a5c0970c135509ab1298906b5a
|
|
BLAKE2b-256 checksum How to use checksums |
e9e43a7fba8f4d51dc69ad8499397be5325383ca6f5bdbd0ef775d9957da28e4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2
|
Release files / sqoopit-0.0.12-py3-none-any.whl
| Download URL | sqoopit-0.0.12-py3-none-any.whl |
|---|---|
| Size | 8.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e344eebf4706050086245166fb7e933d5c0790c54bc2e2b928860264124d5d86
|
|
BLAKE2b-256 checksum How to use checksums |
14feab350c9ef7dd5116d4b463c582985680b48373e020e7bcc849dcc416f503
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2
|
Security for sqoopit 0.0.12
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.