This plugin allows vdk-core to interface with and execute queries against an Impala database. Additionally, it can collect lineage data, assuming a lineage logger has been provided through the vdk-core configuration.
Usage
Run
pip install vdk-impala
After this, data jobs will have access to a Impala database connection, managed by Versatile Data Kit SDK.
If it is the only database plugin installed , vdk would automatically use it.
Otherwise, users need to set VDK_DB_DEFAULT_TYPE=IMPALA as an environment variable or set 'db_default_type' option in the data job config file (config.ini).
For example
def run(job_input: IJobInput):
job_input.execute_query("select 'Hi Impala!'")
Lineage
The package can gather lineage data for all successful Impala SQL queries that have actually read or written data. Other plugins can read and optionally send the lineage data to separate system. They need to provide ILineageLogger implementation and hook this way:
@hookimpl
def vdk_initialize(context: CoreContext) -> None:
context.state.set(StoreKey[ILineageLogger]("impala-lineage-logger"), MyLogger())
Lineage is calculated based on the executed query profile. It is retrieved via the cursor by executing additional RPC request against the same Impala node that has coordinated the query right after the original query has successfully finished. See https://impala.apache.org/docs/build/html/topics/impala_logging.html for more information how profiles are stored and here https://impala.apache.org/docs/build/impala-3.1.pdf for more information about the profiles themselves.
If enabled, query plan is retrieved for every successfully executed query against Impala excluding keepalive queries like "Select 1".
Configuration
Run vdk config-help - search for those prefixed with "IMPALA_" to see what configuration options are available.
Disclaimer
This plugin is tested against a specific impala version. The version comes from the docker-compose.yaml container's impala version. For more information on the imapala version tested against please google the docker image.
Testing
Testing this plugin locally requires installing the dependencies listed in vdk-plugins/vdk-impala/requirements.txt
Run
pip install -r requirements.txt
1 maintainer
Metadata
1 maintainer
Release files for vdk-impala 0.4.607542307
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| vdk-impala-0.4.607542307.tar.gz | 20.5 kB | Details |
Release files / vdk-impala-0.4.607542307.tar.gz
| Download URL | vdk-impala-0.4.607542307.tar.gz |
|---|---|
| Size | 20.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7a97921aecdcd38a09cef55ca0e4b13c439230304a655448fe84291216453492
|
|
BLAKE2b-256 checksum How to use checksums |
4fd437a8d0a2c51356671ce37c09320bef00c51950cf379706f66ca5efec2545
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.7.13
|
