Skip to main content

Package for ingesting Qlik metadata into Google Cloud Data Catalog

Project description

google-datacatalog-qlik-connector

Package for ingesting Qlik Sense metadata into Google Cloud Data Catalog, currently supporting below asset types:

  • Custom Property Definition
  • Stream
  • App (only the published ones)
  • Master Items: Dimension
  • Master Items: Measure
  • Master Items: Visualization
  • Sheet (only the published ones)

Disclaimer: This is not an officially supported Google product.

Table of Contents


1. Installation

Install this library in a virtualenv using pip. virtualenv is a tool to create isolated Python environments. The basic problem it addresses is one of dependencies and versions, and indirectly permissions.

With virtualenv, it's possible to install this library without needing system install permissions, and without clashing with the installed system dependencies. Make sure you use Python 3.6+.

1.1. Mac/Linux

pip3 install virtualenv
virtualenv --python python3.6 <your-env>
source <your-env>/bin/activate
<your-env>/bin/pip install google-datacatalog-qlik-connector

1.2. Windows

pip3 install virtualenv
virtualenv --python python3.6 <your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exe install google-datacatalog-qlik-connector

1.3. Install from source

1.3.1. Get the code

git clone https://github.com/GoogleCloudPlatform/datacatalog-connectors-bi/
cd datacatalog-connectors-bi/google-datacatalog-qlik-connector

1.3.2. Create and activate a virtualenv

pip3 install virtualenv
virtualenv --python python3.6 <your-env>
source <your-env>/bin/activate

1.3.3. Install the library

pip install .

2. Environment setup

2.1. Auth credentials

2.1.1. Create a GCP Service Account and grant it below roles

  • Data Catalog Admin

2.1.2. Download a JSON key and save it as

  • <YOUR-CREDENTIALS_FILES_FOLDER>/qlik2dc-datacatalog-credentials.json

Please notice this folder and file will be required in next steps.

2.2. Set environment variables

The connector uses Windows-based NTLM authentication, which requires the username to be provided in the format of <ẃindows-ad-domain>\<username>. When fulfilling the below environment variables, set QLIK2DC_QLIK_AD_DOMAIN with the Windows Active Directory domain your user belongs to and QLIK2DC_QLIK_USERNAME with the username (no backslash in both).

export GOOGLE_APPLICATION_CREDENTIALS=datacatalog_credentials_file

export QLIK2DC_QLIK_SERVER=qlik_server
export QLIK2DC_QLIK_AD_DOMAIN=qlik_ad_domain
export QLIK2DC_QLIK_USERNAME=qlik_username
export QLIK2DC_QLIK_PASSWORD=qlik_password
export QLIK2DC_DATACATALOG_PROJECT_ID=google_cloud_project_id
export QLIK2DC_DATACATALOG_LOCATION_ID=google_cloud_location_id

Replace above values according to your environment. The Data Catalog credentials file was saved in step 2.1.2.

3. Running the connector

  • The --qlik-ad-domain argument is optional and defaults to ..
  • The --datacatalog-location-id argument is optional and defaults to us.

3.1. Python entry point

  • Virtualenv
google-datacatalog-qlik-connector \
  --qlik-server $QLIK2DC_QLIK_SERVER \
  [--qlik-ad-domain $QLIK2DC_QLIK_AD_DOMAIN \]
  --qlik-username $QLIK2DC_QLIK_USERNAME \
  --qlik-password $QLIK2DC_QLIK_PASSWORD \
  --datacatalog-project-id $QLIK2DC_DATACATALOG_PROJECT_ID \
  [--datacatalog-location-id $QLIK2DC_DATACATALOG_LOCATION_ID]

3.2. Docker entry point

docker build --rm --tag qlik2datacatalog .
docker run --rm --tty -v YOUR-CREDENTIALS_FILES_FOLDER:/data \
  qlik2datacatalog \
  --qlik-server $QLIK2DC_QLIK_SERVER \
  [--qlik-ad-domain $QLIK2DC_QLIK_AD_DOMAIN \]
  --qlik-username $QLIK2DC_QLIK_USERNAME \
  --qlik-password $QLIK2DC_QLIK_PASSWORD \
  --datacatalog-project-id $QLIK2DC_DATACATALOG_PROJECT_ID \
  [--datacatalog-location-id $QLIK2DC_DATACATALOG_LOCATION_ID]

4. Design decisions

4.1. Tag Templates for Custom Property Choice Values

The current implementation creates a Tag Template for each Custom Property Choice Value assigned to Streams or Apps in the provided Qlik Sense site. The rationale behind this decision comprises allowing the connector to synchronize all metadata scraped from each Custom Property, at the same time it enables Qlik assets to be easily found by their custom properties — using query strings such as tag:property_name:"<PROPERTY-NAME>" and tag:value:"<SOME-VALUE>".

Data Catalog accepts attaching only one Tag per Template to a given Entry, so there could be metadata loss if the Tag Templates were created in different ways, e.g. on a Custom Property Definition basis.

Lastly, this approach may lead to the creation of several Tag Templates if there are many Custom Property Values in use in your Qlik Sense site. In case you would like to suggest a different approach to tackle this problem, please file a feature request. We will be happy to discuss alternative solutions!

5. Developer environment

5.1. Install and run Yapf formatter

pip install --upgrade yapf

# Auto update files
yapf --in-place --recursive src tests

# Show diff
yapf --diff --recursive src tests

# Set up pre-commit hook
# From the root of your git project.
curl -o pre-commit.sh https://raw.githubusercontent.com/google/yapf/master/plugins/pre-commit.sh
chmod a+x pre-commit.sh
mv pre-commit.sh .git/hooks/pre-commit

5.2. Install and run Flake8 linter

pip install --upgrade flake8
flake8 src tests

5.3. Run Tests

python setup.py test

5.4. Additional resources

Please refer to the Developer Resources documentation.

6. Troubleshooting

6.1. Qlik APIs compatibility

The connector may fail during the scrape stage if the Qlik APIs do not return metadata in the expected format. As a reference, the below versions were already validated:

  • Qlik Sense Repository Service API
VERSION RESULT
34.16.0 (September2020) SUCCESS
  • Qlik Engine JSON API
VERSION RESULT
12.763.4 (September2020) SUCCESS

6.2. Data Catalog quota

In case a connector execution hits Data Catalog quota limit, an error will be raised and logged with the following details, depending on the performed operation (READ/WRITE/SEARCH):

status = StatusCode.RESOURCE_EXHAUSTED
details = "Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'."
debug_error_string = 
"{"created":"@1587396969.506556000", "description":"Error received from peer ipv4:172.217.29.42:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'.","grpc_status":8}"

For more information on Data Catalog quota, please refer to: Data Catalog quota docs.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for google-datacatalog-qlik-connector, version 0.1.1
Filename, size File type Python version Upload date Hashes
Filename, size google_datacatalog_qlik_connector-0.1.1-py2.py3-none-any.whl (47.2 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size google-datacatalog-qlik-connector-0.1.1.tar.gz (30.7 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page