google-datacatalog-looker-connector 0.6.1

Package for ingesting Looker metadata into Google Cloud Data Catalog

google-datacatalog-looker-connector

Package for ingesting Looker metadata into Google Cloud Data Catalog, currently supporting below asset types:

• Folder
• Look
• Dashboard
• Dashboard Element (aka Tile)
• Query

Disclaimer: This is not an officially supported Google product.

Table of Contents

• 1. Installation
  • 1.1. Mac/Linux
  • 1.2. Windows
  • 1.3. Install from source
    • 1.3.1. Get the code
    • 1.3.2. Create and activate a virtualenv
    • 1.3.3. Install the library
• 2. Environment setup
  • 2.1. Auth credentials
    • 2.1.1. Create a GCP Service Account and grant it below roles
    • 2.1.2. Download a JSON key and save it as
    • 2.1.3. Create Looker API3 credentials
    • 2.1.4. Create a Looker configuration file
  • 2.2. Set environment variables
• 3. Running the connector
  • 3.1. Python entry point
  • 3.2. Docker entry point
• 4. Developer environment
  • 4.1. Install and run Yapf formatter
  • 4.2. Install and run Flake8 linter
  • 4.3. Run Tests
  • 4.4. Additional resources
• 5. Troubleshooting
  • 5.1. Looker APIs compatibility
  • 5.2. Data Catalog quota

---

1. Installation

Install this library in a virtualenv using pip. virtualenv is a tool to create isolated Python environments. The basic problem it addresses is one of dependencies and versions, and indirectly permissions.

With virtualenv, it's possible to install this library without needing system install permissions, and without clashing with the installed system dependencies. Make sure you use Python 3.7.

1.1. Mac/Linux

pip3 install virtualenv
virtualenv --python python3.7 <your-env>
source <your-env>/bin/activate
<your-env>/bin/pip install google-datacatalog-looker-connector

1.2. Windows

pip3 install virtualenv
virtualenv --python python3.7 <your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exe install google-datacatalog-looker-connector

1.3. Install from source

1.3.1. Get the code

git clone https://github.com/GoogleCloudPlatform/datacatalog-connectors-bi/
cd datacatalog-connectors-bi/google-datacatalog-looker-connector

1.3.2. Create and activate a virtualenv

pip3 install virtualenv
virtualenv --python python3.7 <your-env>
source <your-env>/bin/activate

1.3.3. Install the library

pip install .

2. Environment setup

2.1. Auth credentials

2.1.1. Create a GCP Service Account and grant it below roles

• Data Catalog Admin

2.1.2. Download a JSON key and save it as

• <YOUR-CREDENTIALS_FILES_FOLDER>/looker2dc-datacatalog-credentials.json

Please notice this folder and file will be required in next steps.

2.1.3. Create Looker API3 credentials

The credentials required for API access must be obtained by creating an API3 key on a user account in the Looker Admin console. The API3 key consists of a public client_id and a private client_secret.

The shortcut for Looker Admin console is https:///admin/users/api3_key/

2.1.4. Create a Looker configuration file

File content is described in Looker SDK documentation. Save the file as <YOUR-CREDENTIALS_FILES_FOLDER>/looker2dc-looker-credentials.ini

Please notice this folder and file will be required in next steps.

2.2. Set environment variables

export GOOGLE_APPLICATION_CREDENTIALS=datacatalog_credentials_file

Replace above values according to your environment. The Data Catalog credentials file was saved in step 2.1.2.

3. Running the connector

3.1. Python entry point

• Virtualenv

google-datacatalog-looker-connector \
  --datacatalog-project-id <YOUR-DATACATALOG-PROJECT-ID> \
  --looker-credentials-file looker_credentials_ini_file

Replace above values according to your environment. The Looker credentials file was saved in step 2.1.4.

3.2. Docker entry point

docker build --rm --tag looker2datacatalog .
docker run --rm --tty -v <YOUR-CREDENTIALS_FILES_FOLDER>:/data \
  looker2datacatalog \ 
  --datacatalog-project-id <YOUR-DATACATALOG-PROJECT-ID> \
  --looker-credentials-file /data/looker2dc-looker-credentials.ini

4. Developer environment

4.1. Install and run Yapf formatter

pip install --upgrade yapf

# Auto update files
yapf --in-place --recursive src tests

# Show diff
yapf --diff --recursive src tests

# Set up pre-commit hook
# From the root of your git project.
curl -o pre-commit.sh https://raw.githubusercontent.com/google/yapf/master/plugins/pre-commit.sh
chmod a+x pre-commit.sh
mv pre-commit.sh .git/hooks/pre-commit

4.2. Install and run Flake8 linter

pip install --upgrade flake8
flake8 src tests

4.3. Run Tests

python setup.py test

4.4. Additional resources

Please refer to the Developer Resources documentation.

5. Troubleshooting

5.1. Looker APIs compatibility

The connector may fail during the scrape stage if the Looker APIs do not return metadata in the expected format. The code base uses the init31 looker_sdk client.
As a reference, the below versions were already validated:

VERSION	RESULT
Looker API 3.1	SUCCESS

5.2. Data Catalog quota

In the case a connector execution hits Data Catalog quota limit, an error will be raised and logged with the following detailment, depending on the performed operation READ/WRITE/SEARCH:

status = StatusCode.RESOURCE_EXHAUSTED
details = "Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'."
debug_error_string = 
"{"created":"@1587396969.506556000", "description":"Error received from peer ipv4:172.217.29.42:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'.","grpc_status":8}"

For more information on Data Catalog quota, please refer to: Data Catalog quota docs.