A package to manage Google Cloud Data Catalog helper commands and scripts
Project description
A Python package to manage Google Cloud Data Catalog helper commands and scripts.
Disclaimer: This is not an officially supported Google product.
1. Environment setup
1.1. Python + virtualenv
Using virtualenv is optional, but strongly recommended unless you use Docker.
1.1.1. Install Python 3.6+
1.1.2. Create a folder
This is recommended so all related stuff will reside at same place, making it easier to follow below instructions.
mkdir ./datacatalog-util
cd ./datacatalog-util
All paths starting with ``./`` in the next steps are relative to the ``utilsr`` folder.
1.1.3. Create and activate an isolated Python environment
pip install --upgrade virtualenv
python3 -m virtualenv --python python3 env
source ./env/bin/activate
1.1.4. Install the package
pip install --upgrade .
1.2. Docker
Docker may be used as an alternative to run the script. In this case, please disregard the Virtualenv setup instructions.
1.2.1. Get the source code
git clone https://github.com/mesmacosta/datacatalog-util
cd ./datacatalog-util
1.3. Auth credentials
1.3.1. Create a service account and grant it below roles
BigQuery Metadata Viewer
Data Catalog Admin
A custom role with bigquery.datasets.updateTag and bigquery.tables.updateTag permissions
1.3.2. Download a JSON key and save it as
./credentials/datacatalog-util.json
1.3.3. Set the environment variables
This step may be skipped if you’re usingDocker.
export GOOGLE_APPLICATION_CREDENTIALS=~/credentials/datacatalog-util.json
4. Load Templates from CSV file
4.1. Create a CSV file representing the Templates to be created
Templates are composed of as many lines as required to represent all of their fields. The columns are described as follows:
Column |
Description |
Mandatory |
---|---|---|
template_name |
Resource name of the Tag Template for the Tag. |
Y |
display_name |
Resource name of the Tag Template for the Tag. |
Y |
field_id |
Id of the Tag Template field. |
Y |
field_display_name |
Display name of the Tag Template field. |
Y |
field_type |
Type of the Tag Template field. |
Y |
enum_values |
Values for the Enum field. |
N |
4.2. Run the datacatalog-util script - Create the Tag Templates
Python + virtualenv
datacatalog-util create-tag-templates --csv-file CSV_FILE_PATH
4.3. Run the datacatalog-util script - Delete the Tag Templates
Python + virtualenv
datacatalog-util delete-tag-templates --csv-file CSV_FILE_PATH
TIPS - sample-input/create-tag-templates for reference;
5. Export Templates to CSV file
5.1. A CSV file representing the Templates will be created
Templates are composed of as many lines as required to represent all of their fields. The columns are described as follows:
Column |
Description |
---|---|
template_name |
Resource name of the Tag Template for the Tag. |
display_name |
Resource name of the Tag Template for the Tag. |
field_id |
Id of the Tag Template field. |
field_display_name |
Display name of the Tag Template field. |
field_type |
Type of the Tag Template field. |
enum_values |
Values for the Enum field. |
5.2. Run the datacatalog-util script
Python + virtualenv
datacatalog-util export-tag-templates --project-ids my-project --file-path CSV_FILE_PATH
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for datacatalog_util-0.1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 579562ff70ead47419f2128ea79fe2175dedad7f26ed70913cf23c4dbd56ebca |
|
MD5 | bb4e663fc90df75e6e2003642e531602 |
|
BLAKE2b-256 | dbbd7e2734c09b64ee977c446e00548419d661a6f9cadd01e59a219242a57c96 |