datacatalog-util 0.5.2

A package to manage Google Cloud Data Catalog helper commands and scripts

Datacatalog Util

A Python package to manage Google Cloud Data Catalog helper commands and scripts.

Disclaimer: This is not an officially supported Google product.

Commands List

Group	Command	Description	Documentation Link	Code Repo
tags	create	Load Tags from CSV file.	GO	GO
tags	export	Export Tags to CSV file.	GO	GO
tag-templates	create	Load Templates from CSV file.	GO	GO
tag-templates	delete	Delete Templates from CSV file.	GO	GO
tag-templates	export	Export Templates to CSV file.	GO	GO
filesets	enrich	Enrich GCS filesets with Tags.	GO	GO
filesets	create-template	Create Filesets Template in chosen Project.	GO	GO
filesets	clean-up-templates-and-tags	Cleans up Fileset Templates and Tags.	GO	GO
filesets	clean-up-all	Clean up Fileset Entries, Their Tags and Templates.	GO	GO

Executing in Cloud Shell

# Set your SERVICE ACCOUNT, for instructions go to 1.3. Auth credentials
# This name is just a suggestion, feel free to name it following your naming conventions
export GOOGLE_APPLICATION_CREDENTIALS=~/datacatalog-util-sa.json

# Install datacatalog-util 
pip3 install datacatalog-util --user

# Add to your PATH
export PATH=~/.local/bin:$PATH

# Look for available commands
datacatalog-util --help

1. Environment setup

1.1. Python + virtualenv

Using virtualenv is optional, but strongly recommended unless you use Docker.

1.1.1. Install Python 3.6+

1.1.2. Get the source code

git clone https://github.com/mesmacosta/datacatalog-util
cd ./datacatalog-util

All paths starting with ./ in the next steps are relative to the datacatalog-util folder.

1.1.3. Create and activate an isolated Python environment

pip install --upgrade virtualenv
python3 -m virtualenv --python python3 env
source ./env/bin/activate

1.1.4. Install the package

pip install --upgrade .

1.2. Docker

Docker may be used as an alternative to run the script. In this case, please disregard the Virtualenv setup instructions.

1.3. Auth credentials

1.3.1. Create a service account and grant it below roles

• Data Catalog Admin
• Storage Admin

1.3.2. Download a JSON key and save it as

This name is just a suggestion, feel free to name it following your naming conventions

• ./credentials/datacatalog-util-sa.json

1.3.3. Set the environment variables

This step may be skipped if you're using Docker.

export GOOGLE_APPLICATION_CREDENTIALS=~/credentials/datacatalog-util-sa.json

2. Load Tags from CSV file

2.1. Create a CSV file representing the Tags to be created

Tags are composed of as many lines as required to represent all of their fields. The columns are described as follows:

Column	Description	Mandatory
linked_resource	Full name of the asset the Entry refers to.	Y
template_name	Resource name of the Tag Template for the Tag.	Y
column	Attach Tags to a column belonging to the Entry schema.	N
field_id	Id of the Tag field.	Y
field_value	Value of the Tag field.	Y

TIPS

• sample-input/create-tags for reference;
• Data Catalog Sample Tags (Google Sheets) may help to create/export the CSV.

2.2. Run the datacatalog-util script

• Python + virtualenv

datacatalog-util tags create --csv-file CSV_FILE_PATH

• Docker

docker build --rm --tag datacatalog-util .
docker run --rm --tty \
  --volume CREDENTIALS_FILE_FOLDER:/credentials --volume CSV_FILE_FOLDER:/data \
  datacatalog-util create-tags --csv-file /data/CSV_FILE_NAME

3. Export Tags to CSV file

3.1. A list of CSV files, each representing one Template will be created.

One file with summary with stats about each template, will also be created on the same directory.

The columns for the summary file are described as follows:

Column	Description
template_name	Resource name of the Tag Template for the Tag.
tags_count	Number of tags found from the template.
tagged_entries_count	Number of tagged entries with the template.
tagged_columns_count	Number of tagged columns with the template.
tag_string_fields_count	Number of used String fields on tags of the template.
tag_bool_fields_count	Number of used Bool fields on tags of the template.
tag_double_fields_count	Number of used Double fields on tags of the template.
tag_timestamp_fields_count	Number of used Timestamp fields on tags of the template.
tag_enum_fields_count	Number of used Enum fields on tags of the template.

The columns for each template file are described as follows:

Column	Description
relative_resource_name	Full resource name of the asset the Entry refers to.
linked_resource	Full name of the asset the Entry refers to.
template_name	Resource name of the Tag Template for the Tag.
tag_name	Resource name of the Tag.
column	Attach Tags to a column belonging to the Entry schema.
field_id	Id of the Tag field.
field_type	Type of the Tag field.
field_value	Value of the Tag field.

3.2. Run the datacatalog-util script

• Python + virtualenv

datacatalog-util tags export --project-ids my-project --dir-path DIR_PATH

3.3 Run the datacatalog-tag-exporter filtering Tag Templates

• Python + virtualenv

datacatalog-util tags export --project-ids my-project \
--dir-path DIR_PATH \
--tag-templates-names projects/my-project/locations/us-central1/tagTemplates/my-template,\
projects/my-project/locations/us-central1/tagTemplates/my-template-2 

4. Load Templates from CSV file

4.1. Create a CSV file representing the Templates to be created

Templates are composed of as many lines as required to represent all of their fields. The columns are described as follows:

Column	Description	Mandatory
template_name	Resource name of the Tag Template for the Tag.	Y
display_name	Resource name of the Tag Template for the Tag.	Y
field_id	Id of the Tag Template field.	Y
field_display_name	Display name of the Tag Template field.	Y
field_type	Type of the Tag Template field.	Y
enum_values	Values for the Enum field.	N

4.2. Run the datacatalog-util script - Create the Tag Templates

• Python + virtualenv

datacatalog-util tag-templates create --csv-file CSV_FILE_PATH

4.3. Run the datacatalog-util script - Delete the Tag Templates

• Python + virtualenv

datacatalog-util tag-templates delete --csv-file CSV_FILE_PATH

TIPS

• sample-input/create-tag-templates for reference;

5. Export Templates to CSV file

5.1. A CSV file representing the Templates will be created

Templates are composed of as many lines as required to represent all of their fields. The columns are described as follows:

Column	Description
template_name	Resource name of the Tag Template for the Tag.
display_name	Resource name of the Tag Template for the Tag.
field_id	Id of the Tag Template field.
field_display_name	Display name of the Tag Template field.
field_type	Type of the Tag Template field.
enum_values	Values for the Enum field.

5.2. Run the datacatalog-util script

• Python + virtualenv

datacatalog-util tag-templates export --project-ids my-project --file-path CSV_FILE_PATH

6. Filesets Commands

6.1. Enrich GCS Filesets with Tags

Users are able to choose the Tag fields from the list provided at Tags

datacatalog-util filesets --project-ids my-project enrich

6.1.1 Enrich all fileset entries using Tag Template from a different Project (Good way to reuse the same Template)

Check step below to see how to create the template in a different project.

datacatalog-util filesets --project-id my_project \
  enrich --tag-template-name projects/my_different_project/locations/us-central1/tagTemplates/fileset_enricher_findings

6.2. Create Fileset Enricher Tag Template in a different Project

Creates the fileset enricher template in chosen project.

datacatalog-util filesets --project-id my_different_project \
    create-template

6.3. clean up template and tags

Cleans up the Template and Tags from the Fileset Entries, running the main command will recreate those.

datacatalog-util filesets --project-ids my-project clean-up-templates-and-tags 

6.4. clean up all (Non Reversible, be careful)

Cleans up the Fileset Entries, Template and Tags. You have to re create the Fileset entries if you need to restore the state, which is outside the scope of this script.

datacatalog-util filesets --project-ids my-project clean-up-all

History

0.1.0 (2020-04-08)

• First release on PyPI.

0.1.1 (2020-04-09)

• Update description.

0.2.0 (2020-04-09)

• Improved groups command line.

0.3.0 (2020-04-15)

• Added datacatalog-tag-template-processor library.

0.4.0 (2020-04-15)

• Added datacatalog-tag-template-exporter library.

0.5.0 (2020-04-15)

• Added datacatalog-tag-exporter library.