Skip to main content

A small example package

Project description



This program is used to extract data from an implementation of Collibra into a SQL envrionment. In our case, we are extracting from Gore's instance of Collibra and into EXL_MDSDev. The program will extract the following objects:

  • Assets
  • Attributes
  • Attribute Types
  • Domains
  • Communities
  • Relations
  • Relation Types
  • Responsibilities Each of these object types are stored as their own table in SQL.

Instruction For Use


The source executable file is kept in -- Artifact URL --. This folder contains a main.exe file and all of its dependencies. You will need add a 'prod_config.yml' file into the root of this folder. This config file can be found within the source code of the project and should be structured like so:

  limit: 1000000
  username: <Valid admin username in environment>
  password: <Valid admin password in environment>
  auth-header: <Auto-generated basic auth token, generated using postman>
  gore: wlgore-<Envrionment Instance (dev,test,prod)>


Open a cmd prompt in the root of the project folder. Type main.exe and hit enter. The program will start to run and log its progress. During the run, the program will extract all data and overwrite the raw sql tables.

SQL Tables

The following sql tables are created/overwritten during the run on this program:

  • collibra_assets_raw
  • collibra_attributes_raw
  • collibra_attribute_types_raw
  • collibra_communities_raw
  • collibra_domains_raw
  • collibra_relations_raw
  • collibra_relation_types_raw
  • collibra_responsibilities_raw

SQL Stored Procedures

The following stored procedures are run on the raw tables to manipulate the data and add batch id's:

  • collibra.load_collibra_assets
  • collibra.load_collirba_attributes
  • collibra.load_collibra_attribute_types
  • collibra.load_collibra_communities
  • collibra.load_collibra_domains
  • collibra.load_collibra_relations
  • collibra.load_collibra_relation_types
  • collibra.load_collibra_responsibilities

All of these procedures may be run simultaneously with the collibra._load_entire_batch procedure.

Migrating from dev/test to prod

In order to change the environment in which the data extracton runs, the prod_config.yml file within the src folder will need to be changed. The username and password will need to be changed to that of a user in the new environment. Additionally the gore environment variable will need to be changed to the prod instance's URL. Example of a correctly configured prod_config.yml file for the prod environment:

  limit: 1000000
  username: <Valid admin username in PROD environment>
  password: <Valid admin password in PROD environment>
  auth-header: <Auto-generated basic auth token, generated using postman>

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

data-extraction-c.lynch278-0.0.7.tar.gz (8.7 kB view hashes)

Uploaded source

Built Distribution

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page