A small example package
This program is used to extract data from an implementation of Collibra into a SQL envrionment. In our case, we are extracting from Gore's instance of Collibra and into EXL_MDSDev. The program will extract the following objects:
- Attribute Types
- Relation Types
- Responsibilities Each of these object types are stored as their own table in SQL.
Instruction For Use
The source executable file is kept in -- Artifact URL --. This folder contains a main.exe file and all of its dependencies. You will need add a 'prod_config.yml' file into the root of this folder. This config file can be found within the source code of the project and should be structured like so:
API_CONFIG: limit: 1000000 AUTH: username: <Valid admin username in environment> password: <Valid admin password in environment> auth-header: <Auto-generated basic auth token, generated using postman> ENVIRONMENT: gore: wlgore-<Envrionment Instance (dev,test,prod)>.collibra.com
Open a cmd prompt in the root of the project folder. Type main.exe and hit enter. The program will start to run and log its progress. During the run, the program will extract all data and overwrite the raw sql tables.
The following sql tables are created/overwritten during the run on this program:
SQL Stored Procedures
The following stored procedures are run on the raw tables to manipulate the data and add batch id's:
All of these procedures may be run simultaneously with the collibra._load_entire_batch procedure.
Migrating from dev/test to prod
In order to change the environment in which the data extracton runs, the prod_config.yml file within the src folder will need to be changed. The username and password will need to be changed to that of a user in the new environment. Additionally the gore environment variable will need to be changed to the prod instance's URL. Example of a correctly configured prod_config.yml file for the prod environment:
API_CONFIG: limit: 1000000 AUTH: username: <Valid admin username in PROD environment> password: <Valid admin password in PROD environment> auth-header: <Auto-generated basic auth token, generated using postman> ENVIRONMENT: gore: wlgore-prod.collibra.com
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for data-extraction-c.lynch278-0.0.7.tar.gz
Hashes for data_extraction_c.lynch278-0.0.7-py3-none-any.whl