castor-extractor 0.16.3

Extract your metadata assets.

Castor Extractor

This library contains utilities to extract your metadata assets into JSON or CSV files, on your local machine. After extraction, those files can be pushed to Castor for ingestion.

• Visualization assets are typically:

  • dashboards
  • users
  • folders
  • ...

• Warehouse assets are typically:

  • databases
  • schemas
  • tables
  • columns
  • queries
  • ...

It also embeds utilities to help you push your metadata to Castor:

• File Checker to validate your generic CSV files before pushing to Castor
• Uploader to push extracted files to our Google-Cloud-Storage (GCS)

Table of contents

• Castor Extractor
  • Table of contents
  • Installation
    • Create castor-env
    • PIP install
    • Create the output directory
  • Contact

Installation

Requirements: python3.8+

Create castor-env

We advise to create a dedicated Python environment.

Here's an example using Pyenv and Python 3.8.12:

• Install Pyenv

brew install pyenv
brew install pyenv-virtualenv

• [optional] Update your .bashrc if you encounter this issue

eval "$(pyenv init -)"
eval "$(pyenv init --path)"
eval "$(pyenv virtualenv-init -)"

• [optional] Install python 3.8+

pyenv versions # check your local python installations

pyenv install -v 3.8.12 # if none of the installed versions satisfy requirements 8+

• Create your virtual env

pyenv virtualenv 3.8.12 castor-env # create a dedicated env
pyenv shell castor-env # activate the environment

# optional checks
python --version # should be `3.8.12`
pyenv version # should be `castor-env`

PIP install

⚠️ castor-env must be created AND activated first.

pyenv shell castor-env
(castor-env) $ # this means the environment is now active

ℹ️ please upgrade PIP before installing Castor.

pip install --upgrade pip

Run the following command to install castor-extractor:

pip install castor-extractor

Depending on your use case, you can also install one of the following extras:

pip install castor-extractor[bigquery]
pip install castor-extractor[looker]
pip install castor-extractor[metabase]
pip install castor-extractor[mysql]
pip install castor-extractor[powerbi]
pip install castor-extractor[qlik]
pip install castor-extractor[postgres]
pip install castor-extractor[redshift]
pip install castor-extractor[snowflake]
pip install castor-extractor[sqlserver]
pip install castor-extractor[tableau]

Create the output directory

mkdir /tmp/castor

You will provide this path in the extraction scripts as follows:

castor-extract-bigquery --output=/tmp/castor

Alternatively, you can also set the following ENV in your bashrc:

export CASTOR_OUTPUT_DIRECTORY="/tmp/castor"

Contact

For any questions or bug report, contact us at [e-mail]

Castor helps you find, understand, use your data assets