Extract your metadata assets.
Project description
Castor Extractor
This library contains utilities to extract your metadata assets into JSON
or CSV
files, on your local machine.
After extraction, those files can be pushed to Castor for ingestion.
-
Visualization assets are typically:
dashboards
users
folders
- ...
-
Warehouse assets are typically:
databases
schemas
tables
columns
queries
- ...
It also embeds utilities to help you push your metadata to Castor:
File Checker
to validate your generic CSV files before pushing to CastorUploader
to push extracted files to our Google-Cloud-Storage (GCS)
Table of contents
Installation
Requirements: python3.8+
Create castor-env
We advise to create a dedicated Python environment.
Here's an example using Pyenv
and Python 3.8.12
:
- Install Pyenv
brew install pyenv
brew install pyenv-virtualenv
- [optional] Update your
.bashrc
if you encounter this issue
eval "$(pyenv init -)"
eval "$(pyenv init --path)"
eval "$(pyenv virtualenv-init -)"
- [optional] Install python 3.8+
pyenv versions # check your local python installations
pyenv install -v 3.8.12 # if none of the installed versions satisfy requirements 8+
- Create your virtual env
pyenv virtualenv 3.8.12 castor-env # create a dedicated env
pyenv shell castor-env # activate the environment
# optional checks
python --version # should be `3.8.12`
pyenv version # should be `castor-env`
PIP install
⚠️ castor-env
must be created AND activated first.
pyenv shell castor-env
(castor-env) $ # this means the environment is now active
ℹ️ please upgrade PIP
before installing Castor.
pip install --upgrade pip
Run the following command to install castor-extractor
:
pip install castor-extractor
Depending on your use-case, you can also install one of the following extras
:
pip install castor-extractor[looker]
pip install castor-extractor[tableau]
pip install castor-extractor[metabase]
pip install castor-extractor[qlik]
pip install castor-extractor[bigquery]
pip install castor-extractor[redshift]
pip install castor-extractor[snowflake]
Create the output directory
mkdir /tmp/castor
You will provide this path in extraction
scripts as following:
castor-extract-bigquery --output=/tmp/castor
Alternatively, you can also set the following ENV
in your bashrc
:
export CASTOR_OUTPUT_DIRECTORY="/tmp/castor"
Contact
For any questions or bug report, contact us at support@castordoc.com
Castor helps you find, understand, use your data assets
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for castor_extractor-0.11.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 69e2cdc34736ca75280ce1e06e61a33fd2338f11b043e783d2fc8f7f95ae4226 |
|
MD5 | 5e841e32e60b43e58c5c3b0f99ca80bd |
|
BLAKE2b-256 | b6726fa340a1b23104c1160f9298743854101f580c0fa27fe407098ae56f11d0 |