Extract your metadata assets.
Project description
Castor Extractor
This library contains utilities to extract your metadata assets into JSON
or CSV
files, on your local machine.
After extraction, those files can be pushed to Castor for ingestion.
-
Visualization assets are typically:
dashboards
users
folders
- ...
-
Warehouse assets are typically:
databases
schemas
tables
columns
queries
- ...
It also embeds utilities to help you push your metadata to Castor:
File Checker
to validate your generic CSV files before pushing to CastorUploader
to push extracted files to our Google-Cloud-Storage (GCS)
Table of contents
- Installation
- Documentation
- Contact us
Installation
Requirements: python3.7+
Create castor-env
We advise to create a dedicated Python environment.
Here's an example using Pyenv
and Python 3.7.9
:
- Install Pyenv
brew install pyenv
brew install pyenv-virtualenv
- [optional] Update your
.bashrc
if you encounter this issue
eval "$(pyenv init -)"
eval "$(pyenv init --path)"
eval "$(pyenv virtualenv-init -)"
- [optional] Install python 3.7+
pyenv versions # check your local python installations
pyenv install -v 3.7.9 # if none of the installed versions satisfy requirements 3.7+
- Create your virtual env
pyenv virtualenv 3.7.9 castor-env # create a dedicated env
pyenv shell castor-env # activate the environment
# optional checks
python --version # should be `3.7.9`
pyenv version # should be `castor-env`
PIP install
⚠️ castor-env
must be created AND activated first.
pyenv shell castor-env
(castor-env) $ # this means the environment is now active
ℹ️ please upgrade PIP
before installing Castor.
pip install --upgrade pip
Run the following command to install castor-extractor
:
pip install castor-extractor
Depending on your use-case, you can also install one of the following extras
:
pip install castor-extractor[looker]
pip install castor-extractor[tableau]
pip install castor-extractor[bigquery]
pip install castor-extractor[redshift]
pip install castor-extractor[snowflake]
Create the output directory
mkdir /tmp/castor
You will provide this path in extraction
scripts as following:
castor-extract-bigquery --output=/tmp/castor
Alternatively, you can also set the following ENV
in your bashrc
:
export CASTOR_OUTPUT_DIRECTORY="/tmp/castor"
Contact
For any questions or bug report, contact us at support@castordoc.com
Castor helps you find, understand, use your data assets
Changelog
0.0.12 - 2022-05-06
- Improved the file checker to detect repeated quotes in CSV.
0.0.11 - 2022-05-02
- Fix import error in
file_checker
script
0.0.10 - 2022-04-27
- Snowflake: discard 11 more
query_type
values when fetching queries
0.0.9 - 2022-04-13
- allow Looker parameters
CASTOR_LOOKER_TIMEOUT_SECOND
andCASTOR_LOOKER_PAGE_SIZE
to be passed through environment variables - fix import paths in
castor_extractor/commands/upload.py
script - use
storage.Client.from_service_account_info
whencredentials
is a dictionary inuploader/upload.py
0.0.8 - 2022-04-07
Fix links to documentation in the README
0.0.7 - 2022-04-05
Fix dateutil import issue
0.0.6 - 2022-04-05
First version of Castor Extractor, including:
-
Warehouse assets extraction
- BigQuery
- Postgres
- Redshift
- Snowflake
-
Visualization assets extraction
- Looker
- Metabase
- Mode Analytics
- Tableau
-
Utilities
- Uploader to cloud-storage
- File Checker (for generic metadata)
End-User License Agreement (EULA) of "Castor Extractor"
This End-User License Agreement ("EULA") is a legal agreement between you and Castor. Our EULA was created by EULA Template (https://www.eulatemplate.com) for "Castor Extractor".
This EULA agreement governs your acquisition and use of our "Castor Extractor" software ("Software") directly from Castor or indirectly through a Castor authorized reseller or distributor (a "Reseller").
Please read this EULA agreement carefully before completing the installation process and using the "Castor Extractor" software. It provides a license to use the "Castor Extractor" software and contains warranty information and liability disclaimers.
If you register for a free trial of the "Castor Extractor" software, this EULA agreement will also govern that trial. By clicking "accept" or installing and/or using the "Castor Extractor" software, you are confirming your acceptance of the Software and agreeing to become bound by the terms of this EULA agreement.
If you are entering into this EULA agreement on behalf of a company or other legal entity, you represent that you have the authority to bind such entity and its affiliates to these terms and conditions. If you do not have such authority or if you do not agree with the terms and conditions of this EULA agreement, do not install or use the Software, and you must not accept this EULA agreement.
This EULA agreement shall apply only to the Software supplied by Castor herewith regardless of whether other software is referred to or described herein. The terms also apply to any Castor updates, supplements, Internet-based services, and support services for the Software, unless other terms accompany those items on delivery. If so, those terms apply.
License Grant
Castor hereby grants you a personal, non-transferable, non-exclusive licence to use the "Castor Extractor" software on your devices in accordance with the terms of this EULA agreement.
You are permitted to load the "Castor Extractor" software (for example a PC, laptop, mobile or tablet) under your control. You are responsible for ensuring your device meets the minimum requirements of the "Castor Extractor" software.
You are not permitted to:
- Edit, alter, modify, adapt, translate or otherwise change the whole or any part of the Software nor permit the whole or any part of the Software to be combined with or become incorporated in any other software, nor decompile, disassemble or reverse engineer the Software or attempt to do any such things
- Reproduce, copy, distribute, resell or otherwise use the Software for any commercial purpose
- Allow any third party to use the Software on behalf of or for the benefit of any third party
- Use the Software in any way which breaches any applicable local, national or international law
- use the Software for any purpose that Castor considers is a breach of this EULA agreement
Intellectual Property and Ownership
Castor shall at all times retain ownership of the Software as originally downloaded by you and all subsequent downloads of the Software by you. The Software (and the copyright, and other intellectual property rights of whatever nature in the Software, including any modifications made thereto) are and shall remain the property of Castor.
Castor reserves the right to grant licences to use the Software to third parties.
Termination
This EULA agreement is effective from the date you first use the Software and shall continue until terminated. You may terminate it at any time upon written notice to Castor.
It will also terminate immediately if you fail to comply with any term of this EULA agreement. Upon such termination, the licenses granted by this EULA agreement will immediately terminate and you agree to stop all access and use of the Software. The provisions that by their nature continue and survive will survive any termination of this EULA agreement.
Governing Law
This EULA agreement, and any dispute arising out of or in connection with this EULA agreement, shall be governed by and construed in accordance with the laws of France.
End-User License Agreement (EULA) of "Castor Extractor"
This End-User License Agreement ("EULA") is a legal agreement between you and Castor. Our EULA was created by EULA Template (https://www.eulatemplate.com) for "Castor Extractor".
This EULA agreement governs your acquisition and use of our "Castor Extractor" software ("Software") directly from Castor or indirectly through a Castor authorized reseller or distributor (a "Reseller").
Please read this EULA agreement carefully before completing the installation process and using the "Castor Extractor" software. It provides a license to use the "Castor Extractor" software and contains warranty information and liability disclaimers.
If you register for a free trial of the "Castor Extractor" software, this EULA agreement will also govern that trial. By clicking "accept" or installing and/or using the "Castor Extractor" software, you are confirming your acceptance of the Software and agreeing to become bound by the terms of this EULA agreement.
If you are entering into this EULA agreement on behalf of a company or other legal entity, you represent that you have the authority to bind such entity and its affiliates to these terms and conditions. If you do not have such authority or if you do not agree with the terms and conditions of this EULA agreement, do not install or use the Software, and you must not accept this EULA agreement.
This EULA agreement shall apply only to the Software supplied by Castor herewith regardless of whether other software is referred to or described herein. The terms also apply to any Castor updates, supplements, Internet-based services, and support services for the Software, unless other terms accompany those items on delivery. If so, those terms apply.
License Grant
Castor hereby grants you a personal, non-transferable, non-exclusive licence to use the "Castor Extractor" software on your devices in accordance with the terms of this EULA agreement.
You are permitted to load the "Castor Extractor" software (for example a PC, laptop, mobile or tablet) under your control. You are responsible for ensuring your device meets the minimum requirements of the "Castor Extractor" software.
You are not permitted to:
- Edit, alter, modify, adapt, translate or otherwise change the whole or any part of the Software nor permit the whole or any part of the Software to be combined with or become incorporated in any other software, nor decompile, disassemble or reverse engineer the Software or attempt to do any such things
- Reproduce, copy, distribute, resell or otherwise use the Software for any commercial purpose
- Allow any third party to use the Software on behalf of or for the benefit of any third party
- Use the Software in any way which breaches any applicable local, national or international law
- use the Software for any purpose that Castor considers is a breach of this EULA agreement
Intellectual Property and Ownership
Castor shall at all times retain ownership of the Software as originally downloaded by you and all subsequent downloads of the Software by you. The Software (and the copyright, and other intellectual property rights of whatever nature in the Software, including any modifications made thereto) are and shall remain the property of Castor.
Castor reserves the right to grant licences to use the Software to third parties.
Termination
This EULA agreement is effective from the date you first use the Software and shall continue until terminated. You may terminate it at any time upon written notice to Castor.
It will also terminate immediately if you fail to comply with any term of this EULA agreement. Upon such termination, the licenses granted by this EULA agreement will immediately terminate and you agree to stop all access and use of the Software. The provisions that by their nature continue and survive will survive any termination of this EULA agreement.
Governing Law
This EULA agreement, and any dispute arising out of or in connection with this EULA agreement, shall be governed by and construed in accordance with the laws of France.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for castor_extractor-0.0.12-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca96509e6072b0c02f8acf667c7146762ca9f703914adeba5585be4f79d259ef |
|
MD5 | c057e5b37f3c4f4f34c93391ef96d50b |
|
BLAKE2b-256 | 7ad295ba8e556470f45f956329bfe0276f30e4e10e7583c0026a0056194937b4 |