carte-cli 0.2.3

A static site generator for data catalogs

Carte

Carte is a Python library for extracting metadata from your data sources and generating structured Markdown files from it. It's a companion library to the Carte front end, a lightweight data catalog for fast moving data teams. Carte currently supports the AWS Glue catalog and PostgreSQL but more sources are coming.

The files generated by Carte can be fed into the front end as-is to get a statically generated data catalog.

Installation

pip install carte-cli

If you plan to use PostgreSQL as a data source, you should also install the related optional dependencies using the following command instead of the first one:

pip install carte-cli[postgres]

Usage

AWS Glue

Carte uses the Boto3 library to fetch the Glue catalog. This means you need an AWS CLI profile defined that can access the required databases and tables.

PostgreSQL

For PostgreSQL, you need to construct a connection string with your username, password, the DB's host, and the database you'd like to extract on the host.

Carte CLI works from a YAML config that you have to specify when running an extraction. An example config with both Glue and Postgres:

connections:
  - type: glue
    name: aws_glue
  - type: postgresql
    config:
      connection_string: 'postgresql://<user>:<password>@<host>:<port>/<database>'

If you name this config extract.config.yml, then you can run an extraction with:

carte extract extract.config.yml -o data/datasets

where the -o flag specifies the output folder.