revefi-ingestion 0.1.0b1

Revefi metadata ingestion CLI, built on OpenMetadata's ingestion framework

revefi-ingestion

Revefi's customer-side metadata ingestion CLI. It runs an OpenMetadata connector inside your network, under your own BI-tool admin credentials, and streams discovered metadata to Revefi's ingestion API over HTTPS. Your credentials never leave your environment — Revefi never sees your Looker password, client secret, or GitHub token.

The first supported source is Looker (dashboards, looks, explores, and warehouse-table lineage).

Install

The package bundles the full OpenMetadata ingestion framework, so install it into an isolated environment — not your project/shared venv. Recommended: pipx:

pipx install "revefi-ingestion[looker]==0.1.0-b1"
# private index: pipx install --index-url <your-index> "revefi-ingestion[looker]==0.1.0-b1"

Or a dedicated venv:

python -m venv ~/.revefi-ingestion
~/.revefi-ingestion/bin/pip install "revefi-ingestion[looker]==0.1.0-b1"

Requires Python 3.9–3.11. This is a beta — pin the exact 0.1.0-bN version (or pass --pre).

Usage

Metadata-only (no lineage):

revefi-ingestion looker \
  --api-url   https://ingest.revefi.com \
  --token     <token-from-revefi> \
  --looker-host   https://yourco.cloud.looker.com \
  --client-id     <looker-api3-client-id> \
  --client-secret <looker-api3-client-secret>

With warehouse-table lineage (BigQuery example):

revefi-ingestion looker \
  --api-url   https://ingest.revefi.com \
  --token     <token-from-revefi> \
  --looker-host   https://yourco.cloud.looker.com \
  --client-id     <looker-api3-client-id> \
  --client-secret <looker-api3-client-secret> \
  --github-repo   yourorg/your-lookml-repo \
  --github-token  <github-read-token> \
  --db-service-prefix bq_prod \
  --warehouse-type BigQuery

• The --github-* and --db-service-prefix flags are required to resolve warehouse-table lineage from your LookML; omit them for metadata-only ingestion.
• --warehouse-type (default BigQuery; one of BigQuery, Snowflake, Redshift, Databricks) sets the SQL dialect used to parse your LookML sql_table_name references. Set it to match the warehouse your Looker models read from — a wrong dialect silently produces incorrect lineage.

Secrets via environment variables

The three secret flags can be supplied as environment variables instead of on the command line, so they don't end up in your shell history or process list:

Flag	Environment variable
--token	REVEFI_INGEST_TOKEN
--client-secret	LOOKER_CLIENT_SECRET
--github-token	GITHUB_TOKEN

export REVEFI_INGEST_TOKEN=<token-from-revefi>
export LOOKER_CLIENT_SECRET=<looker-api3-client-secret>
export GITHUB_TOKEN=<github-read-token>      # only needed for warehouse-table lineage

revefi-ingestion looker \
  --api-url     https://ingest.revefi.com \
  --looker-host https://yourco.cloud.looker.com \
  --client-id   <looker-api3-client-id>

If both a flag and its environment variable are set, the flag wins.

Run revefi-ingestion looker --help for the full flag list.

Development

From the repo's ingestion module, with the Python venv active:

cd ingestion
make install_dev_env
python -m pytest tests/unit/revefi -q

To exercise the CLI end-to-end without the Revefi backend, run a standalone mock ingestion server that accepts the wire protocol over https (the client only supports https) and point --api-url at it.