revefi-ingestion

Revefi's customer-side metadata ingestion library and CLI. It runs an OpenMetadata connector inside your network, under your own BI-tool admin credentials, and streams discovered metadata to Revefi's ingestion API over HTTPS. Your credentials never leave your environment — Revefi never sees your Looker password, client secret, or GitHub token.

It is built on top of the published openmetadata-ingestion framework (pulled in as a dependency, not forked), so this repository contains only Revefi's own code.

The first supported source is Looker (dashboards, looks, explores, and warehouse-table lineage).

Install

The package pulls in the OpenMetadata ingestion framework, so install it into an isolated environment — not your project/shared venv. Recommended: pipx:

pipx install "revefi-ingestion==0.1.1"

Or a dedicated venv:

python -m venv ~/.revefi-ingestion
~/.revefi-ingestion/bin/pip install "revefi-ingestion==0.1.1"

Requires Python 3.10–3.11. Pin the exact version so an upgrade is always a deliberate step.

Use as a library (recommended)

Import the module and run ingestion from your own Python script. run() returns a RunResult on success and raises RevefiIngestionError on failure (after a best-effort abort is delivered):

from revefi_ingestion import (
    LineageOptions,
    LookerCredentials,
    LookerIngestion,
    RevefiIngestionError,
    RevefiTarget,
)

ingestion = LookerIngestion(
    target=RevefiTarget(
        api_url="https://ingest.revefi.com",
        token="<token-from-revefi>",
        # Optional: also write every payload sent to a JSONL file (see "Inspecting what's sent").
        # dump_file="./looker-run.jsonl",
    ),
    looker=LookerCredentials(
        host="https://yourco.cloud.looker.com",
        client_id="<looker-api3-client-id>",
        client_secret="<looker-api3-client-secret>",
    ),
    # Optional: resolve warehouse-table lineage from your LookML (omit for metadata-only).
    lineage=LineageOptions(
        github_repo="yourorg/your-lookml-repo",
        github_token="<github-read-token>",
        db_service_prefixes=["bq_prod"],
        warehouse_type="BigQuery",
    ),
)

try:
    result = ingestion.run()
    print(f"{result.run_id}: {result.status}")
    print(f"  entities: {result.entity_count} in {result.entity_batch_count} batch(es)")
    print(f"  edges:    {result.edge_count} in {result.lineage_batch_count} batch(es)")
    print(f"  failed:   {result.records_failed}")
    for step in result.steps:
        print(f"  {step.name}: {step.processed} processed, {step.failed} failed")
except RevefiIngestionError as exc:
    # Message is already redacted of secrets; the failure was logged and aborted server-side.
    raise SystemExit(f"ingestion failed: {exc}")

dump_file and lineage are both optional. See What a run reports for the full RunResult and Inspecting what's sent for the dump file. A complete, environment-driven version of this script lives in examples/run_ingestion.py.

A metadata-only run is just the target + looker arguments:

LookerIngestion(
    target=RevefiTarget(api_url="https://ingest.revefi.com", token="<token-from-revefi>"),
    looker=LookerCredentials(host="...", client_id="...", client_secret="..."),
).run()

Use as a CLI

A revefi-ingestion console script wraps the same library.

Metadata-only (no lineage):

revefi-ingestion looker \
  --api-url   https://ingest.revefi.com \
  --token     <token-from-revefi> \
  --looker-host   https://yourco.cloud.looker.com \
  --client-id     <looker-api3-client-id> \
  --client-secret <looker-api3-client-secret>

With warehouse-table lineage (BigQuery example):

revefi-ingestion looker \
  --api-url   https://ingest.revefi.com \
  --token     <token-from-revefi> \
  --looker-host   https://yourco.cloud.looker.com \
  --client-id     <looker-api3-client-id> \
  --client-secret <looker-api3-client-secret> \
  --github-repo   yourorg/your-lookml-repo \
  --github-token  <github-read-token> \
  --db-service-prefix bq_prod \
  --warehouse-type BigQuery

• The --github-* and --db-service-prefix flags are required to resolve warehouse-table lineage from your LookML; omit them for metadata-only ingestion.
• --warehouse-type (default BigQuery; one of BigQuery, Snowflake, Redshift, Databricks) sets the SQL dialect used to parse your LookML sql_table_name references. Set it to match the warehouse your Looker models read from — a wrong dialect silently produces incorrect lineage.

Secrets via environment variables

The three secret flags can be supplied as environment variables instead of on the command line, so they don't end up in your shell history or process list:

Flag	Environment variable
--token	REVEFI_INGEST_TOKEN
--client-secret	LOOKER_CLIENT_SECRET
--github-token	GITHUB_TOKEN

If both a flag and its environment variable are set, the flag wins. Run revefi-ingestion looker --help for the full flag list.

Optional tuning & diagnostics

revefi-ingestion looker \
  --api-url https://ingest.revefi.com --token <token-from-revefi> \
  --looker-host https://yourco.cloud.looker.com \
  --client-id <id> --client-secret <secret> \
  --dump-file ./looker-run.jsonl \
  --log-level DEBUG \
  --connect-timeout 30 --read-timeout 60 \
  --retry-total 5 --retry-backoff 2.0 \
  --max-batch-entities 100 --max-batch-bytes 204800

• --dump-file PATH — also write every sent payload to a JSONL file (see below).
• --log-level — DEBUG | INFO | WARNING | ERROR (default INFO).
• --connect-timeout / --read-timeout — HTTP timeouts, in seconds.
• --retry-total / --retry-backoff — retry budget for transient failures.
• --max-batch-entities / --max-batch-bytes — caps per streamed request.

All are optional and default sensibly.

Inspecting what's sent

Pass --dump-file PATH (CLI) or dump_file="PATH" on RevefiTarget (library) to also write every payload to a JSONL file. It is optional and additive — the run still streams to the API; omit it and nothing is written.

The first line is a header (the target URL the payloads are POSTed to, plus run metadata); each following line is one request, serialized identically to the POST body (camelCase, unset fields omitted; the wire body is this JSON gzipped) — including the message that fails, if one does:

{"type":"run_header","target_url":"https://ingest.revefi.com/api/v1/discovery/ingest","run_id":"3f…","cli_version":"0.1.1","written_at":"2026-…"}
{"runId":"3f…","messageSequenceNo":0,"messageType":"BEGIN_RUN","beginRun":{"startTime":"2026-…"}}
{"runId":"3f…","messageSequenceNo":1,"messageType":"ENTITY_BATCH","entityBatch":{"entities":[{"entityType":"CHART","entityJson":"{…}"}]}}
{"runId":"3f…","messageSequenceNo":2,"messageType":"END_ENTITY_RUN","endEntityRun":{"clientEmittedEntityCount":412,"clientEmittedEntityBatchCount":5}}

entityJson is the OpenMetadata Create*Request / AddLineageRequest as a nested JSON string, so decode it too when inspecting an entity:

# message types, in send order:
jq -r 'if .type=="run_header" then "→ "+.target_url else "\(.messageSequenceNo)\t\(.messageType)" end' looker-run.jsonl

# the OM entities inside the first entity batch:
jq -r 'select(.messageType=="ENTITY_BATCH") | .entityBatch.entities[].entityJson' looker-run.jsonl | head -1 | jq .

What a run reports

A run logs progress as it goes (each batch as it's sent, with running totals) and ends with a summary; the library's RunResult carries the same numbers:

INFO  Sent entity batch 5: 100 entities (412 entities total)
INFO  Sent lineage batch 3: 20 edges (120 edges total)
INFO  Ingestion run 3f… finished: SUCCESS
INFO    entities published: 412 in 5 batch(es)
INFO    lineage edges published: 120 in 3 batch(es)
INFO    LookerSource: 410 processed, 2 failed, 8 filtered, 0 warning(s)

RunResult fields:

Field	Meaning
run_id	Unique id for this run
status	SUCCESS, or PARTIAL_SUCCESS if some records failed
entity_count / entity_batch_count	Entities published, and in how many batches
edge_count / lineage_batch_count	Lineage edges published, and in how many batches
records_failed	Records the workflow could not process (each logged at WARNING)
steps	Per-step tally: name, processed, failed, filtered, warnings

Individual record failures are not fatal: they're logged (redacted) and reported via records_failed / steps, and the run still publishes what succeeded. A failure that aborts the whole run (auth, connectivity, or a run that produced nothing) raises RevefiIngestionError after a best-effort abort is delivered to the server.

How your Looker instance is identified

You don't choose an OpenMetadata "service name" — every run records its Looker metadata under a fixed internal name, revefi_looker. This is deliberate: Looker exposes no instance, account, or "service" name to read back (unlike Tableau's site or Snowflake's account), so there is nothing to name it after. A Looker deployment is identified by its host URL.

Your runs are still told apart — by two things, neither of which is that service name:

• Your ingest token (--token) — Revefi resolves your tenant and integration from the token, server-side. The CLI never sends, and never even knows, a tenant or integration id.
• Your Looker host (--looker-host) — recorded on every message, identifying which Looker instance the run came from.

So revefi_looker is only an internal namespace label; the identity that actually scopes your data (who you are, and which Looker instance this is) travels on the token and the host — not on the service name.

How it works

revefi-ingestion drives OpenMetadata's Looker connector with no OpenMetadata server: a set of monkey-patches replaces the server-bound calls (entity lookups, owner resolution, usage percentiles) with synthetic entities, and a custom sink (revefi-http) batches the connector's Create*Request / AddLineageRequest records and POSTs them to Revefi's ingestion API. The revefi-http sink type is resolved to the bundled sink by patching OpenMetadata's sink importer, since OpenMetadata is an installed dependency and owns the metadata.ingestion.sink namespace.

Development

python -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'
pytest tests/unit -q

License

Apache License 2.0. See LICENSE. OpenMetadata is used as a dependency under its own license.