Looker Demo Creator (demo-create)

From zero to production Looker, Dataplex Knowledge Catalog, and Real-Time BigQuery CDC demo in minutes. Automated, deterministic 15-gate orchestrator for creating end-to-end Looker demos, synthetic or adopted BigQuery datasets, Dataplex Universal Catalog (Knowledge Catalog) semantic curations & business glossaries, 3NF Snowflake LookML semantic models, 100% test-verified executive dashboards, Conversational Analytics (CA) Agents published to Gemini Enterprise (GE), private-labeled Embedded Analytics portals, and real-time BigQuery Storage Write API CDC streaming pipelines on locked-down GCE VMs.

Overview: How demo-create Operates

demo-create unifies the full-stack data & analytics demo lifecycle into a deterministic 15-Gate State Machine (Gates 0..14) persisted in .demo-state.json. It supports two primary deployment modes at Gate 1 (confirm-targets):

• --mode full (Default — End-to-End Looker + BigQuery + Catalog + Stream): Executes all 15 gates (0..14) across BigQuery, Dataplex Knowledge Catalog, LookML modeling/polish/optimization/deployment, Conversational Analytics Agent, Gemini Enterprise, Embedded Portal, and Live BigQuery Streaming.
• --mode bq_only (BigQuery + Knowledge Catalog + Stream Only): Executes Gates 0..5 (BigQuery dataset synthesis/adoption + Dataplex Knowledge Catalog curation/seeding), automatically marks Looker Gates 6..13 satisfied, and routes directly to Gate 14 (stream) without requiring Looker credentials.

Core Lifecycle Stages

1. Pre-Flight & Target Confirmation (pre-check, confirm-targets — Gates 0–1): Audits GCP/Looker credentials, prunes deprecated MCP servers/skills, syncs CLI-first skills into ~/.gemini/config/skills/, and records human-confirmed targets (--mode full|bq_only).
2. Schema Co-Design, Modular DAG Synthesis or Dataset Adoption (data — Gates 2–4): Designs 3NF relational schemas (propose-schema --preview + approve-schema) and synthesizes Parquet tables (>7,500 rows/sec) loaded into BigQuery with Day Partitioning & Clustering (generate --upload), or adopts an existing BigQuery dataset (data adopt).
3. Dataplex Knowledge Catalog Semantic Curation (catalog — Gate 5): Seeds or inspects live Dataplex Universal Catalog AspectTypes (semantic-curation), Business Glossaries (<dataset>-glossary), Glossary Terms, definition EntryLinks, and BigQuery PRIMARY KEY / FOREIGN KEY constraints (catalog seed / inspect / sync), persisting .demo-catalog-<dataset>.json.
4. LookML Modeling, 3-Pass Polish, Optimization & QA Release (lookml — Gates 6–10): Generates catalog-enriched LookML views/explores/dashboards (lookml model), enforces 3-Pass Executive Dashboard Polish & SELECT DISTINCT filtered measure grounding (lookml certify-polish), applies server caching/partition optimizations (lookml optimize), validates 100% of dashboard queries before production deployment (lookml deploy), and records post-deploy visual critique approval (lookml approve-critique).
5. Conversational Analytics & Gemini Enterprise (agent / ge — Gates 11–12): Grounds 1:1 Golden Queries (expanded_share_url) from dashboard tiles, links them to a Looker CA Agent (agent create), and publishes to Gemini Enterprise (agent publish).
6. Private-Labeled Embed Portal (embed — Gate 13): Provisions Looker instance embed settings (Service Account auth, allowlist, cookieless v2, 2-step shared folder inheritance, LookML dashboard move, CA agent sharing, brand themes) and scaffolds a React/Vite + FastAPI portal (embed scaffold).
7. Real-Time BigQuery Storage Write API & CDC Streaming (stream — Gate 14): Plans (stream plan), scaffolds & customizes a Go managedwriter DefaultStream CDC service (_CHANGE_TYPE = UPSERT/DELETE, max_staleness = INTERVAL 0 SECOND) driven by live Dataplex Knowledge Catalog metadata, and deploys it to a locked-down GCE e2-micro VM (--no-address private IP + IAP tunnel + <dataset>-streamer.service) (stream deploy --execute-infra), with single-command teardown (stream delete).

What Gets Automated: Production Deliverables Inventory

Subagent Invocation Matrix (looker-demo-orchestrator)

demo-create follows a CLI-First Architecture with Explicit & Conditional Subagents. Deterministic operations run directly via compiled CLI subcommands in the parent session, while specialized creative, modeling, healing, and multi-file Go engineering tasks delegate to isolated subagents under skills/looker-demo-orchestrator/subagents/:

Prerequisites

Before running demo-create, ensure the following tools, cloud roles, and Looker settings are in place. For the full IAM Principal Matrix, copy-paste gcloud setup script, granular Looker role permissions, OAuth client registration, and SSH port-forwarding guide, see docs/PREREQUISITES.md.

👉 Full Setup, IAM Matrix & gcloud Bootstrap Script → docs/PREREQUISITES.md

Getting Started: Two Ways to Build

🚀 Mode 1: AI Agent Pair-Programmer (Primary Hero Flow)

Run interactively with your AI coding assistant using the looker-demo-orchestrator skill.

Step 1: Install Persistent CLI Tools

uv tool install looker-demo-cli
# Or from a local checkout during development:
uv tool install --editable /path/to/looker-demo-cli --force

Installs demo-create, looker-demo-cli, and lkr (lkr-dev-cli) globally across terminal sessions.

Step 2: Run Pre-Flight Audit & Auto-Fix

demo-create pre-check --fix

Verifies GCP/Looker credentials, prunes deprecated MCP servers (data-designer, bigquery, knowledge-catalog), and syncs CLI-first skills.

Step 3: Launch the AI Demo Creation Flow

Instruct your AI assistant in chat:

"Create an end-to-end Looker demo for Mobile Gaming Telemetry (or IoT Fleet Analytics, SaaS ARR, Retail, Fintech)."

💻 Mode 2: Standalone CLI (15-Gate Headless Engine)

Execute demo-create directly from your terminal or CI/CD pipeline, one gate at a time. State is persisted to .demo-state.json between calls:

# Gates 0–1: Pre-check & Target Confirmation
demo-create pre-check --fix --gcp-project my-gcp-project
demo-create confirm-targets --mode full --gcp-account me@example.com --gcp-project my-gcp-project \
  --looker-account prod --connection my_bq_connection --dataset mobile_gaming_analytics

# Gates 2–4: Propose/Approve Schema & Generate/Upload Data (or `demo-create data adopt --dataset ...`)
demo-create data propose-schema --schema-file schema.json --preview
demo-create data approve-schema --row-count 25000 --dataset mobile_gaming_analytics
demo-create data generate --schema-file schema.json --row-count 25000 --gcp-project my-gcp-project \
  --dataset mobile_gaming_analytics --upload --json-scorecard

# Gate 5: Seed or Inspect Dataplex Knowledge Catalog (Glossaries, Terms, Aspects, EntryLinks)
demo-create catalog seed --dataset mobile_gaming_analytics --gcp-project my-gcp-project --mode execute

# Gates 6–10: LookML Model, 3-Pass Polish Certification, Optimizer, QA Deploy & Visual Critique
demo-create lookml model --looker-project mobile_gaming --dataset mobile_gaming_analytics \
  --connection my_bq_connection --gcp-project my-gcp-project --profile rich
demo-create lookml certify-polish --lookml-dir lookml
demo-create lookml optimize --lookml-dir lookml
demo-create lookml deploy --looker-project mobile_gaming --lookml-dir lookml
demo-create lookml approve-critique --looker-project mobile_gaming

# Gates 11–13: Conversational Analytics Agent, Gemini Enterprise Publish & Embed Portal
demo-create agent create --model mobile_gaming --explore fct_game_sessions --dashboards-dir lookml/dashboards
demo-create agent publish --agent-id <agent-id>
demo-create embed scaffold --looker-project mobile_gaming --skip

# Gate 14: Real-Time BigQuery Storage Write API & CDC Streamer on Locked-Down GCE (--no-address + IAP)
demo-create stream plan --gcp-project my-gcp-project --dataset mobile_gaming_analytics \
  --target-table fct_game_sessions --frequency 1m --crud-mode upsert_cdc --backfill-mode skip
demo-create stream deploy --gcp-project my-gcp-project --dataset mobile_gaming_analytics --scaffold-only
# (After 4-agent customization of ./streamer)
demo-create stream deploy --gcp-project my-gcp-project --dataset mobile_gaming_analytics --execute-infra

The 15-Stage Gate Reference (demo-create status)

Use demo-create status (or demo-create status --json) at any point to inspect completed gates and get the literal next command to run:

Commands Quick Reference

Documentation