🚚 dataform-view-migrator

Export BigQuery VIEW definitions into Dataform with ease. 🚀

dataform-view-migrator is a handy CLI tool that discovers your BigQuery views and turns them into Dataform-ready .sqlx or .sql files. It handles complex folder structures, provides safe backup policies, and automatically generates Dataform configuration headers.

⚡ Super Quickstart

1. Install

   pip install dataform-view-migrator
   

2. Authenticate

   gcloud auth application-default login
   

3. Migrate!

   dataform-view-migrator migrate-views --source-project my-project --dest ./my-dataform-repo
   

💡 Example Input & Output

BigQuery View (my_project.my_dataset.daily_sales):

SELECT date, SUM(amount) as total_sales
FROM `my_project.my_dataset.raw_transactions`
GROUP BY 1

Generated Dataform File (my-dataform-repo/my_dataset/daily_sales.sqlx):

config {
  type: "view",
  schema: "my_dataset",
  name: "daily_sales",
  description: "CREATED BY DATAFORM.",
  tags: ["my-tag","another-tag"],
}

SELECT date, SUM(amount) as total_sales
FROM `my_project.my_dataset.raw_transactions`
GROUP BY 1

✨ Features

• 🔍 Auto-discovery: Find all views across multiple datasets automatically.
• 📂 Flexible Layout: Map BigQuery datasets to custom subfolders in your Dataform project.
• 🛡️ Safe Writes: Choose to skip, backup, or force overwrite existing files.
• 📝 Dataform Headers: Automatically adds config { type: "view", ... } to your .sqlx files.
• 🧪 Dry Run: Preview exactly what will happen before making any changes.
• 🚀 Fast: Uses INFORMATION_SCHEMA for high-performance discovery in large projects.

🛠 Prerequisites

• Python 3.10+
• GCP Authentication via Application Default Credentials (ADC).

📖 Commands

• ping-bq: 📡 Verify your authentication and BigQuery access.

  dataform-view-migrator ping-bq --project my-project
  

• migrate-views: 🚜 The main event. Discovers and exports views.

  dataform-view-migrator migrate-views [options]
  

Run --help on any command for a full list of available options.

⚙️ Configuration

While you can use CLI flags, using a TOML file is often easier for recurring tasks. Copy dataform_view_migrator.example.toml to dataform_view_migrator.toml and customize it.

Available Keys

📂 Output Layout

By default, files are saved as dest/<dataset>/<view_name>.sqlx. You can remap dataset names to specific folders using the dataset_folders setting in your config:

[dataset_folders]
raw_data = "sources/raw"
analytics = "definitions/reporting"

🤝 Contributing & Development

We love contributions! Please check out DEVELOPMENT.md for setup instructions, linting rules, and testing guidelines.

🔗 Links

• 📦 PyPI: https://pypi.org/project/dataform-view-migrator/
• 💻 GitHub: https://github.com/elvainch/dataform-view-migrator
• 📝 License: MIT

Developed by Alan Vainsencher.