dataform-view-migrator 0.1.6

Migrate BigQuery views into Dataform view definitions.

🚚 dataform-view-migrator

Export BigQuery VIEW definitions into Dataform with ease. 🚀

dataform-view-migrator is a handy CLI tool that discovers your BigQuery views and turns them into Dataform-ready .sqlx or .sql files. It handles complex folder structures, provides safe backup policies, and automatically generates Dataform configuration headers.

---

⚡ Super Quickstart

1. Install

   pip install dataform-view-migrator
   

2. Authenticate

   gcloud auth application-default login
   

3. Migrate!

   dataform-view-migrator migrate-views --source-project my-project --dest ./my-dataform-repo
   

💡 Example Input & Output

BigQuery View (my_project.my_dataset.daily_sales):

SELECT date, SUM(amount) as total_sales
FROM `my_project.my_dataset.raw_transactions`
GROUP BY 1

Generated Dataform File (my-dataform-repo/my_dataset/daily_sales.sqlx):

config {
  type: "view",
  schema: "my_dataset",
  name: "daily_sales",
  description: "CREATED BY DATAFORM.",
  tags: ["my-tag","another-tag"],
}

SELECT date, SUM(amount) as total_sales
FROM `my_project.my_dataset.raw_transactions`
GROUP BY 1

---

✨ Features

• 🔍 Auto-discovery: Find all views across multiple datasets automatically.
• 📂 Flexible Layout: Map BigQuery datasets to custom subfolders in your Dataform project.
• 🛡️ Safe Writes: Choose to skip, backup, or force overwrite existing files.
• 📝 Dataform Headers: Automatically adds config { type: "view", ... } to your .sqlx files.
• 🧪 Dry Run: Preview exactly what will happen before making any changes.
• 🚀 Fast: Uses INFORMATION_SCHEMA for high-performance discovery in large projects.

---

🛠 Prerequisites

• Python 3.10+
• GCP Authentication via Application Default Credentials (ADC).

---

📖 Commands

• ping-bq: 📡 Verify your authentication and BigQuery access.

  dataform-view-migrator ping-bq --project my-project
  

• migrate-views: 🚜 The main event. Discovers and exports views.

  dataform-view-migrator migrate-views [options]
  

Run --help on any command for a full list of available options.

---

⚙️ Configuration

While you can use CLI flags, using a TOML file is often easier for recurring tasks. Copy dataform_view_migrator.example.toml to dataform_view_migrator.toml and customize it.

Available Keys

Key	Type	Default	Description
source_project	string		Required. The GCP project ID containing your views.
dest	string		Required. Local path to your Dataform repository.
datasets_include	list	[]	List of datasets to process. If empty, processes all.
datasets_exclude	list	[]	List of datasets to ignore.
location	string		BigQuery region (e.g., US, EU). Required for high-performance discovery.
ext	string	sqlx	File extension: sqlx or sql.
overwrite	string	skip	Policy: skip, backup (renames existing), or force.
add_dataform_header	boolean	true	Prepend config {} block to files.
dry_run	boolean	false	If true, only show what would happen.
dataform_header.description	string		Optional description for the Dataform config block.
dataform_header.tags	list	[]	Optional list of tags for the Dataform config block.
dataset_folders	dict		Map dataset IDs to custom subfolders (e.g., ds_id = "path/to/dir").

---

📂 Output Layout

By default, files are saved as dest/<dataset>/<view_name>.sqlx. You can remap dataset names to specific folders using the dataset_folders setting in your config:

[dataset_folders]
raw_data = "sources/raw"
analytics = "definitions/reporting"

---

🤝 Contributing & Development

We love contributions! Please check out DEVELOPMENT.md for setup instructions, linting rules, and testing guidelines.

---

🔗 Links

• 📦 PyPI: https://pypi.org/project/dataform-view-migrator/
• 💻 GitHub: https://github.com/elvainch/dataform-view-migrator
• 📝 License: MIT

---

Developed by Alan Vainsencher.