looker_loader 0.1.42

a tool for flexibly loading schemas from bigquery into looker

looker-loader

Write rules for how you want fields and measures to be generated from tables into looker.

Are you tired of adding timeframes to dimension_groups manually, of defining simple measures, of adding descriptions to measures? looker_loader removes a lot of the tedious work involved with loading new tables into Looker.

Define field_label, group_label, value_format_name etc in a common source of truth

If you find discrepancies in value formats and labelling across tables to be annoying and tedious, wouldnt it be nice to be able to input a label one place, and have it applied for every table? Looker_loader enables this.

Setup scheduled refreshes of lookml automatically

Fed up with generating new autogenerated tables every time a new field is added to a table? Use looker_loader to refresh tables on a schedule!

Quick Start

1. Install the package:

   uv add looker_loader
   

2. Create configuration files:

   • loader_config.yml - Define your BigQuery connections and settings
   • loader_recipe.yml - Define rules for field transformations

loader_config.yml

This file defines your BigQuery connections and loader settings:

config:
  loader:
    lexicanum: true
    output_path: ./views
  bigquery:
    - project_id: "your-project-id"
      dataset_id: "your_dataset"
      config:
        regex_include: "^fct_|^dim_|^obt_"
        regex_exclude: "_temp$|_backup$"

loader_recipe.yml

This file defines rules for how different field types should be transformed:

recipes:
  - name: primary_key
    filters:
      field_order: [0]
    dimension:
      primary_key: true

  - name: hide_records
    filters:
      db_types: [RECORD]
    dimension:
      hidden: true

  - name: group_ids
    dimension:
      group_label: Identifiers
      value_format_name: id
      measures:
        - type: count_distinct
    filters:
      regex_include: _id$|^pk.*_|_code$

  - name: numbers
    filters:
      types: [number]
      regex_exclude: _id$|^pk_|_code$
    dimension:
      group_label: Numbers
      value_format_name: decimal_1
      measures:
        - type: sum
          label: "Sum {{ parent_label }}"

  - name: time_date_values
    filters:
      regex_include: _date
      types: [date]
    dimension:
      timeframes:
        - raw
        - time
        - date
        - week
        - month
        - quarter
        - year

3. Run the loader:

   uv run looker_loader
   

Check the Output

The loader will create LookML files in your specified output directory (default: ./views). You should see files like:

views/
├── fct_sales.view.lkml
├── dim_customers.view.lkml
└── obt_daily_summary.view.lkml

Common Issues

No tables found

• Check your project_id and dataset_id are correct
• Verify your BigQuery permissions
• Check if your regex_include pattern matches any tables

Authentication errors

• Ensure your Google Cloud credentials are properly set up

Next Steps

Now that you have the basics working:

• Explore Configuration Options - Learn about all available configuration settings
• Create Custom Recipes - Build specialized field transformation rules
• Use Lexicanum - Enhance field definitions by using a common source of truth