looker-cst

A lossless LookML parser written in Rust with Python bindings. It parses a LookML file into a concrete syntax tree (CST) that keeps every comment, blank line and indentation, so a file can be read, modified and written back out with its formatting intact:

from looker_cst import parse

doc = parse(source)
assert str(doc) == source  # byte for byte, for every file in the looker repo

The parser is built with nom. The Python package is an abi3 wheel built against Python 3.9, so one wheel works on Python 3.9 and 3.11.

Python usage

from looker_cst import parse, LookmlSyntaxError

doc = parse(path.read_text())

view = doc.find("view", "users")             # first `view: users { ... }`
dim = view.find("dimension", "id")

dim["type"]                                  # "number"
dim["type"] = "string"                       # keeps the value's kind: type: string
dim["label"] = "User ID"                     # adds a missing pair: label: "User ID"
dim["hidden"] = True                         # hidden: yes
dim["sql"] = "${TABLE}.user_id"              # sql: ${TABLE}.user_id ;;
dim.comments = ["# primary key"]             # rewrites the comment lines above the pair

email = view.add("dimension", name="email", index=2)   # new block, spaced like its siblings
email["type"] = "string"

measure = view.find("measure", "count")
measure["drill_fields"] = ["id", "email"]               # list layout and quoting are kept
measure["filters"] = {"status": "active"}               # filters: [status: "active"]

view.add_source("""
measure: total {
  type: sum
  sql: ${TABLE}.amount ;;
}
""")                                         # parsed and re-indented to fit

view.remove("dimension", name="legacy")      # removes the pair and its comments
dim.detach()                                 # removes a pair from wherever it is

path.write_text(str(doc))

API

parse(source) -> Document raises LookmlSyntaxError (a ValueError) with line, column and offset attributes.

Document and block Pairs are containers with the same methods:

Pair also has key, name (block name), kind, value, set_value(value, kind=None), comments, parent, detach() and to_string(include_leading=False). Pairs are live handles: editing a pair returned by find edits the document, and two handles to the same node compare equal.

Values

Assigning to value (or c[key]) keeps the current kind, except that a literal becomes a string when the new text needs quotes. A new value's kind is inferred: sql, html, sql_* and *_sql keys are expressions; identifier-like text such as number is a literal, except for keys that are conventionally quoted (label, description, group_label, ...); booleans become yes/no; lists, tuples and dicts become lists. Pass kind= to choose explicitly.

New pairs copy the spacing of their neighbours, so a dimension added to a view gets the blank line and indentation the other dimensions have, and a replaced list keeps its inline or one-per-line layout, comma style and trailing comma.

Building objects from Python

View, Explore, Join, Dimension, DimensionGroup and Measure build LookML from Python values. Each fixes only its LookML key; fields are arbitrary keyword arguments, so any parameter Looker supports can be written, and nested objects are passed positionally:

from looker_cst import Block, Case, Dimension, Explore, Join, Measure, Named, Quoted, View

view = View(
    "users",
    Dimension("id", primary_key=True, type="number", sql="${TABLE}.id"),
    Dimension(
        "status",
        case=Case({"${TABLE}.status = 1": "Active", "${TABLE}.status = 0": "Inactive"}, else_="Unknown"),
    ),
    Dimension(
        "email",
        type="string",
        sql="${TABLE}.email",
        tags=["pii"],
        link=[Block(label="Profile", url="https://admin.example.com/{{ value }}")],
    ),
    Measure("count", type="count", drill_fields=["id", "email"], filters={"status": "Active"}),
    sql_table_name="public.users",
)
path.write_text(str(view))                       # a new file

doc = parse(path.read_text())
Dimension("name", type="string").add_to(doc.find("view", "users"), index=3)   # into a parsed file
Explore("users", Join("orders", from_="orders_v2", sql_on="${users.id} = ${orders.user_id}")).add_to(doc)

• Values convert as in add: sql style keys become ;; expressions, booleans yes/no, lists [a, b] and dicts [key: "value"].
• Block(...) is an unnamed { } block such as link or derived_table; a list of them repeats the key. Case(...) writes a case's when blocks and else.
• Quoted("usd"), Literal("Count") and Expr("${id}") force a value's form.
• A trailing underscore is dropped from field names, for Python keywords: from_, else_.
• Named(key, name, ...) covers any other block, e.g. Named("parameter", "metric", type="unquoted").
• add_to spaces the new object like its siblings; where there are none to copy, named blocks are set apart by a blank line and unnamed ones are not.

Rust usage

let doc = looker_cst::parse(&source)?;
let view = doc.body.find("view", Some("users")).unwrap();
let id = view.read().body().unwrap().find("dimension", Some("id")).unwrap();
id.write().body_mut().unwrap().find("type", None).unwrap().write().set_text("string");
assert!(doc.to_string().contains("type: string"));

The tree is in src/cst.rs: every node owns the whitespace and comments before it (leading), and a block or list owns the trivia before its closing bracket (trailing), so printing is a plain concatenation. Editing helpers are in src/edit.rs.

Development

cargo test                       # Rust unit, fixture and looker repo tests
uv run pytest                    # builds the extension, then runs the Python tests
cargo run --release --example roundtrip -- ../looker   # round trip check of a directory
uv run scripts/view_cst.py ../looker/models/analytics.model.lkml   # browse a file's tree

The looker repo tests read every .lkml file in the looker repo. For each file they check the round trip, rewrite every value with itself, edit every value and read it back, and add then remove pairs in every block, checking the source is restored exactly.

The first run shallow clones main of mozilla/looker-hub into target/looker and later runs reuse it; delete the directory (or cargo clean) to pick up newer commits. To test another checkout, point the tests at it: LOOKER_REPO=../looker-hub cargo test.

Benchmarks

cargo bench                                  # criterion: fixture, looker repo, largest file
uv run --with lkml scripts/benchmark.py      # Python: looker_cst against lkml

Both use the looker repo checkout the tests clone (target/looker, or $LOOKER_REPO), and scripts/benchmark.py also takes a path. On an Apple Silicon laptop over the 3433 files (71 MB) of mozilla/looker-hub, through Python (median of 3 runs):

From Rust, cargo bench parses the same repo in about 240 ms (~280 MiB/s). The lkml comparison takes several minutes on the full hub; pass a smaller directory to scripts/benchmark.py for a quick check.

To test a specific Python version:

UV_PROJECT_ENVIRONMENT=.venv-3.9 uv run --python 3.9 pytest

To build a wheel:

uvx maturin build --release      # target/wheels/looker_cst-*-cp39-abi3-*.whl