DataFrame Transformation Language from YAML (dftly)

Dftly (pronounced "deftly") is a simple, expressive, human-readable DSL for encoding simple tabular transformations over dataframes, designed for expression in YAML files. With dftly, you can transform your data, deftly!

Installation

pip install dftly

You can also install it locally via uv via:

uv sync

from the root of the repository.

Usage

Dftly is designed to make it easy to specify simple dataframe transformations in a YAML file (or a mapping-like format). In particular, with dftly, you can specify a mapping of output column names to expressions over input columns, then easily execute that over an input table.

Suppose we have an input dataframe that looks like this:

>>> import polars as pl
>>> from datetime import date
>>> df = pl.DataFrame({
...     "col1": [1, 2],
...     "col2": [3, 4],
...     "foo": ["5", "6"],
...     "col3": ["2020-01-01", "2021-06-15"],
...     "bp": ["120/80", "NULL"],
... })
>>> df
shape: (2, 5)
┌──────┬──────┬─────┬────────────┬────────┐
│ col1 ┆ col2 ┆ foo ┆ col3       ┆ bp     │
│ ---  ┆ ---  ┆ --- ┆ ---        ┆ ---    │
│ i64  ┆ i64  ┆ str ┆ str        ┆ str    │
╞══════╪══════╪═════╪════════════╪════════╡
│ 1    ┆ 3    ┆ 5   ┆ 2020-01-01 ┆ 120/80 │
│ 2    ┆ 4    ┆ 6   ┆ 2021-06-15 ┆ NULL   │
└──────┴──────┴─────┴────────────┴────────┘

with dftly, we can write a yaml file like this:

>>> ops = r"""
... sum: "$col1 + $col2"
... diff: "$foo::int - $col1"
... compare: "$col1 > ($col2 - 3) * 3"
... str_interp: 'f"value: {$foo} {$col1}"'
... max: "max($col1, $col2)"
... conditional: '"big" if $col1 > 1 else "small"'
... sys_bp: "extract group 1 of /(\\d+)\\/(\\d+)/ from $bp if /(\\d+)\\/(\\d+)/ in $bp"
... dia_bp: "(extract group 2 of /(\\d+)\\/(\\d+)/ from $bp if /(\\d+)\\/(\\d+)/ in $bp) as float"
... """

Then use it to transform the dataframe like so:

>>> from dftly import Parser
>>> df.select(**Parser.to_polars(ops))
shape: (2, 8)
┌─────┬──────┬─────────┬────────────┬─────┬─────────────┬────────┬────────┐
│ sum ┆ diff ┆ compare ┆ str_interp ┆ max ┆ conditional ┆ sys_bp ┆ dia_bp │
│ --- ┆ ---  ┆ ---     ┆ ---        ┆ --- ┆ ---         ┆ ---    ┆ ---    │
│ i64 ┆ i64  ┆ bool    ┆ str        ┆ i64 ┆ str         ┆ str    ┆ f32    │
╞═════╪══════╪═════════╪════════════╪═════╪═════════════╪════════╪════════╡
│ 4   ┆ 4    ┆ true    ┆ value: 5 1 ┆ 3   ┆ small       ┆ 120    ┆ 80.0   │
│ 6   ┆ 4    ┆ false   ┆ value: 6 2 ┆ 4   ┆ big         ┆ null   ┆ null   │
└─────┴──────┴─────────┴────────────┴─────┴─────────────┴────────┴────────┘

Other supported operations include string to time parsing, conversion to duration, datetime arithmetic, and more:

>>> ops = r"""
... as_date: '$col3::"%Y-%m-%d"'
... days_later: '($col3 as "%Y-%m-%d") + $col1::days'
... at_time: '$col3::"%Y-%m-%d" @ 11:30 a.m.'
... """
>>> df.select(**Parser.to_polars(ops))
shape: (2, 3)
┌────────────┬────────────┬─────────────────────┐
│ as_date    ┆ days_later ┆ at_time             │
│ ---        ┆ ---        ┆ ---                 │
│ date       ┆ date       ┆ datetime[μs]        │
╞════════════╪════════════╪═════════════════════╡
│ 2020-01-01 ┆ 2020-01-02 ┆ 2020-01-01 11:30:00 │
│ 2021-06-15 ┆ 2021-06-17 ┆ 2021-06-15 11:30:00 │
└────────────┴────────────┴─────────────────────┘

Dftly also supports exponentiation with the familiar ** operator, which binds tighter than *// and is right-associative:

>>> ops = r"""
... squared: "$col1 ** 2"
... cubed: "$col2 ** 3"
... hypotenuse: "($col1**2 + $col2**2) ** 0.5"
... """
>>> df.select(**Parser.to_polars(ops))
shape: (2, 3)
┌─────────┬───────┬────────────┐
│ squared ┆ cubed ┆ hypotenuse │
│ ---     ┆ ---   ┆ ---        │
│ i64     ┆ i64   ┆ f64        │
╞═════════╪═══════╪════════════╡
│ 1       ┆ 27    ┆ 3.162278   │
│ 4       ┆ 64    ┆ 4.472136   │
└─────────┴───────┴────────────┘

Datetime values expose their components via cast syntax — every <field>_of_<period> name (e.g. hour_of_day, day_of_week, month_of_year, year_of_date) works in both :: and as forms and compiles to the corresponding polars .dt.* method:

>>> ops = r"""
... event: '$col3::"%Y-%m-%d"'
... year: '$col3::"%Y-%m-%d" as year_of_date'
... month: '$col3::"%Y-%m-%d" as month_of_year'
... day_of_week: '$col3::"%Y-%m-%d" as day_of_week'
... quarter: '$col3::"%Y-%m-%d" as quarter_of_year'
... """
>>> df.select(**Parser.to_polars(ops))
shape: (2, 5)
┌────────────┬──────┬───────┬─────────────┬─────────┐
│ event      ┆ year ┆ month ┆ day_of_week ┆ quarter │
│ ---        ┆ ---  ┆ ---   ┆ ---         ┆ ---     │
│ date       ┆ i32  ┆ i8    ┆ i8          ┆ i8      │
╞════════════╪══════╪═══════╪═════════════╪═════════╡
│ 2020-01-01 ┆ 2020 ┆ 1     ┆ 3           ┆ 1       │
│ 2021-06-15 ┆ 2021 ┆ 6     ┆ 2           ┆ 2       │
└────────────┴──────┴───────┴─────────────┴─────────┘

Note that ::year is already the integer→date constructor (2024::year → date(2024, 1, 1)), so the year-extraction accessor is spelled ::year_of_date to keep the direction unambiguous. Every other component name (month_of_year, day_of_month, etc.) follows the same pattern.

Duration values project to numeric totals via total_<unit> cast names (total_seconds, total_minutes, total_hours, total_days, total_milliseconds, total_microseconds, total_nanoseconds). This is the dual to the existing ::days / ::seconds construction — numeric to Duration goes through plural unit names (nanoseconds, microseconds, milliseconds, seconds, minutes, hours, days, weeks, months, years), Duration to numeric goes through total_-prefixed ones, and every unit is available in both directions. Combined with datetime subtraction, this covers most time-derived feature engineering in one line:

>>> ops = r"""
... days_since_start: '(($col3 as "%Y-%m-%d") - 2020-01-01) as total_days'
... hours_since_start: '(($col3 as "%Y-%m-%d") - 2020-01-01) as total_hours'
... age_years: '(2030-01-01 - ($col3 as "%Y-%m-%d"))::total_microseconds / 31557600000000'
... """
>>> df.select(**Parser.to_polars(ops))
shape: (2, 3)
┌──────────────────┬───────────────────┬───────────┐
│ days_since_start ┆ hours_since_start ┆ age_years │
│ ---              ┆ ---               ┆ ---       │
│ i64              ┆ i64               ┆ f64       │
╞══════════════════╪═══════════════════╪═══════════╡
│ 0                ┆ 0                 ┆ 10.001369 │
│ 531              ┆ 12744             ┆ 8.54757   │
└──────────────────┴───────────────────┴───────────┘

Sub-second units matter most when a source table records offsets in them. An offset column documented as milliseconds can say so directly, rather than being divided into a coarser unit:

>>> from datetime import datetime
>>> offsets = pl.DataFrame({
...     "origin": [datetime(2020, 1, 1), datetime(2021, 6, 15)],
...     "measuredat": [1500, 90000],  # milliseconds since admission
... })
>>> offsets.select(**Parser.to_polars({"measured_time": "$origin + $measuredat::milliseconds"}))
shape: (2, 1)
┌─────────────────────────┐
│ measured_time           │
│ ---                     │
│ datetime[μs]            │
╞═════════════════════════╡
│ 2020-01-01 00:00:01.500 │
│ 2021-06-15 00:01:30     │
└─────────────────────────┘

Note the use of ::total_microseconds rather than as total_microseconds in the age_years line — the two cast forms are semantically equivalent but differ in grammar precedence. :: binds tighter than *//, so the division applies to the result of the accessor; as binds loosely and would capture the remainder of the expression, causing a parse error in this case. Use :: inside larger expressions, as when the cast is the outermost operation.

Every datetime/duration accessor also exists as a function-call form — dt_hour_of_day($event), dt_total_seconds($delta), etc. — for use in programmatic construction or when the cast form doesn't compose cleanly. The two are always equivalent.

Chaining casts

Casts chain, and read left to right — each one applies to everything to its left, so $col::int::year::datetime is (($col::int)::year)::datetime. There is no alternative grouping parentheses could disambiguate here, so they are not required:

>>> years = pl.DataFrame({"admissionyeargroup": ["2003-2004", "2010-2011"]})
>>> ops = r"""
... admit_year: '(extract /2003|2010/ from $admissionyeargroup)::int::year::datetime'
... """
>>> years.select(**Parser.to_polars(ops))
shape: (2, 1)
┌─────────────────────┐
│ admit_year          │
│ ---                 │
│ datetime[μs]        │
╞═════════════════════╡
│ 2003-01-01 00:00:00 │
│ 2010-01-01 00:00:00 │
└─────────────────────┘

Chaining is sugar only: each link is still its own cast node, so the chained form and the parenthesized form produce the identical tree. as chains the same way ($col as int as year), and the two spellings can be mixed — with their usual precedence difference, so a :: link binds tighter than surrounding arithmetic while an as link does not.

Non-strict conversion with ::?

Real-world source columns are often "mostly" the type they claim to be — a VARCHAR dose column holding "25" alongside "1000 MG", or a date column with a few unparsable entries. Prefixing the cast target with ? makes the conversion non-strict: values that cannot be converted become null instead of raising. This works for both type casts and strptime formats, and means the same thing in both (it sets Polars' strict=False):

>>> messy = pl.DataFrame({
...     "dose": ["25", "1000 MG", "1.5E-3", ""],
...     "dod": ["2020-06-20", "not a date", "2021-01-05", "2019-12-31"],
... })
>>> ops = r"""
... numeric_value: '$dose::?float64'
... death_date: '$dod::?"%Y-%m-%d"'
... """
>>> messy.select(**Parser.to_polars(ops))
shape: (4, 2)
┌───────────────┬────────────┐
│ numeric_value ┆ death_date │
│ ---           ┆ ---        │
│ f64           ┆ date       │
╞═══════════════╪════════════╡
│ 25.0          ┆ 2020-06-20 │
│ null          ┆ null       │
│ 0.0015        ┆ 2021-01-05 │
│ null          ┆ 2019-12-31 │
└───────────────┴────────────┘

Without the ?, both forms raise on the first unconvertible value, which is the right default — ? is how you opt in to dropping data. Because it delegates to Polars' own parser, ::?float64 accepts everything Polars accepts (1.5E-3, +5, inf, nan) rather than whatever subset a hand-written regex guard happens to cover.

The ? prefix applies only to real dtype casts. The duration/date unit casts (::minutes, ::year) build values with pl.duration()/pl.date(), and the datetime accessors (::hour_of_day) extract a component — none of these have a strictness to relax, so ::?minutes and ::?hour_of_day are errors rather than silent no-ops.

Regex extraction and capture groups

extract /re/ from $col returns the whole match. To pull out a capture group, name it with extract group N of /re/ from $col — the same node, with group_index set:

>>> bands = pl.DataFrame({"agegroup": ["40-49", "80+"]})
>>> regex_ops = {
...     "whole_match": r"extract /^[0-9]{2}/ from $agegroup",
...     "age_lo": r"extract group 1 of /^([0-9]{2})/ from $agegroup",
...     "age_hi": r"(extract group 1 of /([0-9]{2}).?$/ from $agegroup)::int",
...     "span": r'f"{extract group 1 of /^([0-9]{2})/ from $agegroup} to {extract group 1 of /([0-9]{2}).?$/ from $agegroup}"',
... }
>>> bands.select(**Parser.to_polars(regex_ops))
shape: (2, 4)
┌─────────────┬────────┬────────┬──────────┐
│ whole_match ┆ age_lo ┆ age_hi ┆ span     │
│ ---         ┆ ---    ┆ ---    ┆ ---      │
│ str         ┆ str    ┆ i32    ┆ str      │
╞═════════════╪════════╪════════╪══════════╡
│ 40          ┆ 40     ┆ 49     ┆ 40 to 49 │
│ 80          ┆ 80     ┆ 80     ┆ 80 to 80 │
└─────────────┴────────┴────────┴──────────┘

Because it is an ordinary expression, the group form chains with :: and nests inside f"{...}" just like everything else — no need to break out into an intermediate column.

A pattern that writes capture groups but never names one is almost always a request for group 1 that would silently return the whole match instead, so that combination warns and points at the syntax above. To keep the whole match deliberately, either make the group non-capturing ((?:...)) or ask for it explicitly with extract group 0 of /re/ from $col.

What goes inside f"{...}"

Every {...} field holds a full dftly expression, taken verbatim — casts, regex forms with brace quantifiers, subscripts, ??, conditionals:

>>> interp_df = pl.DataFrame({"dose": [3.7], "icd": ["12345"], "unit": [None]})
>>> interp_ops = {
...     "rounded": 'f"dose={$dose::int}"',
...     "dotted": r'f"{extract group 1 of /^([0-9]{3})/ from $icd}.{$icd[3:]}"',
...     "guarded": '''f"{$icd}//{$unit ?? 'UNK'}"''',
...     "braced": 'f"{{{$icd}}}"',
... }
>>> interp_df.select(**Parser.to_polars(interp_ops))
shape: (1, 4)
┌─────────┬────────┬────────────┬─────────┐
│ rounded ┆ dotted ┆ guarded    ┆ braced  │
│ ---     ┆ ---    ┆ ---        ┆ ---     │
│ str     ┆ str    ┆ str        ┆ str     │
╞═════════╪════════╪════════════╪═════════╡
│ dose=3  ┆ 123.45 ┆ 12345//UNK ┆ {12345} │
└─────────┴────────┴────────────┴─────────┘

A field ends at the brace that closes it, never at a brace belonging to something inside it: dftly reads the field with its own parser, so a {2} quantifier, or a brace inside a string literal, regex literal, or backtick-quoted column name, is simply part of the expression — f"{$a ?? '}'}" means what it looks like. Literal braces in the surrounding text are doubled, as in Python: {{ and }}.

Python's format-spec and conversion syntax ({x:>10}, {x!r}) is not supported — a : or ! inside a field is ordinary dftly syntax, which is what makes f"{$dose::int}" a cast rather than a field named $dose with a :int format spec. Unmatched or empty braces are parse errors rather than silently reinterpreted text.

Position-based string operations

len_chars($col) returns the Unicode character count of a string column, and substring($col, start, stop) returns a slice with Python-style [start, stop) semantics (inclusive start, exclusive stop; omit stop for "to end of string"). Negative indices count from the end, and the $col[start:stop] postfix shorthand parses to the same AST as the function form:

>>> codes_df = pl.DataFrame({"code": ["12345", "1", "A420"]})
>>> substring_ops = {
...     "length": "len_chars($code)",
...     "first_three": "$code[0:3]",
...     "trailing": "$code[3:]",
...     "last_two": "$code[-2:]",
...     "dotted": 'f"{$code[0:3]}.{$code[3:]}" if len_chars($code) > 3 else $code',
... }
>>> codes_df.select(**Parser.to_polars(substring_ops))
shape: (3, 5)
┌────────┬─────────────┬──────────┬──────────┬────────┐
│ length ┆ first_three ┆ trailing ┆ last_two ┆ dotted │
│ ---    ┆ ---         ┆ ---      ┆ ---      ┆ ---    │
│ u32    ┆ str         ┆ str      ┆ str      ┆ str    │
╞════════╪═════════════╪══════════╪══════════╪════════╡
│ 5      ┆ 123         ┆ 45       ┆ 45       ┆ 123.45 │
│ 1      ┆ 1           ┆          ┆ 1        ┆ 1      │
│ 4      ┆ A42         ┆ 0        ┆ 20       ┆ A42.0  │
└────────┴─────────────┴──────────┴──────────┴────────┘

The dotted column above is the idiomatic translation of the Python guard-and-splice pattern used for ICD-code normalization: apply the dot only when the code is long enough, otherwise pass through unchanged. The length guard is what makes substring strictly more expressive than a pure-regex solution for this pattern.

Slice step ([i:j:k]) and single-index subscription ([i]) are intentionally unsupported — polars' str.slice has no step, and single-index is expressible as substring(expr, i, i+1). Both produce a clear parse error pointing at the supported forms.

Null-coalescing with ??

The ?? operator returns its first non-null operand, so it's the concise way to substitute a sentinel for nulls. It is pure sugar for the coalesce node — $a ?? UNK parses to exactly the same tree as coalesce($a, UNK) — and works anywhere an expression is allowed, including inside f"{...}" string interpolation (where each field is itself a full expression). This makes it easy to guard interpolated columns against nulls:

>>> null_df = pl.DataFrame({"a": ["x", None], "b": [None, "y"], "n": [None, 5]})
>>> coalesce_ops = {
...     "filled": "$a ?? 'UNK'",
...     "count": "$n ?? 0",
...     "first_present": "$a ?? $b ?? 'UNK'",
...     "joined": '''f"{$a ?? 'UNK'}//{$b ?? 'UNK'}"''',
... }
>>> null_df.select(**Parser.to_polars(coalesce_ops))
shape: (2, 4)
┌────────┬───────┬───────────────┬────────┐
│ filled ┆ count ┆ first_present ┆ joined │
│ ---    ┆ ---   ┆ ---           ┆ ---    │
│ str    ┆ i64   ┆ str           ┆ str    │
╞════════╪═══════╪═══════════════╪════════╡
│ x      ┆ 0     ┆ x             ┆ x//UNK │
│ UNK    ┆ 5     ┆ y             ┆ UNK//y │
└────────┴───────┴───────────────┴────────┘

?? is left-associative and binds looser than the boolean operators (matching C#/JS), so $x or $y ?? $z groups as ($x or $y) ?? $z. Chains like $a ?? $b ?? UNK nest as coalesce(coalesce($a, $b), UNK), which returns the same first-non-null result as the flat coalesce($a, $b, UNK).

You can also add literal columns:

>>> ops = r"""
... str: '"hello"'
... int: '42'
... float: '3.14'
... bool: 'true'
... time: '11:30 a.m.'
... date: '2024-01-01'
... datetime: '2024-01-01 11:30 a.m.'
... """
>>> df.select(**Parser.to_polars(ops))
shape: (1, 7)
┌───────┬─────┬───────┬──────┬──────────┬────────────┬─────────────────────┐
│ str   ┆ int ┆ float ┆ bool ┆ time     ┆ date       ┆ datetime            │
│ ---   ┆ --- ┆ ---   ┆ ---  ┆ ---      ┆ ---        ┆ ---                 │
│ str   ┆ i32 ┆ f64   ┆ bool ┆ time     ┆ date       ┆ datetime[μs]        │
╞═══════╪═════╪═══════╪══════╪══════════╪════════════╪═════════════════════╡
│ hello ┆ 42  ┆ 3.14  ┆ true ┆ 11:30:00 ┆ 2024-01-01 ┆ 2024-01-01 11:30:00 │
└───────┴─────┴───────┴──────┴──────────┴────────────┴─────────────────────┘

Column names that aren't identifiers

$name covers column names that look like identifiers. Source tables — especially clinical ones — routinely ship columns like Variable Name or Unit, which $Variable Name cannot express. Wrap those in backticks:

>>> wide = pl.DataFrame({
...     "Variable Name": ["HR", "SpO2"],
...     "Unit": ["bpm", "%"],
...     "Value 1": [80, 97],
... })
>>> quoted_ops = {
...     "code": 'f"OBS//{$`Variable Name`}//{$`Unit`}"',
...     "numeric_value": "$`Value 1`::float",
... }
>>> wide.select(**Parser.to_polars(quoted_ops))
shape: (2, 2)
┌──────────────┬───────────────┐
│ code         ┆ numeric_value │
│ ---          ┆ ---           │
│ str          ┆ f32           │
╞══════════════╪═══════════════╡
│ OBS//HR//bpm ┆ 80.0          │
│ OBS//SpO2//% ┆ 97.0          │
└──────────────┴───────────────┘

This is a quoted spelling of the same column node, not a different one, so $`a` and $a are interchangeable and the quoted form composes everywhere a column reference can appear — arithmetic, casts, regex sources, f"{...}" fields.

There is no escape for a literal backtick inside a quoted name, and an empty quoted name (a $ followed by two backticks) is a parse error rather than a reference to a column called "". A column whose name contains a backtick is still reachable through the dict form, {column: "..."}, which has no lexer to escape from.

Bare words as string literals

When dftly expressions are embedded in YAML config files, string literals normally require awkward double-quoting because YAML strips its own quotes before dftly sees the value. To avoid this, dftly treats bare words — identifiers without a $ prefix, quotes, or parentheses — as string literals when they appear as a standalone expression:

>>> ops = r"""
... code: MEDS_BIRTH
... col_ref: $col1 + $col2
... quoted_str: '"hello"'
... number: 42
... bool_val: true
... """
>>> df.select(**Parser.to_polars(ops))
shape: (2, 5)
┌────────────┬─────────┬────────────┬────────┬──────────┐
│ code       ┆ col_ref ┆ quoted_str ┆ number ┆ bool_val │
│ ---        ┆ ---     ┆ ---        ┆ ---    ┆ ---      │
│ str        ┆ i64     ┆ str        ┆ i32    ┆ bool     │
╞════════════╪═════════╪════════════╪════════╪══════════╡
│ MEDS_BIRTH ┆ 4       ┆ hello      ┆ 42     ┆ true     │
│ MEDS_BIRTH ┆ 6       ┆ hello      ┆ 42     ┆ true     │
└────────────┴─────────┴────────────┴────────┴──────────┘

Only bare words are affected — column references ($col1 + $col2), quoted strings ("hello"), numbers, booleans, and all other expression types work without dftly-level quoting. Note that number: 42 and bool_val: true are parsed by YAML itself as int/bool and passed directly to dftly as POD literals — they never go through the expression grammar. This is unambiguous because column references always require the $ prefix, so a bare word cannot be confused with a column, function call, or any other expression.

Warning: If a bare word appears as part of a larger expression (e.g., $col + TYPO), dftly will still interpret it as a string literal but will emit a warning, since this usually indicates a missing $ prefix rather than an intentional literal:

>>> import warnings
>>> with warnings.catch_warnings(record=True) as w:
...     warnings.simplefilter("always")
...     expr = Parser.expr_to_polars("$col1 + TYPO")
...     assert len(w) == 1
...     print(w[0].message)
Bare word 'TYPO' interpreted as string literal in a subexpression. Did you mean the column '$TYPO'? Use $TYPO for a column reference or "TYPO" for an explicit string literal.

Design Philosophy

Dftly expressions can be written in three equivalent forms. Understanding this hierarchy is key to using dftly effectively:

1. String form -- Concise, human-readable syntax designed for YAML configs. Parsed by a Lark grammar.

   sum: $col1 + $col2 * 3
   

2. Dict/YAML form -- The fully explicit base form. Every node type, argument, and keyword argument is spelled out. This is the canonical representation that all other forms reduce to.

   sum:
     add:
       - column: col1
       - multiply:
           - column: col2
           - literal: 3
   

3. Class form -- Python objects, isomorphic to the dict form. Used for programmatic construction.

   Add(Column("col1"), Multiply(Column("col2"), Literal(3)))
   

All three forms produce the same internal AST and the same Polars expression. The string form is syntactic sugar over the dict form; any expression you can write as a string can also be written as an equivalent dict/YAML structure. When in doubt about what a string expression means, look at its dict form -- that is the unambiguous specification.

Detailed Documentation

Internally, this simply parses the yaml file into a mapping, then treats the mapping as a map from desired output column name to input column expression, parsing each expression via the dftly grammar. In particular, the below is equivalent to the above:

>>> ops = {
...     "sum": "$col1 + $col2",
...     "diff": "$col2 - $col1",
...     "compare": "$col1 > ($col2 - 3) * 3",
...     "str_interp": 'f"value: {$foo} {$col1}"',
...     "max": "max($col1, $col2)",
...     "conditional": '"big" if $col1 > 1 else "small"',
...     "sys_bp": r"extract group 1 of /(\d+)\/(\d+)/ from $bp if /(\d+)\/(\d+)/ in $bp",
...     "dia_bp": r"extract group 2 of /(\d+)\/(\d+)/ from $bp if /(\d+)\/(\d+)/ in $bp",
... }
>>> from dftly import Parser
>>> parser = Parser()
>>> ops = {k: parser(v).polars_expr for k, v in ops.items()}
>>> df.select(**ops)
shape: (2, 8)
┌─────┬──────┬─────────┬────────────┬─────┬─────────────┬────────┬────────┐
│ sum ┆ diff ┆ compare ┆ str_interp ┆ max ┆ conditional ┆ sys_bp ┆ dia_bp │
│ --- ┆ ---  ┆ ---     ┆ ---        ┆ --- ┆ ---         ┆ ---    ┆ ---    │
│ i64 ┆ i64  ┆ bool    ┆ str        ┆ i64 ┆ str         ┆ str    ┆ str    │
╞═════╪══════╪═════════╪════════════╪═════╪═════════════╪════════╪════════╡
│ 4   ┆ 2    ┆ true    ┆ value: 5 1 ┆ 3   ┆ small       ┆ 120    ┆ 80     │
│ 6   ┆ 2    ┆ false   ┆ value: 6 2 ┆ 4   ┆ big         ┆ null   ┆ null   │
└─────┴──────┴─────────┴────────────┴─────┴─────────────┴────────┴────────┘

The way dftly works is that strings are parsed into dictionary forms representing the specified operations, and an AST over those nodes is built up once they are resolved into dictionary form. This means you can also specify the operations in a fully explicit manner using these dictionary views for a more expansive, but precise syntax:

>>> ops = r"""
... sum: # "$col1 + $col2"
...   add:
...     - column: col1
...     - column: col2
... diff: # "$col2 - $col1"
...   subtract:
...     - column: col2
...     - column: col1
... compare: # "$col1 > ($col2 - 3) * 3"
...   greater_than:
...     - column: col1
...     - multiply:
...         - subtract:
...             - column: col2
...             - literal: 3
...         - literal: 3
... str_interp: # 'f"value: {$foo} {$col1}"'
...   string_interpolate:
...     - literal: "value: {} {}"
...     - column: foo
...     - column: col1
... max: # "max($col1, $col2)"
...   max:
...     - column: col1
...     - column: col2
... conditional: # '"big" if $col1 > 1 else "small"'
...   conditional:
...     when:
...       greater_than:
...         - column: col1
...         - literal: 1
...     then:
...       literal: "big"
...     otherwise:
...       literal: "small"
... sys_bp: # "extract group 1 of /(\\d+)\\/(\\d+)/ from $bp if /(\\d+)\\/(\\d+)/ in $bp"
...   conditional:
...     when:
...       regex_match:
...         pattern:
...           literal: (\d+)\/(\d+)
...         source:
...           column: bp
...     then:
...       regex_extract:
...         group_index:
...           literal: 1
...         pattern:
...           literal: (\d+)\/(\d+)
...         source:
...           column: bp
... dia_bp: # "extract group 2 of /(\\d+)\\/(\\d+)/ from $bp if /(\\d+)\\/(\\d+)/ in $bp"
...   conditional:
...     when:
...       regex_match:
...         pattern:
...           literal: (\d+)\/(\d+)
...         source:
...           column: bp
...     then:
...       regex_extract:
...         group_index:
...           literal: 2
...         pattern:
...           literal: (\d+)\/(\d+)
...         source:
...           column: bp
... """
>>> df.select(**Parser.to_polars(ops))
shape: (2, 8)
┌─────┬──────┬─────────┬────────────┬─────┬─────────────┬────────┬────────┐
│ sum ┆ diff ┆ compare ┆ str_interp ┆ max ┆ conditional ┆ sys_bp ┆ dia_bp │
│ --- ┆ ---  ┆ ---     ┆ ---        ┆ --- ┆ ---         ┆ ---    ┆ ---    │
│ i64 ┆ i64  ┆ bool    ┆ str        ┆ i64 ┆ str         ┆ str    ┆ str    │
╞═════╪══════╪═════════╪════════════╪═════╪═════════════╪════════╪════════╡
│ 4   ┆ 2    ┆ true    ┆ value: 5 1 ┆ 3   ┆ small       ┆ 120    ┆ 80     │
│ 6   ┆ 2    ┆ false   ┆ value: 6 2 ┆ 4   ┆ big         ┆ null   ┆ null   │
└─────┴──────┴─────────┴────────────┴─────┴─────────────┴────────┴────────┘

Note that literals are parsed by the string parser into either (a) a literal of the appropriate type (int, float, bool) or into literal nodes which have the syntax literal: [value]. In some cases, what looks like a string in the string syntax is actually parsed directly to a literal; for example, the syntax $col3::"%Y-%m-%d" @ 11:30 a.m. features a string literal for the format, but a time literal for the time. In this way, using the string syntax is often more concise, as you would need to explicitly construct or cast a string to a time were you to use the dictionary syntax. Note that these circumstances can be identified by the lack of quotes around the time literal in the string syntax; string literals will always be quoted, things without quotes will be interpreted as non-string literals.