SemBaker

Compile-then-execute semantic operators — an external optimizer for Palimpzest, LOTUS, Nirvana, and DocETL.

Semantic-operator systems let you write data operations in natural language — sem_filter("clearly positive review"), sem_map("extract the patient's age"), sem_join("opposite sentiment"). The usual way to execute these is interpretation: for every row (or candidate pair, or document) the system issues an LLM call to evaluate the predicate. Expressive, but it puts an expensive LLM call inside the data loop — high latency, high cost, poor scaling.

SemBaker does the alternative: compilation. It calls the LLM once to translate a semantic operator into a deterministic Python function, then runs that function locally over the whole dataset — no per-item LLM call at execution.

interpretation:   N items  ->  N LLM calls           (cost/latency grow with N)
compilation:      N items  ->  1 compile call + N local function calls (~0)

The compiled function is a reusable, cacheable artifact; the LLM cost is a fixed one-shot "compile" charge that amortizes across the input.

SemBaker implements the approach introduced in our vision paper From Interpretation to Compilation: Compilation-Based Execution of Semantic Operators (Dong & Wang, 2026) — see Citation.

Install

pip install sembaker                 # core only (compile / cache / decide)
pip install sembaker[pz]             # + Palimpzest adapter deps
pip install sembaker[lotus]          # + LOTUS (lotus-ai)
pip install sembaker[nirvana]        # + Nirvana (nirvana-ai)
pip install sembaker[docetl]         # + DocETL
pip install sembaker[all]            # everything

Set OPENAI_API_KEY in your environment (or a .env you load yourself). The default compile model is gpt-5-mini; override with CX_COMPILE_MODEL.

Usage: keep your engine's native API

You keep writing each engine's native pipeline; SemBaker is one extra call that walks the plan, decides per operator whether to compile-then-execute or stay native, and swaps in the compiled path — without modifying the engine's source.

Note: the cost model only picks the compiled path once the input is large enough to amortize the one-shot compile (roughly a dozen rows; see "Compile-vs-native decision" below). On toy-sized data everything routes native — the runnable scripts in examples/ use 20 rows so the compiled path actually engages.

Palimpzest

import palimpzest as pz
import sembaker.backends.pz as cxpz

ds = (pz.MemoryDataset(id="reviews", vals=df)
        .sem_filter("the review is clearly positive"))
report = cxpz.optimize(ds)        # walk plan, decide per op, gate rules
out = ds.run(config)              # PZ executes; compiled ops run locally

LOTUS

import lotus
from lotus.ast import LazyFrame
import sembaker.backends.lotus as cxlotus

lf = LazyFrame(df).sem_filter("The {reviewText} is clearly positive.")
cxlotus.optimize(lf)              # SemFilterNode -> CXFilterNode, in place
out = lf.execute(df)

Nirvana

import nirvana as nv
import sembaker.backends.nirvana as cxnv

ndf = nv.DataFrame(df)
ndf.semantic_filter("the review is clearly positive", input_columns=["reviewText"])
cxnv.optimize(ndf)                # inject compiled fn into the native UDF slot
out, cost, secs = ndf.execute()   # automatic LLM fallback if the fn raises

DocETL

import docetl
import sembaker.backends.docetl as cxdoc
cxdoc.apply()                     # wrap Frame.map / Frame.filter

f = docetl.from_list(docs)
f = f.filter(prompt="Keep clearly positive reviews. {{ input.reviewText }}")
out = f.collect()                 # rewritten to native code_filter, 0 LLM/doc

Your own engine: the compile core is public API; an adapter is typically ~100 lines. See docs/writing_a_backend.md.

Operator support

Every pipeline still runs in full. sembaker's decision is strictly per-operator: an operator it can't (or shouldn't) compile is simply left on the engine's native path — it is reported in the rewrite report, never rewritten, never broken. What CAN be compiled today is filter / map / join:

Also left native by design: operators where the user already supplied their own UDF (never touched), and predicates the codifiability judge rejects (e.g. tone/sarcasm/nuance judgments that keyword-or-regex logic can't faithfully capture).

The compile pipeline & knobs

Every backend routes its compiles through one entry point (sembaker.optimizer.compile_op.compile_operator), so these environment switches apply uniformly across all backends:

• refine (sembaker.core.refine) — one LLM call that turns a fuzzy predicate into a concrete recipe (e.g. discovers a structured scoreSentiment column to compare instead of guessing sentiment from text).
• validate (sembaker.core.validate) — labels a few sample items once (LLM, in parallel) and scores each compile draw against them, keeping the best.
• cache (sembaker.core.cache) — persistent, keyed on (method, op, canonicalized predicate, columns, model). The key excludes the backend: an artifact compiled for one engine is reused by another.

IR-decoupled compilation (CX_COMPILE=ir1|ir2)

Free-form code generation is high-variance. The IR path decouples finding the logic from writing the code: one LLM call emits a small structured IR, then a deterministic transpiler (no LLM) renders it to Python — so temperature-1 variance is confined to a small structured space, and the code step is byte-reproducible.

• ir1 — a flat "feature + decision" DSL (sembaker.core.ir).
• ir2 — a nestable expression tree (AST) produced via grammar prompting; richer (nesting, comparisons, to_int, first_nonempty, ifelse) yet still one LLM call (sembaker.core.ir2). Design follows TRANX (Yin & Neubig, EMNLP 2018) with grammar prompting (Wang et al., 2023).

Compile-vs-native decision

sembaker.optimizer.decide() routes each operator: a codifiability judge (heuristic or LLM) asks whether the predicate is expressible as deterministic logic over the visible columns, and a cost model amortizes the one-shot compile cost against the per-item native cost at the operator's estimated cardinality. Force the compiled path with each adapter's force flag / the FORCE_CX=1 convention where supported.

Tested versions

Developed and tested against: palimpzest 1.5.x · lotus-ai 1.1.x · nirvana-ai 1.3.x · docetl 0.3.x · Python 3.10+.

Citation

If you use SemBaker in your research, please cite:

@misc{dong2026compilation,
  title         = {From Interpretation to Compilation: Compilation-Based
                   Execution of Semantic Operators},
  author        = {Dong, Wenkai and Wang, Yifan},
  year          = {2026},
  eprint        = {2607.13407},
  archivePrefix = {arXiv},
  primaryClass  = {cs.DB},
  url           = {https://arxiv.org/abs/2607.13407},
}

License

SemBaker is dual-licensed:

• AGPL-3.0 (LICENSE) — free for research, evaluation, and any use that complies with the AGPL's copyleft, including its network-service provision (running a modified version as a service requires releasing your corresponding source under the AGPL).
• Commercial license — for closed-source or proprietary use that cannot meet AGPL obligations. Contact dongw@hawaii.edu or yifanw@hawaii.edu.

Patent applications covering techniques in this software have been filed; see NOTICE.