Fabric-RLM

Run verifiable data tasks inside Microsoft Fabric.

fabric-rlm gives a model a Python workspace next to your Fabric data. It can read Lakehouse files, analyze Delta tables, query Power BI semantic models, calculate results, and write new artifacts. Output contracts and validators decide whether the work is accepted. Failed checks go back into the run for another attempt.

The runtime uses a CPython subprocess inside the notebook session. pandas, DuckDB, Polars, openpyxl, PyMuPDF, and other installed packages remain available. Large files stay on disk. The model sees previews, summaries, and computed results rather than every raw byte.

Quick start in Fabric

Install fabric-rlm in a Fabric notebook. The Python notebook experience is recommended for this example.

%pip install "fabric-rlm[analytics]"

After installation, restart the session. The example below uses the roughly 140 MB, 1.57-million-row IMF CSV created by the flagship notebook. You can replace it with another large CSV in Lakehouse Files.

from fabric_rlm import FabricLM, File, RLM

result = RLM.task(
    task="""
    Analyze the complete IMF price dataset. Inspect its schema and confirm the
    dimensions used. Select the monthly, all-items, year-over-year CPI series:
    INDEX_TYPE='CPI', COICOP_1999='_T',
    TYPE_OF_TRANSFORMATION='YOY_PCH_PA_PT', and FREQUENCY='M'. For the latest
    complete year, calculate average inflation by country and return the 10
    countries with the highest average. Report the matched row count, source
    columns, and filters applied.
    """,
    inputs={
        "prices": File("/lakehouse/default/Files/imf_cpi.csv"),
    },
    outputs={
        "year": int,
        "top_countries": list,
        "rows_analyzed": int,
        "source_columns": list,
        "filters_applied": list,
    },
    lm=FabricLM("gpt-5.1"),
    skills=["data_exploration"],
    max_turns=8,
).run()

print(result.top_countries)
print(result.filters_applied)

FabricLM uses the model endpoint available to the Fabric capacity. The notebook identity supplies authentication. There is no API key to place in the notebook and no separate Azure OpenAI resource to configure. You can see the list of supported models here. You can also use models from OpenAI, Anthropic, Foundry, and OpenRouter through LiteLLM.

Why this works beyond the context window

A CSV with 1.57 million rows cannot be placed in a model prompt. The File input passes its Lakehouse path into the Python worker instead. The model can write DuckDB, Polars, or pandas code to inspect the schema, filter rows, and calculate aggregates. Only bounded execution feedback, such as schema details, previews, aggregates, and errors, enters the next model call. The raw dataset does not enter the model prompt.

This pattern also works with wide Excel workbooks, Parquet files, JSONL streams, PDFs, and combinations of those sources. File size is constrained by the libraries and compute available in the notebook session rather than the model's context window.

What happens during a run

1. You provide a task, named inputs, and an output contract.
2. The model writes Python for the task.
3. The Python runs in a persistent subprocess with access to mounted Lakehouse files and installed packages.
4. Execution output returns to the model. It can inspect results and revise the code.
5. The run calls SUBMIT(...).
6. Type checks, skill verifiers, and your validators inspect the submission.
7. A rejected submission returns with specific repair feedback.
8. An accepted submission becomes an RLMResult with the payload and full trajectory.

flowchart LR
    A["Task + Fabric data"] --> B["Model writes Python"]
    B --> C["CPython subprocess"]
    C --> D["Inspect and revise"]
    D --> E["SUBMIT"]
    E --> F{"Contract passes?"}
    F -- "No" --> D
    F -- "Yes" --> G["Result + trajectory"]

Why Fabric developers use it

Work with Fabric data in place

Bind individual Lakehouse files with File(...), discover Delta tables and Files with LakehouseSource(...), publish generated files with FileDestination(...), or connect Power BI semantic models with SemanticModel(...). A single task can combine these handles. DAX runs in the tabular engine, Delta reads honor the transaction log, file processing runs in Python, and generated artifacts can be written back to Files/ without giving the isolated worker OneLake credentials.

from fabric_rlm import FabricLM, FileDestination, LakehouseSource, RLM

lakehouse = LakehouseSource(
    "abfss://<workspace-id>@onelake.dfs.fabric.microsoft.com/<lakehouse-id>"
)

with FileDestination(
    "abfss://<workspace-id>@onelake.dfs.fabric.microsoft.com/"
    "<lakehouse-id>/Files/reports"
) as destination:
    result = RLM.task(
        task=(
            "Create a formatted revenue workbook. Save it to a staged file, "
            "verify it by reopening it, then publish it through destination."
        ),
        inputs={"lakehouse": lakehouse, "destination": destination},
        outputs={"workbook_path": str, "summary": dict, "sources_used": list},
        skills=["excel_modify", "delta_lakehouse"],
        lm=FabricLM("gpt-5.1"),
    ).run()

Worker code uses destination.stage("revenue.xlsx") for the local openpyxl path and destination.publish(staged) after verification. The trusted parent performs the final copy and returns a manifest containing path, name, and size. Publishing refuses path traversal, files outside the private staging area, oversized files, and accidental overwrites. The context manager removes local staging files whether the run succeeds or fails. Pass overwrite=True to destination.publish(...) only when replacing an existing OneLake file is intentional.

Use the Python packages already in the notebook

The subprocess runs the same Python environment as the notebook. Generated code can import native packages and work with real file paths. This is the main difference from the Deno and Pyodide interpreter used by DSPy's standard RLM.

Check the work before accepting it

Output mappings enforce runtime types. output_validator can enforce business rules. output_validator_context can inspect files and other side effects. Markdown skills can include their own verifier. A failed check becomes feedback for the next attempt.

Keep domain rules beside the data

Skills are Markdown playbooks. They capture field definitions, procedures, tripwires, and executable checks. Store custom skills in Lakehouse Files and load them with SkillLoader. Bundled and custom skills can be used together.

Inspect each run

RLMResult includes the submitted payload, executed turns, errors, timings, token usage, validation repairs, and a deterministic report. Trajectories can be saved and replayed without calling the model again.

Where it fits

Use fabric-rlm for tasks that need one or more of these:

• Data that is too large to place in a model prompt
• Exact calculation across many rows or files
• Power BI semantic model queries mixed with Lakehouse files
• Workbook, report, or document generation
• Multi-step analysis with checkable outputs
• Reusable domain instructions and validation rules
• An execution trail for debugging and review

A direct model call is usually a better choice for short questions, rewriting, or judgment over text that already fits in context. The runtime adds Python execution and iterative checks, so each run takes longer than a single call.

Start with the API tour

Import examples/notebooks/rlm_api_tour.ipynb into a Fabric workspace. It covers task construction, typed outputs, files, custom Lakehouse skills, validators, result inspection, and worker controls.

The other notebooks cover PDF work, Spark log analysis, spreadsheet editing, semantic models, and multi-source tasks.

Measured results

The benchmarks are included for readers who want the evaluation setup, costs, and caveats. They are not required to use the library.

xychart-beta
    title "SpreadsheetBench V1 Verified-400 pass rate (percent)"
    x-axis ["fabric-rlm + MiniMax M3", "WPS AI", "GPT for Excel", "Data Analysis Agent", "Qingqiu Agent"]
    y-axis "pass rate" 0 --> 100
    bar [82.5, 91.25, 92.5, 96.5, 98.25]

Fabric data sources

Fabric notebooks already provide mounted Lakehouse storage, sempy, notebook identity, and the Python analytics stack. fabric-rlm exposes those resources to the run through typed handles. The task can combine semantic model queries with CSV, PDF, Excel, Parquet, and JSONL files without building a separate ingestion path.

Every source is bound as an input and the model decides which one answers which part of the brief. The semantic models are queried with DAX in the tabular engine, so aggregation happens where the data is and only the result comes back. The files are read in the subprocess. The workbook is written straight to Files/.

Installation

Python 3.10 or newer is required. In Fabric, select the Python 3.12 (jupyter_python) kernel. Install the package and restart the session:

%pip install fabric-rlm

If imports fail on the Synapse PySpark kernel, see docs/fabric-runtime-deps.md.

Optional extras install packages used by specific workloads:

Other Python environments

The runtime also works on a laptop, in CI, or in an Azure Function. Use OpenAILM or AnthropicLM with the matching environment variable:

from fabric_rlm import File, OpenAILM, RLM  # set OPENAI_API_KEY in your environment

rlm = RLM.task(
    task="Sum every integer from 1 to 1,000,000 that is divisible by 3 or 5.",
    outputs=["answer"],
    lm=OpenAILM("gpt-4o-mini"),
)
print(rlm.run().answer)

RLM.task(...) is the short constructor; RLM.from_task(...) is the explicit form. OpenAILM, AnthropicLM, and FabricLM are thin wrappers over dspy.LM, so any OpenAI, Anthropic, Azure, or local Ollama model works.

Use a mapping when an output needs a runtime type contract:

result = RLM.task(
    task="Return the highest-revenue region and its revenue.",
    inputs={"sales": File("sales.csv")},
    outputs={"result": dict},
    lm=OpenAILM("gpt-4o-mini"),
).run()

If SUBMIT(result=...) receives the wrong type, the submission is rejected and the model gets repair feedback. Name-only lists such as outputs=["answer"] remain supported and do not add type enforcement.

Inputs and worker API

Values and files

Bind values, including large files, as inputs. Files arrive inside the worker as File(...) handles with .path, .read_text(), .read_bytes(), and .exists(), so a Lakehouse path or a local path is just a file path.

Lakehouses

LakehouseSource builds a metadata catalog in the parent Fabric notebook, then passes that catalog and the direct OneLake paths to the isolated worker. The model can choose relevant Delta tables and Files without you naming each table:

from fabric_rlm import FabricLM, LakehouseSource, RLM

source = LakehouseSource(
    "abfss://workspace-id@onelake.dfs.fabric.microsoft.com/lakehouse-id",
    tables="Tables",
    files="Files/data",
)

result = RLM.task(
    task="Which customer segments have declining usage and rising support demand?",
    inputs={"lakehouse": source},
    outputs={"answer": dict, "sources_used": list},
    lm=FabricLM("gpt-5.1"),
).run()

The root may use workspace and Lakehouse names or GUIDs. You can also pass a path ending in /Tables, /Tables/<schema>, /Tables/<schema>/<table>, or /Files/<path> to narrow discovery. Multiple LakehouseSource objects can be nested in input lists or dictionaries. Automatic Delta schema discovery requires fabric-rlm[analytics]. Outside a Fabric notebook, or when you already maintain a catalog, pass catalog=[...] to bypass discovery.

Inside the worker, use the resolved catalog helpers instead of trying to call Fabric discovery APIs again:

lakehouse.list_sources(kind="delta")
lakehouse.find_sources("usage")
lakehouse.find_sources("customer_id", kind="delta")

summary = lakehouse.query(
    """
    SELECT c.region, SUM(s.mrr) AS active_mrr
    FROM companies AS c
    JOIN subscriptions AS s USING (company_id)
    WHERE s.status = 'active'
    GROUP BY c.region
    ORDER BY active_mrr DESC
    """,
    sources={
        "companies": "dbo.companies",
        "subscriptions": "dbo.subscriptions",
    },
)

Catalog searches match source names, paths, columns, and data types. They do not widen the Tables or Files scopes supplied by the caller. query() runs in the trusted parent process against only the named catalog entries and returns bounded JSON-safe rows, so Fabric storage credentials are never exposed to the isolated model-generated worker. Queries are parsed before execution: every relation must resolve to a supplied alias or a CTE derived from one, and user-authored table functions, dynamic SQL, external paths, and unrecognized or side-effecting functions fail closed. The parent also applies a 30-second deadline, a 256 MiB DuckDB memory limit with temporary spill disabled, a 10,000-row ceiling, and a 5 MiB serialized-result ceiling. Results are fetched and sized one row at a time so an oversized scalar or row is rejected before the complete result is materialized.

Semantic models

A Power BI semantic model binds the same way and arrives as a connected handle:

from fabric_rlm import FabricLM, RLM, SemanticModel

RLM.task(
    task="Which product line has the highest recurring revenue?",
    inputs={"arr": SemanticModel("ARR Model SF (79)")},
    outputs=["answer"],
    lm=FabricLM("gpt-5.1"),
).run()

Inside the run, arr.schema() returns formatted schema text. For programmatic inspection, arr.metadata() returns ordinary pandas DataFrames with stable snake-case columns for tables, columns, measures, and relationships. arr.dax("EVALUATE ...", normalize_columns=True) returns an ordinary pandas DataFrame with predictable names such as period_year and arr, avoiding SemPy's bracketed result-column conventions. The raw metadata methods and raw arr.dax("EVALUATE ...") behavior remain available. arr.measure(name, groupby=[...], filters={...}) evaluates a model measure without authoring DAX. Pass workspace= for a model outside the attached workspace.

Bind several at once and the model routes between them:

inputs={
    "mfg": SemanticModel("Manufacturing Ops"),
    "arr": SemanticModel("ARR Model SF (79)"),
}

This needs a Fabric notebook, where sempy ships in the runtime. The dataset name is checked when you construct SemanticModel, so a typo fails on that line rather than several turns into a run.

The handle gives generated code a clear entry point. Across two semantic models and two model families, tasks scored 18-19/19 and 13/15 with the handle. The same tasks scored 7/19 and 5/15 when they only named the semantic model.

Submission contract

The runtime injects SUBMIT(...). Call it with keyword arguments matching the declared outputs, or with positional arguments in the same order. After a valid submission, result.payload holds the dictionary and each field is also available as an attribute such as result.answer.

Inspect a run

In a Fabric or Jupyter notebook, render an interactive turn timeline:

result.inspect()

Each turn expands to show the observable model response, executed code, output, errors, validator feedback, submitted payload, timing, and token usage. Slow, error, repair, and submission turns are labeled in the timeline. Model-provider private chain-of-thought is not exposed. The inspector renders open with each turn collapsed, and the turn list scrolls after 15 rows. Use result.inspect(visible_turns=10) to change the viewport or result.inspect(expanded=False) when the whole inspector should start collapsed.

The inspector is dependency-free and escapes trajectory content before rendering. Save the same view as a standalone file when you need to share or archive it:

result.inspect().save_html("rlm-run.html")

Submitted fields normally remain available as result attributes. If an output is named inspect, result.inspect is that submitted value; use RLMResult.inspect(result) to open the run inspector for that result.

Nested model calls

Inside its Python, the model can call a nested model with predict_sync("english -> french", english=phrase) (or the async predict), optionally routed to a cheaper sub_lm=.

Engines

RLM ships with three stable engines, plus the experimental adaptive:

The default core skill carries a PLAN / VERIFY / REFLECT contract: plan before running code, self-check before SUBMIT, and carry prior-attempt failures into retries. It is on by default. Set FABRIC_RLM_PVR=0 to turn it off for token-sensitive batch runs on trivial tasks.

Skills

Skills are Markdown playbooks that tell the model how to do a kind of work properly: which library to reach for, the traps to avoid, and what to check before submitting. Eleven ship with the package. Name the ones a task needs and they are prepended to the prompt. The keyword router can also select them from the task and input names.

from fabric_rlm import RLM, File, FabricLM

rlm = RLM.task(
    task="Rebuild the summary tab from the raw export and flag any variance over 5 percent.",
    inputs={"workbook": File("/lakehouse/default/Files/finance/q3.xlsx")},
    outputs=["answer"],
    lm=FabricLM("gpt-5.1"),
    skills=["excel_modify", "data_exploration"],   # load as many as the task needs
)
print(rlm.run().answer)

The eight domain skills are keyword-routed. For example, data_exploration activates when a task or input name mentions logs or CSV files. The core and utility skills provide planning, validation, and error recovery.

Writing your own

A skill is one Markdown file with a small frontmatter block. Put it anywhere the notebook can read, including Lakehouse Files, and point a SkillLoader at that folder:

from fabric_rlm import RLM, SkillLoader, FabricLM

loader = SkillLoader(skill_dir="/lakehouse/default/Files/skills")
print(loader.list_skills())        # your skills plus the bundled ones

rlm = RLM.task(
    task="Extract the vendor totals from this invoice.",
    inputs={"doc": File("/lakehouse/default/Files/invoices/2026-07.pdf")},
    outputs=["totals"],
    lm=FabricLM("gpt-5.1"),
    skill_loader=loader,
    skills=["invoice_rules", "pdf_document_analysis"],
)

Your folder layers over the bundled skills rather than replacing them, so you can mix your own with the shipped ones in the same skills=[...] list. Pass several folders as a list if you keep them apart, and a file named after a bundled skill overrides it. If you want only your own, pass include_packaged=False.

Contributed skills

contrib-skills/ in the repository holds playbooks that are not installed with the package, either because they are narrower than the bundled ones or because the measurements behind them are thinner than a default install should carry. Point a loader at the folder to use one:

from fabric_rlm import RLM, File, SkillLoader

loader = SkillLoader(skill_dir="contrib-skills")

rlm = RLM.task(
    task="What was Boeing's FY2022 core operating loss? Report the figure with its sign.",
    inputs={"filing": File("BOEING_2022_10K.pdf")},
    outputs=["answer"],
    skill_loader=loader,
    skills=["pdf_document_analysis", "financial_documents"],
)

financial_documents is the first of these: reporting conventions for 10-K, 10-Q and earnings releases, covering parentheses as negative, scale stated in a header, fiscal against calendar year, adjacent period columns and subtotal rows. It is scoped to financial reporting on purpose, since parentheses mean something else in legal and scientific documents. docs/contrib-skills.md records what it was measured on and what the measurement does not support.

If a skill needs a library fabric-rlm does not depend on, install it in the notebook (%pip install python-docx) before running. The sandbox blocks pip and subprocess, so a skill cannot install its own dependencies. See docs/authoring-skills.md.

This is how house rules stop being tribal knowledge: your chart of accounts, the naming conventions your reports use, the columns that are always dates. Write it once, store it beside the data, and every run reads from the same copy.

Start from docs/skill-template.md; the structure is documented in docs/authoring-skills.md.

Security

This library runs model-generated code. The default SecurityPolicy scrubs secret-bearing environment variables from the worker, screens generated code, and blocks destructive Lakehouse operations such as notebookutils.fs.rm and mssparkutils.fs.mv. The worker remains inside the notebook trust boundary. Read SECURITY.md before using untrusted prompts with sensitive data or credentials.

CLI

fabric-rlm --version
fabric-rlm run examples/simple_math/task.json      # run a task from JSON
fabric-rlm trace inspect path/to/trajectory.jsonl  # summarize and diagnose a saved run

Documentation

• QUICKSTART.md: step-by-step guide covering install, first run, Fabric notebook usage, sub-LM calls, traces, and skill authoring.
• docs/fabric-runtime-deps.md: read this if a Fabric notebook fails at import time (Sentinel, yarl.Query, aiohttp.ConnectionTimeoutError).
• docs/lossless-submit-payloads.md: how final payloads avoid namespace-snapshot truncation.
• examples/notebooks/: ready-to-import Fabric recipes. Start with rlm_vs_plain_llm_imf_cpi.ipynb (the with-and-without comparison) and rlm_api_tour.ipynb, then the PDF workflows, the Spark-log root-cause analysis, and the SpreadsheetBench benchmarks.
• CHANGELOG.md: release history.

Develop

git clone https://github.com/pawarbi/fabric-rlm-core.git
cd fabric-rlm-core
pip install -e ".[dev]"
pytest -q

See CONTRIBUTING.md for the development workflow.

Acknowledgments

fabric-rlm builds on the following work:

• The Recursive Language Model paradigm comes from the paper Recursive Language Models by Alex L. Zhang, Tim Kraska, and Omar Khattab (MIT CSAIL), which showed that letting a model programmatically examine and recursively query its own prompt beats stuffing everything into context.
• DSPy provides the RLM predictor and the interpreter protocol this library plugs into, and dspy.LM powers every model backend here.
• Predict-RLM by Trampoline AI, a production-focused RLM runtime built on DSPy signatures, inspired the direction of this project.

License

MIT. See LICENSE.