binaryrain-fabric-local-pyspark

Run Microsoft Fabric PySpark code on your own machine without paying for capacity on every run.

The package gives you a local Spark session. The first time your code reads a lakehouse table, a lakehouse file, or a Fabric SQL endpoint query, the data is downloaded to a cache folder in your project. Every later run reads from that folder and does not touch Fabric.

Installation

pip install binaryrain-fabric-local-pyspark

Requirements:

• Python 3.12 or newer.
• Java 17 or newer, with JAVA_HOME set. PySpark needs it to start.
• ODBC Driver 18 for SQL Server, only if you use spark.sql(...) against the Fabric SQL endpoint.
• An Azure sign-in that can read the lakehouse, for example az login. The package uses DefaultAzureCredential.

Quick start

Set the environment variables before you import the package. The cache location is read at import time.

FABRIC_WORKSPACE_ID=<workspace id or name>
FABRIC_LAKEHOUSE_ID=<lakehouse id or name>
FABRIC_SERVER=<sql endpoint host>.datawarehouse.fabric.microsoft.com
FABRIC_DEFAULT_DATABASE=<lakehouse name>

from binaryrain_fabric_local_pyspark.spark import get_local_spark

spark = get_local_spark()

# Delta table in the default lakehouse, cached as parquet.
customers = spark.read.parquet("Tables/dbo/kunden")

# File or folder under Files, cached as is.
orders = spark.read.parquet("Files/onPrem/mssql/Orders.parquet")

# Text file or folder under Files, one row per line in the "value" column.
lines = spark.read.text("Files/raw/export.csv")

# Full abfss URI, any workspace or lakehouse you can read.
other = spark.read.parquet(
    "abfss://<workspace>@onelake.dfs.fabric.microsoft.com/<lakehouse>/Tables/dbo/kunden"
)

# T-SQL against the SQL endpoint. The result is cached by query text.
top = spark.sql("SELECT TOP 100 * FROM dbo.kunden")

get_local_spark() returns a normal SparkSession with a few changes. spark.read.parquet(...) and spark.read.text(...) resolve Fabric paths through the cache, spark.sql(...) runs your query on the Fabric SQL endpoint when you pass no parameters, and writes to lakehouse paths and tables go to the cache, see Writes.

The Fabric SQL endpoint speaks T-SQL, not Spark SQL. If you pass args or keyword parameters, spark.sql skips the endpoint and runs on local Spark instead.

Temp views work too. When the session has at least one temp view, spark.sql first runs the query on local Spark. If local Spark cannot parse the query, for example because of T-SQL syntax like TOP 100, or cannot find a referenced table, for example a name like lh_gold.dbo.kunden, the query goes to the Fabric SQL endpoint as before. Any other error, such as a missing column on a temp view, is raised directly.

spark.read.parquet("Files/onPrem/mssql/Kunden.parquet").createOrReplaceTempView("REGIO_MAPPING")

# Runs on local Spark against the temp view.
mapping = spark.sql("SELECT Kundennummer FROM REGIO_MAPPING WHERE Kundennummer IS NOT NULL")

Paths that are not Fabric paths, such as C:\data\file.parquet, are passed through unchanged.

Fabric notebooks provide a global display function. Import the local version to use the same calls outside Fabric:

from binaryrain_fabric_local_pyspark.display import display

display(mapping)  # Spark DataFrame, first 1000 rows
display(mapping, summary=True)  # Spark DataFrame.summary() statistics
display(pandas_frame)  # pandas DataFrame, describe() when summary=True
display(rows)  # List of Rows, dicts or named tuples
display(widget)  # Any IPython widget or object

In Jupyter it renders through IPython. Without IPython it prints to the console.

Writes

Writes to a lakehouse go to the local cache, never to OneLake. Your local runs cannot overwrite production data, and you need no Delta Lake or Hadoop setup.

customers.write.mode("overwrite").option("overwriteSchema", "true").format("delta").saveAsTable(
    "lh_gold.dbo.customers"
)

# Reads the table you just wrote from the cache.
gold = spark.table("lh_gold.dbo.customers")

• saveAsTable always writes to the cache. customers and dbo.customers use the default lakehouse. lh_gold.dbo.customers uses the lakehouse named lh_gold in the default workspace, and ws.lh_gold.dbo.customers also names the workspace. A missing schema means dbo.
• save and parquet write to the cache when the path is an abfss URI or a relative path such as Tables/dbo/x or Files/out. Local paths go to Spark unchanged.
• delta and parquet formats are stored as parquet files. Other formats raise FabricWriteError. Without a format, delta is used, like in Fabric.
• Save modes behave like Spark. overwrite replaces the data, append adds files, ignore skips existing targets, and the default mode fails when the target exists.
• append checks that columns, types and partitionBy columns match. Set option("mergeSchema", "true") to allow new or changed columns.
• partitionBy writes column=value folders, which are read back as columns.
• spark.table and spark.read.table read local Spark tables and temp views first, then resolve Fabric table names through the cache. Uncached tables are downloaded from OneLake as Parquet. For example, spark.table("LH_Silver.src_banqpro.KTO") uses FABRIC_WORKSPACE_ID and reads LH_Silver.Lakehouse/Tables/src_banqpro/KTO. Set FABRIC_CACHE_REFRESH=1 to download again. Authentication and download errors propagate instead of being replaced by Spark namespace errors.

What gets cached

Default layout:

<project>/.cache/
    fabric/<workspace>/<lakehouse>/<Tables|Files>/<relative path>
    fabric/<workspace>/<lakehouse name>.Lakehouse/Tables/<schema>/<table>
    sql/<hash>.parquet
    spark.log

The second line is for tables written with a lakehouse name, such as lh_gold.dbo.x.

Cached folders are read file by file, so Spark on Windows needs no winutils.exe or hadoop.dll.

A cache hit never checks Fabric. If the remote data changes, refresh the cache yourself (see below).

Downloads go to a .tmp path first and are moved into place when they finish, so a failed download does not leave a half-written cache entry. Add .cache/ to your project's .gitignore.

Configuration

All settings are environment variables.

The default cache folder is relative to the directory you start Python from. Run your script from the project root, or set FABRIC_CACHE_DIR to an absolute path. spark.log and the sql cache sit one level above FABRIC_CACHE_DIR.

The package does not load .env files for you. Load them first, for example with python-dotenv, and do it before importing the package.

Workspace and lakehouse names

OneLake rejects paths that mix a workspace GUID with a lakehouse name such as LH_Silver.Lakehouse. On a cache miss, the package therefore looks up workspace and lakehouse names through the Fabric REST API and downloads with the GUIDs. The .Lakehouse suffix is optional. GUIDs pass through without a lookup.

The lookup uses the same Azure credential as the downloads and needs at least Viewer access to the workspace. Listings are kept in memory for the session. Call binaryrain_fabric_local_pyspark.lakehouses.clear_cache() after renaming a lakehouse.

The cache stays keyed by the name you wrote, so cache hits need no network call and existing caches stay valid.

from binaryrain_fabric_local_pyspark.lakehouses import resolve_lakehouse_id, resolve_workspace_id

workspace_id = resolve_workspace_id("My Workspace")
lakehouse_id = resolve_lakehouse_id(workspace_id, "LH_Silver")

Refreshing the cache

Set FABRIC_CACHE_REFRESH=1 to refresh everything for one run. To refresh a single table or file, call the helpers directly:

from binaryrain_fabric_local_pyspark.fabric import cache_file, cache_table, resolve_local_path

cache_table("Tables/dbo/kunden", refresh=True)
cache_file("Files/raw/orders.parquet", refresh=True)

# Returns a local path for Fabric inputs, the same string for anything else.
local_path = resolve_local_path("Tables/dbo/kunden")

You can also delete the folder under .cache.

Errors

Every error the package raises on purpose derives from FabricLocalSparkError in binaryrain_fabric_local_pyspark.exceptions.

A path that does not exist in the lakehouse, or a workspace or lakehouse name the REST API does not know, raises the built-in FileNotFoundError. A malformed abfss URI raises ValueError.

from binaryrain_fabric_local_pyspark.exceptions import FabricAuthenticationError

try:
    df = spark.read.parquet("Tables/dbo/kunden")
except FabricAuthenticationError:
    print("Run 'az login' and try again.")

Modules

Limits

• Tables are cached as plain parquet. Read them with spark.read.parquet, not format("delta").
• Only spark.read.parquet and spark.read.text are wired to the cache. Other readers such as csv or json get no Fabric path resolution.
• spark.read.text accepts only Files paths. A Tables path raises FabricConfigurationError, because tables are cached as parquet.
• Wildcards such as Files/raw/*.txt are not supported. Pass the folder instead. Files and folders whose names start with _ or . are skipped.
• A single spark.sql query cannot join a temp view with a table that exists only on the SQL endpoint. Load that table with spark.read.parquet and register it as a temp view first.
• display renders a static table. It has no interactive chart view like Fabric.
• The cache has no expiry and no size limit.
• The package only reads from Fabric. Writes go to the local cache and never reach OneLake or the SQL endpoint.
• spark.sql does not see tables you wrote locally. It queries the Fabric SQL endpoint, which has the production data. Use spark.table(...), or register the table as a temp view first.
• FABRIC_CACHE_REFRESH=1 and cache_table(..., refresh=True) replace locally written tables with the Fabric version.
• lh_gold.dbo.x and Tables/dbo/x are cached in different folders, even when lh_gold is your default lakehouse. The same holds for a lakehouse referenced once by name and once by GUID.
• A name that matches several workspaces or lakehouses raises FabricConfigurationError. Use the GUID instead.
• Writes collect the whole DataFrame on the driver, so very large writes need enough memory.
• insertInto and writeTo are not redirected. Delta options such as overwriteSchema and replaceWhere are ignored.
• A spark.read.parquet call with several Fabric paths lists folders in Spark, which needs Hadoop's native libraries on Windows. Read one path per call instead. spark.read.text has no such limit, because it lists each cached folder itself.

License

See LICENSE.