notebookutils.fs: file system and mounting library

generic notebookutils.fs library on arbitrary local/VM/K8s environment, for developing and running pySpark notebooks without Databricks / Azure Synapse / Microsoft Fabric environments.

Introduction to dbutils and mssparkutils

In cloud Platform-as-a-Service (PaaS) environments, dbutils and mssparkutils are essential, built-in utility packages designed to bridge the gap between PySpark code and the underlying cloud infrastructure.

While Apache Spark natively excels at distributed data processing, it lacks built-in mechanisms for environment-specific orchestration. These vendor-specific libraries step in to provide that critical operational control layer.

• dbutils (Databricks Utilities): The native toolset for the Databricks platform. It allows engineers to interactively navigate the Databricks File System (DBFS), securely retrieve credentials via Azure Key Vault or AWS Secrets Manager, parameterize notebooks using widgets, and chain multiple notebooks into modular workflows.
• mssparkutils / notebookutils in Fabric (Microsoft Spark Utilities): The direct counterpart built for Microsoft Azure Synapse Analytics and Microsoft Fabric. It offers near-identical functionality and APIs tailored for the Microsoft ecosystem, enabling developers to manage files in Azure Data Lake Storage (ADLS) Gen2, handle Microsoft Entra ID tokens, and control notebook pipeline execution paths.

Ultimately, both mssparkutils and dbutils abstract away complex cloud APIs, allowing data engineers to write secure, scalable, and maintainable data pipelines without leaving their interactive pySpark notebook environments.

Technical overview of notebookutils.fs

Cloud file operations(put, get, rm) use fsspec backend. While the original dbutils use hadoop backend via Py4J, this library choose fsspec for compatibility with non-Spark environments, such as pure python, DuckDB and StarRocks.

POSIX mounts use

• blobfuse v2 for Azure Blob Storage and ADLFS G2
• s3-mount for Amazon S3 and s3-compatible object storage(OVH/Hertzner/DigitalOcean/Linode etc)
• gcsfuse for Google Cloud Storage

Installation

Install notebookutils

pip install notebookutils

Cloud storage backends are optional extras — install only what you need:

pip install notebookutils[s3]     # Amazon S3 / S3-compatible (s3fs)
pip install notebookutils[azure]  # ADLS Gen2 / Blob (adlfs + azure-identity)
pip install notebookutils[gcs]    # Google Cloud Storage (gcsfs)
pip install notebookutils[all]    # everything

For POSIX mounting, install the FUSE binary for your storage type: mount-s3, blobfuse2, or gcsfuse.

Usage

from notebookutils import fs

# Memory-space file operations (fsspec-backed)
fs.ls("s3://my-bucket/data")                    # -> [FileInfo(...), ...]
fs.put("s3://my-bucket/hello.txt", "hi", overwrite=True)
fs.head("s3://my-bucket/hello.txt")
fs.cp("s3://my-bucket/data", "abfss://cont@acct.dfs.core.windows.net/data", recurse=True)  # cross-cloud
fs.mkdirs("gs://my-bucket/new_dir")
fs.exists("s3://my-bucket/hello.txt")
fs.rm("s3://my-bucket/hello.txt")

# POSIX mounting (FUSE-backed, process-safe)
fs.mount("s3://my-bucket", "/mydata")
path = fs.getMountPath("/mydata")
with open(f"{path}/hello.txt") as f:
    print(f.read())
fs.mounts()          # list active mount points
fs.unmount("/mydata")

fs.help()            # full API overview

Legacy Synapse code works too: import notebookutils installs an in-session mssparkutils alias, or simply import notebookutils as mssparkutils.

Full API documentation: see MANUAL.md.

Explicit configuration (optional)

If environment discovery isn't enough (e.g. S3-compatible endpoints), override per protocol:

fs.configure("s3", key="...", secret="...",
             client_kwargs={"endpoint_url": "https://minio.example:9000"},
             mount_options={"allow_other": False})

Configuration Files

notebookutils use the same environmental credentials as major cloud vendor SDKs for authenticating to cloud storage. This includes:

• DefaultAzureCredential Chain
• AWS Boto3 Default Credential Provider Chain
• GCP Application Default Credentials Chain
• ~/.aws/credentials for generic S3-compatible storage

The expectation is if you can access storage with cloud vendor python SDKs, notebookutils.fs is able to access and mount storage.

Development

pip install -e '.[dev]'
pytest tests/unit          # unit tests (no credentials needed)
python -m build            # build sdist + wheel

Integration tests (opt-in)

Integration tests use the same credential chains as the official SDKs and are skipped unless credentials resolve and target URLs are set:

pytest tests/integration -m s3        # requires AWS credentials + NBU_TEST_S3_URL
pytest tests/integration -m fuse      # additionally requires FUSE binaries