fsspec-databricks

File system interface for Databricks file system"s".

fsspec-databricks provides a fsspec-compliant file system implementation that unifies access to Databricks file systems, including:

• Unity Catalog Volumes
• Workspace files
• Legacy DBFS (Databricks File System)

The library routes dbfs:/ and POSIX-style paths to the appropriate Databricks file system implementation and supports copying and streaming between them.

Features

• Provides seamless access to files in different Databricks file systems with DBFS URLs (dbfs:/path/to/file) or POSIX paths (/path/to/file).
  • Automatically routes file operations to appropriate file systems based on file path patterns.
  • Implements file operations across different file systems, for example, copying a file from Workspace to Unity Catalog Volume or vice versa.
• Falls back to the local file system when running inside a Databricks workspace.
• Implemented on Databricks Python SDK.
  • Uses Databricks Unified Authentication

Compatibility

• Python 3.10 to 3.14
• databricks-sdk: 0.99.0 or later
• fsspec: 2024.6.0 or later
• aiohttp: 3.12.0 or later
• Databricks workspace: Tested on the following environments at the moment.
  • Databricks Free Edition
  • Azure Databricks
  • Databricks on Google Cloud

Project status

The current status of this library is early beta. Its API and behavior are subject to change during further development and testing.

• The current version relies on the undocumented multipart upload API for Unity Catalog Volume file write, which Databricks does not officially support and may change without notice.
• For more details about the current limitations, see the Limitations section below.

Getting started

Installation

You can install fsspec-databricks from PyPI.

# with pip
pip install fsspec-databricks
# with UV
uv add fsspec-databricks

Usage

Then you can directly instantiate DatabricksFileSystem in fsspec_databricks module.

from fsspec_databricks import DatabricksFileSystem

fs = DatabricksFileSystem()

Or, you can register DatabricksFileSystem as the default file system implementation for dbfs:/ URL scheme by calling fsspec_databricks.use().

import fsspec
import fsspec_databricks

fsspec_databricks.use()

fs = fsspec.filesystem("dbfs")  # DatabricksFileSystem

For more details on how to use the fsspec file system objects, see fsspec's documentation.

For detailed usage — common operations, POSIX paths, backend-specific behaviour and configuration parameters, see USAGE.md.

Supported file paths

fsspec-databricks supports file paths with dbfs:/ scheme.

It uses the same path patterns as Databricks to map dbfs:/ and POSIX paths to the appropriate file system implementation.

Examples:

fs.ls("dbfs:/Volumes/my_catalog/my_schema/my_volume/path")  # Access Unity Catalog Volume files
fs.ls("dbfs:/Workspace/Users/user-a/path")  # Access workspace files
fs.ls("dbfs:/data/path")  # Access legacy DBFS files

fsspec-databricks supports also stripped, POSIX-like paths without dbfs:/ scheme.

Examples:

fs.ls("/Volumes/my_catalog/my_schema/my_volume/path")  # Access Unity Catalog Volume files
fs.ls("/Workspace/Users/user-a/path")  # Access workspace files (only in DBFS-disabled workspace)
fs.ls("/data/path")  # Access legacy DBFS files

For more details about dbfs:/ and POSIX path support in Databricks, see the official documentation.

Authentication

fsspec-databricks uses Databricks Unified Authentication provided by Databricks Python SDK.

You can find information about supported authentication parameters and environment variables in the Databricks Python SDK documentation.

Default authentication

If Databricks Unified Authentication is configured, fsspec-databricks will pick up credentials from the default profile. For more, see the above Databricks SDK docs.

from fsspec_databricks import DatabricksFileSystem

fs = DatabricksFileSystem()

with fs.open("dbfs:/Volumes/...") as f:
    ...

Via constructor parameters

You can programmatically configure authentication by passing parameters to DatabricksFileSystem constructor.

# Authentication with PAT
fs = DatabricksFileSystem(host=host_url, token=access_token)

# Use different profile
fs = DatabricksFileSystem(profile="production")

Via environment variables

Or, you can configure authentication via environment variables.

# Shell
export DATABRICKS_CONFIG_PROFILE=production

# Then in Python
fs = DatabricksFileSystem()  # will use the "production" profile

By fsspec configuration

You can use the fsspec's configuration model to configure and persist authentication parameters.

With WorkspaceClient

You can create DatabricksFileSystem by explicitly setting Databricks SDK's WorkspaceClient object. The created DatabricksFileSystem instance will use the authentication configured in the provided WorkspaceClient object.

from databricks.sdk import WorkspaceClient

client = WorkspaceClient(...)
...

fs = DatabricksFileSystem(client=client)

Note: a DatabricksFileSystem created with a WorkspaceClient will generally not be serializable, because WorkspaceClient instances are not serializable. Consider using other configuration methods if you need serializable filesystem objects.

Configuration options

In addition to the authentication parameters, fsspec-databricks supports the following configuration options.

For backend-specific parameters and all other options, see USAGE.md.

Differences from the original DatabricksFileSystem in fsspec

fsspec provides its own implementation of DatabricksFileSystem (fsspec.implementations.DatabricksFileSystem).

The main difference between DatabricksFileSystem in fsspec-databricks and the original one in fsspec is that the original one is for legacy DBFS (Databricks File System), which Databricks has already deprecated.

Databricks currently supports workspace files and Unity Catalog volumes in addition to the legacy DBFS, and it continues to use the dbfs:/ URL scheme for both legacy DBFS and the other file systems (documentation).

fsspec-databricks primarily aims to support new file systems (workspace files and Unity Catalog volumes) and enable seamless access to them using the same dbfs:/ URL scheme supported in Databricks workspaces.

Limitations

The following features are not yet implemented or have not been tested yet.

• Compatibility with Databricks on AWS (not tested)
• Legacy DBFS support (not tested)

We are actively developing and testing the library, and we welcome contributions and feedback from the community.

License

Apache License 2.0. See LICENSE for more details.