opendalfs

opendalfs is a fsspec filesystem backed by Apache OpenDAL. It lets Python libraries that work with fsspec use storage services supported by OpenDAL.

Documentation: opendalfs.readthedocs.io

Installation

opendalfs requires Python 3.12 or newer.

pip install opendalfs

Quick start

The OpenDAL memory service provides a small example that needs no credentials and writes nothing to disk:

from opendalfs import OpendalFileSystem

fs = OpendalFileSystem("memory")
fs.pipe_file("hello.txt", b"hello from opendalfs\n")

assert fs.cat_file("hello.txt") == b"hello from opendalfs\n"

OpendalFileSystem implements the fsspec filesystem interface, including methods such as open, ls, glob, info, and rm.

Connect to storage

The package registers fsspec protocols for S3, Google Cloud Storage, and Azure Blob Storage:

import fsspec

fs = fsspec.filesystem(
    "opendal+s3",
    bucket="my-bucket",
    region="us-east-1",
)

These protocols also work in URLs accepted by fsspec-compatible libraries:

opendal+s3://my-bucket/path/to/file
opendal+gcs://my-bucket/path/to/file
opendal+azblob://my-container/path/to/file

Register other OpenDAL services at runtime:

import fsspec

from opendalfs import register_opendal_service

protocol = register_opendal_service("memory")
fs = fsspec.filesystem(protocol)

Service options are passed to OpenDAL without being renamed. See the documentation for storage configuration, URL rules, supported operations, tested integrations, and the API reference.

Community

• Read the contributing guide to set up a development environment and submit changes.
• Open an issue for bugs and feature requests.
• Use GitHub Discussions for questions and general discussion.

License

opendalfs is licensed under the Apache License 2.0.