opendalfs

opendalfs is a Python-based interface for file systems that enables interaction with different storage services by Apache OpenDAL. Through opendalfs, users can utilize fsspec's standard API to operate on all storage services supported by OpenDAL.

URL Protocols

opendalfs registers multiple fsspec protocols in the form of opendal+<service>, for example:

import fsspec

f = fsspec.open(
    "opendal+s3://my-bucket/path/to/file",
    mode="rb",
    endpoint="http://localhost:9000",
    access_key_id="minioadmin",
    secret_access_key="minioadmin",
)

The URL host is mapped to the service container (e.g. bucket for s3/gcs, container for azblob), and the URL path is used as the object key.

For other OpenDAL services, register protocols at runtime:

import opendalfs

opendalfs.register_opendal_service("oss")

Installation

Basic Installation

pip install opendalfs

Development Installation

Install uv and just, then run:

just install

Development Setup

This project uses:

• Python 3.12+ for the Python interface
• ruff for code formatting and linting
• ty for type checking
• pytest for testing

Run the local checks and unit tests with:

just check
just unit

For development setup and guidelines, see our Contributing Guide.

Benchmarks

The benchmark script compares Arrow direct, opendalfs (fsspec), and s3fs (fsspec) on MinIO.

just install
just bench --sizes 16,32,64 --files 4 --workers 4

Configure MinIO access via OPENDAL_S3_ENDPOINT, OPENDAL_S3_BUCKET, OPENDAL_S3_REGION, OPENDAL_S3_ACCESS_KEY_ID, and OPENDAL_S3_SECRET_ACCESS_KEY. Compose, tests, and benchmarks read the same values.

The benchmark target starts MinIO from the root docker-compose.yml. Stop it when you finish:

just bench-down

For profiling, you can install a tool with uv (for example py-spy) and run:

uv tool install py-spy
uv tool run py-spy record -o bench.svg -- python bench/bench_read_write.py --sizes 16,32,64 --files 4 --workers 4

High write concurrency can stall on some systems. If runs time out, reduce --fsspec-workers.

Status

See Tracking issues of 0.1.0 version for opendalfs

Contributing

opendalfs is an exciting project currently under active development. Whether you're looking to use it in your projects or contribute to its growth, there are several ways you can get involved:

• Follow the Contributing Guide to contribute
• Create new Issue for bug reports or feature requests
• Join discussions in Discussions

Getting Help

• Submit issues for bug reports
• Ask questions in discussions

License

Licensed under Apache License, Version 2.0.