Skip to main content

PyPI Status Python Version License

Tests Codecov pre-commit Black

Features

morefs provides standalone fsspec-based filesystems like:

  • AsyncLocalFileSystem that provides async implementation of LocalFileSystem.

  • In-memory filesystems DictFileSystem built on nested dictionaries and MemFS built on tries, and are much faster than fsspec’s MemoryFileSystem.

  • OverlayFileSystem that allows to overlay multiple fsspec-based filesystems.

Installation

You can install morefs via pip from PyPI:

$ pip install morefs

You might need to install with extras for some filesystems:

$ pip install morefs[asynclocal]  # for installing aiofile dependency for AsyncLocalFileSystem
$ pip install morefs[memfs]  # for installing pygtrie dependency for MemFS

Usage

AsyncLocalFileSystem

Extended version of LocalFileSystem that also provides async methods.

import asyncio
from morefs.asyn_local import AsyncLocalFileSystem

async def main():
    fs = AsyncLocalFileSystem(auto_mkdir=False)

    f = await fs.open_async("foo", mode="w")
    async with f:
        await f.write("foobar")

    content = await fs._cat("foo")
    print(content)
    print(fs.cat("foo"))  # you can still use sync methods

asyncio.run(main())

DictFS

DictFS is a nested dictionary-based, in-memory filesystem and acts more like a real LocalFileSystem.

from morefs.dict import filesystem

fs = DictFS()

MemFS

MemFS is a trie-based in-memory filesystem, and acts like a bucket storage.

from morefs.memory import MemFS

fs = MemFS()

OverlayFileSystem

from morefs.overlay import OverlayFileSystem

# use localfilesystem for write, overlay all filesystems for read
fs = OverlayFileSystem(file={"auto_mkdir": True}, s3={"anon": True})
# or you can pass filesystem instances directly
# as variable positional arguments or with keyword argument `filesystems=[]`
fs = OverlayFileSystem(LocalFileSystem(), s3={"anon": True})

Contributing

Contributions are very welcome. To learn more, see the Contributor Guide.

License

Distributed under the terms of the Apache 2.0 license, morefs is free and open source software.

Issues

If you encounter any problems, please file an issue along with a detailed description.

Metadata

Release files for morefs 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for morefs 0.2.2
File Size Uploaded
morefs-0.2.2.tar.gz 24.7 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for morefs 0.2.2
File Interpreter ABI Platform
morefs-0.2.2-py3-none-any.whl Python 3 none any Details

Total release size: 39.1 kB

Release files / morefs-0.2.2.tar.gz

Download URL morefs-0.2.2.tar.gz
Size 24.7 kB
Tags Source
SHA-256 checksum
How to use checksums
6119d12c975d1a7e89d0837c99cd5fc82e7e392e74ecb041634ab394d137f8e7
BLAKE2b-256 checksum
How to use checksums
1c979aabe5577bb6829dd42067c9ed787fe413a4006f23a7f47587406374cae0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/5.1.0 CPython/3.12.4

Release files / morefs-0.2.2-py3-none-any.whl

Download URL morefs-0.2.2-py3-none-any.whl
Size 14.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e55fa00287f5c41f242dbe431ea229b8eac4490fb8e09e2cc3c32c049a4c093b
BLAKE2b-256 checksum
How to use checksums
521e48d88a1bd17dfc54e712099f4090030b2bba12458c772c29ee18f730f707
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/5.1.0 CPython/3.12.4

Release history Release notifications | RSS feed

This release

0.2.2 This release

2 release files

0.2.1

2 release files

0.2.0

2 release files

0.1.2

2 release files

0.1.1

2 release files

0.1.0

2 release files

0.0.3

2 release files

0.0.2

2 release files

0.0.1

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page