Pythonic file system for CHFS
Project description
fsspec-chfs
fsspec-chfs is a file system interface to CHFS. CHFS is a parallel consistent hashing file system created instantly using node-local storages such as persistent memory and NVMe SSD for high performance computing. This repository includes the integration for Dask.
Requirements
Getting Started
Installation
$ pip install fsspec-chfs
Create file system
You can create CHFS by chfsctl
and set CHFS_SERVER
environmental variable.
$ eval `chfsctl start`
$ chlist # show started servers
How to use fsspec-chfs
import fsspec
fs = fsspec.filesystem('chfs')
with fs.open('/hello') as f:
f.write(b'world')
Use in Dask
fsspec-chfs provides CHFSClientDaemon
plugin for Dask worker, and it optimizes CHFS initialization/termination in Dask.
client = Client(LocalCluster())
plugin = CHFSClientDaemon()
client.register_worker_plugin(plugin)
def func(path, data):
fs = fsspec.filesystem("chfs_stub")
fs.pipe(path, data)
return 0
future = client.submit(func, "/tmp/foo", b'abcde')
counts = future.result()
Developing
VSCode devcontainer
You can use VSCode devcontainer to develop fsspec-chfs. The setup steps are follows:
- Install Docker and Remote-Container extension.
- Press Ctrl+Shift+P in VSCode.
- select
Remote-Container: Open the folder in the Container
Testing
$ eval `chfsctl start` # start the server and set CHFS_SERVER
$ tox
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
fsspec-chfs-0.0.1.tar.gz
(4.9 kB
view hashes)
Built Distribution
Close
Hashes for fsspec_chfs-0.0.1-py3-none-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 40969c05bc841f9fbce534fe695a945daa7d8a07b6d4a26cfed4c0b7514f9196 |
|
MD5 | 91705ae6ff208398cb98d1a80db96fe5 |
|
BLAKE2b-256 | 8c80bc7c41c8f5d4b6b0abe5cedf06f2c3cded509cbb329418bb2d90b84d0685 |