An FSSpec Implementation using the Pelican System
Project description
PelicanFS
Overview
PelicanFS is a file system interface (fsspec) for the Pelican Platform. For more information about pelican, see our main website or Github page. For more information about fsspec, visit the filesystem-spec page.
Limitations
PelicanFS is built on top of the http fsspec implementation. As such, any functionality that isn’t available in the http implementation is also not available in PelicanFS.
Installation
To install pelican, run:
pip install pelicanfs
To install from source, run:
git clone https://github.com/PelicanPlatform/pelicanfs.git
cd pelicanfs
pip install -e .
Using PelicanFS
To use pelicanfs, first create a PelicanFileSystem
and provide it with the url for the director of your data federation. As an example using the OSDF director
from pelicanfs.core import PelicanFileSystem
pelfs = PelicanFileSystem("https://osdf-director.osg-htc.org/")
Once pelfs
is pointed at your federation's director, fsspec commands can be applied to Pelican namespaces. For example:
hello_world = pelfs.cat('/ospool/uc-shared/public/OSG-Staff/validation/test.txt')
print(hello_world)
Getting an FSMap
Sometimes various systems that interact with an fsspec want a key-value mapper rather than a url. To do that, call the PelicanMap
function with the namespace path and a PelicanFileSystem
object rather than using the fsspec get_mapper
call. For example
from pelicanfs.core import PelicanFileSystem, PelicanMap
pelfs = PelicanFileSystem(“some-director-url”)
file1 = PelicanMap(“/namespace/file/1”, pelfs=pelfs)
file2 = PelicanMap(“/namespace/file/2”, pelfs=pelfs)
ds = xarray.open_mfdataset([file1,file2], engine='zarr')
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pelicanfs-0.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 31c354f944f411292f795276211fdf32d30a27438615704c28e86058b68f89e4 |
|
MD5 | 22d644f85d4b07e0bf1d7968da880c1f |
|
BLAKE2b-256 | 1e01d59cdf7885947a2c7abdf4f8cd5e67d1b32e1092d0bedffb3ec4398cf2ec |