Skip to main content

OnedataRESTFSSpec

OnedataRESTFSSpec is an fsspec filesystem implementation for Onedata, providing a unified interface for accessing Onedata spaces using Python fsspec API.

Features

  • Full fsspec compatibility
  • Support for reading and writing files
  • Directory operations (list, create, remove)
  • File metadata access
  • Authentication via access tokens
  • Environment variable configuration
  • URI-based configuration
  • OpenTelemetry metrics export

Installation

Install from PyPi:

pip install onedatarestfsspec

Install from the local directory:

pip install -e ./onedatarestfsspec

Requirements

  • Python >= 3.10
  • fsspec >= 2021.10.0
  • onedatafilerestclient >= 25.0.0
  • aiohttp

Usage

Basic Usage

import fsspec

# Using explicit parameters
fs = fsspec.filesystem(
    'onedata',
    onezone_host='https://datahub.egi.eu',
    token='your_access_token'
)

# List spaces
spaces = fs.ls('/')
print("Available spaces:", spaces)

# List files in a space
files = fs.ls('/your-space/')
print("Files in space:", files)

# Read a file
with fs.open('/your-space/path/to/file.txt', 'rb') as f:
    content = f.read()

# Write a file
with fs.open('/your-space/path/to/newfile.txt', 'wb') as f:
    f.write(b'Hello, Onedata!')

# Get file info
info = fs.info('/your-space/path/to/file.txt')
print("File size:", info['size'])

For a full example see here.

Environment Variables

Set environment variables for automatic configuration:

export ONEDATA_ONEZONE_HOST="https://datahub.egi.eu"
export ONEDATA_TOKEN="your_access_token"
export ONEDATA_PREFERRED_PROVIDERS="provider1.example.com,provider2.example.com"
export ONEDATA_VERIFY_SSL="true"
export ONEDATA_TIMEOUT="30"

Then use without explicit parameters:

import fsspec

# Configuration will be loaded from environment variables
fs = fsspec.filesystem('onedata')

URL-based Configuration

import fsspec

# URL format: onedata://token@onezone_host/path?query_params
url = "onedata://your_token@datahub.egi.eu/your-space/path/to/file.txt"

# Open file directly from URL
with fsspec.open(url, 'r') as f:
    content = f.read()

Advanced Usage

import fsspec

fs = fsspec.filesystem(
    'onedata',
    onezone_host='https://datahub.egi.eu',
    token='your_access_token',
    preferred_providers=['provider1.example.com'],
    verify_ssl=True,
    timeout=60,
    auto_mkdir=True  # Automatically create parent directories
)

# Copy files
fs.cp('/your-space/source.txt', '/your-space/destination.txt')

# Upload local file
fs.put('/local/path/file.txt', '/your-space/remote/file.txt')

# Download file
fs.get('/your-space/remote/file.txt', '/local/path/downloaded.txt')

# Create directory
fs.makedirs('/your-space/new/directory')

# Remove file
fs.rm('/your-space/file/to/remove.txt')

# Check if file exists
if fs.exists('/your-space/some/file.txt'):
    print("File exists!")

# Get detailed listing
files = fs.ls('/your-space/', detail=True)
for file_info in files:
    print(f"{file_info['name']}: {file_info['size']} bytes")

OpenTelemetry Metrics

OnedataRESTFSSpec can export data-access metrics to any OpenTelemetry-compatible collector (Prometheus, Grafana Tempo, Jaeger, etc.) using the OTLP protocol.

Installation

Metrics support is an optional feature. Install the HTTP exporter (recommended) or the gRPC exporter, or both:

# HTTP/protobuf transport (default)
pip install 'onedatarestfsspec[monitoring]'

# gRPC transport only
pip install opentelemetry-exporter-otlp-proto-grpc

Enabling metrics

Metrics are disabled by default. Enable them either via a constructor keyword argument or an environment variable:

import fsspec

fs = fsspec.filesystem(
    'onedata',
    onezone_host='https://datahub.egi.eu',
    token='your_access_token',
    metrics_enabled=True,
)
export ONEDATA_METRICS_ENABLED=true

Configuring the OTLP exporter

The exporter is configured through standard OpenTelemetry environment variables or the corresponding constructor keyword arguments. Keyword arguments take precedence over environment variables.

Constructor kwarg Environment variable Default Description
metrics_enabled ONEDATA_METRICS_ENABLED false Enable metrics export
otlp_endpoint OTEL_EXPORTER_OTLP_METRICS_ENDPOINT collector default Full URL of the OTLP collector endpoint
otlp_protocol OTEL_EXPORTER_OTLP_PROTOCOL http/protobuf Transport protocol: http/protobuf or grpc
otlp_export_interval_ms — 60000 How often metrics are flushed (milliseconds)
otlp_session_id ONEDATA_OTLP_SESSION_ID auto-generated UUID Session identifier attached to every metric as the session_id attribute. Useful for correlating metrics from a single run across restarts or multiple filesystem instances.

Example — HTTP/protobuf transport:

import fsspec

fs = fsspec.filesystem(
    'onedata',
    onezone_host='https://dev-onezone.default.svc.cluster.local',
    token='your_access_token',
    metrics_enabled=True,
    verify_ssl=False,
    otlp_endpoint='http://localhost:9090/api/v1/otlp/v1/metrics',
    otlp_protocol='http/protobuf',
    otlp_export_interval_ms=30_000,
)

Example — pinning a session ID:

In order to make it easier to identify sets of metrics collected from a specific OnedataRESTFSSpec client, it's possible to add otlp_session_id parameter to the constructor with custom ID representing an OnedataRESTFSSpec session:

import fsspec

fs = fsspec.filesystem(
    'onedata',
    onezone_host='https://datahub.egi.eu',
    token='your_access_token',
    metrics_enabled=True,
    otlp_session_id='my-experiment-run-1',
)

# Retrieve the active session ID (auto-generated when not set explicitly)
print(fs.otlp_session_id)

Alternatively the session id can be specified using an environment variable:

export ONEDATA_OTLP_SESSION_ID=my-experiment-run-1

Example — gRPC transport via environment variables:

export ONEDATA_METRICS_ENABLED=true
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://otel-collector.example.com:4317
fs = fsspec.filesystem(
    'onedata',
    onezone_host='https://datahub.egi.eu',
    token='your_access_token',
)

Available metrics

All metrics carry the following attributes, allowing time-series to be filtered and grouped by space, file, session, or operation type:

Attribute Description
space_id Onedata space identifier
file_id Onedata internal file identifier
provider_id Domain of the Oneprovider that served the request
session_id Identifier for this filesystem instance's session (see otlp_session_id)
operation "read" or "write"
Metric name Type Unit Description
onedata_file_access_total Counter ops Total number of read and write operations. Use the operation attribute to distinguish reads from writes.
onedata_read_bytes Counter bytes Cumulative bytes read from Onedata.
onedata_written_bytes Counter bytes Cumulative bytes written to Onedata.
onedata_read_duration Histogram seconds Per-operation read latency.
onedata_write_duration Histogram seconds Per-operation write latency.
onedata_file_throughput_bytes_per_second Histogram bytes/s Observed transfer throughput (bytes / duration) for each operation.

Example: querying metrics in Prometheus

After scraping via an OpenTelemetry Collector with a Prometheus exporter, the metrics are available under their original names (dots replaced by underscores where required by Prometheus).

# Read throughput (bytes/s) averaged over 5 minutes, by space
rate(onedata_read_bytes_total[5m]) by (space_id)

# 99th-percentile read latency per file
histogram_quantile(0.99, rate(onedata_read_duration_bucket[5m])) by (file_id)

# Write operation rate
rate(onedata_file_access_total[1m]) by (space_id, operation)

# Total bytes read for a specific session
sum(onedata_read_bytes_total{session_id="my-experiment-run-1"})

Path Format

Paths in OnedataRESTFSSpec follow the format: /space-name/path/within/space

  • Root path / lists all available spaces
  • Space path /space-name refers to the root of a specific space
  • File path /space-name/dir/file.txt refers to a file within a space

onedatarestfsspec client also supports URI-based paths, i.e. specifying onedata:///space-name/path/within/space when opening a file or directory using generic fsspec calls such as:

 f = fsspec.open('onedata:///MySpace/file.txt')

will automatically select onedatarestfsspec client to open the file via Onedata, provided that the necessary parameters are provided (see here).

Configuration Priority

Configuration is resolved in the following order (highest to lowest priority):

  1. Explicit parameters passed to fsspec.filesystem()
  2. URL parameters (when using URL-based access)
  3. Environment variables

Error Handling

OnedataRESTFSSpec translates Onedata REST API errors to standard Python exceptions:

  • FileNotFoundError for missing files/directories
  • IOError for general I/O errors
  • ValueError for invalid parameters
  • PermissionError for access denied errors

Limitations

  • Cross-space file operations are not supported
  • Symbolic links are treated as regular files

Development

To set up for development:

# Install dependencies
pip install -e ./onedatafilerestclient
pip install -e ./onedatarestfsspec[test]

# Run tests
pytest onedatarestfsspec/tests/

# Run with coverage
pytest --cov=onedatarestfsspec onedatarestfsspec/tests/

License

This project is licensed under the MIT License - see the LICENSE file for details.

Metadata

Release files for onedatarestfsspec 25.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for onedatarestfsspec 25.2.0
File Size Uploaded Provenance Actions
onedatarestfsspec-25.2.0.tar.gz 27.9 kB Without provenance Details for onedatarestfsspec-25.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for onedatarestfsspec 25.2.0
File Interpreter ABI Platform Actions
onedatarestfsspec-25.2.0-py3-none-any.whl Python 3 none any Details for onedatarestfsspec-25.2.0-py3-none-any.whl

Total release size: 45.1 kB

Release files / onedatarestfsspec-25.2.0.tar.gz

Download URL onedatarestfsspec-25.2.0.tar.gz
Size 27.9 kB
Tags Source
SHA-256 checksum
How to use checksums
568a5eed737b7503f1a3a07c0d5ca9a5924660b7267bc6f870c76f887ed83a5b
BLAKE2b-256 checksum
How to use checksums
289025223d65ecb665a269a464e005e31edeca50a5dd16657e50a3e9db1c4db1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.3

Release files / onedatarestfsspec-25.2.0-py3-none-any.whl

Download URL onedatarestfsspec-25.2.0-py3-none-any.whl
Size 17.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1a51bd4e49429a20e1a7e47fdd06d217e65c102db47487d13b458f9621455f30
BLAKE2b-256 checksum
How to use checksums
591d34904d2ad172e83202b56ce35a249ba32e24e8ab6a5b9cd29b7724ff4b7b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.3

Security for onedatarestfsspec 25.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page