onedatafilerestclient 25.0.0

Onedata REST file API client

OnedataFileRESTClient

OnedataFileRESTClient is a pure Python client to the Onedata file REST API. It offers basic operations on files as a concise, low-level library. Most users will probably be more interested in OnedataRESTFS library, which is a plugin for PyFilesystem2, implemented on top of this library, providing more stable and user friendly interface.

Supported Onezone versions: >= 21.02.5.

Supported Oneprovider versions: >= 21.02.5.

Installation

You can install OnedataFileRESTClient from PyPI as follows:

pip install onedatafilerestclient

Make sure to install a version not newer than the Onedata Onezone service in your deployment. New versions of this library are published only when some changes are made or a new major Onedata release is published, so there might not be an exact version matching current Onedata release.

Usage

Creating a OnedataFileRESTClient client instance

In order to use the OnedataFileRESTClient we need to first import necessary module and then create an instance of OnedataFileRESTClient, providing at least two required parameters:

• onedata_onezone_host – Hostname of the Onezone instance to connect to
• onedata_access_token – Onedata access token with Oneprovider REST API privileges (see Access tokens guide)

from onedatafilerestclient import *

onedata_onezone_host = 'onezone.example.com'
onedata_access_token = 'MDAzM2xvY2F00aW9uIGRldi1vbmV6b25lLmRlZmF1bHQuc3...'
client = OnedataFileRESTClient(onedata_onezone_host, onedata_access_token)

The OnedataFileRESTClient class provides the following operations (examples below assume OnedataFileRESTClient has been setup as shown above). For more details you can check the method documentation in the implementation of the OnedataFileRESTClient on GitHub.

Common API conventions

For most methods of OnedataFileRESTClient, there are following common conventions for passing arguments to methods which perform operations on Onedata filesystem:

• first argument is the name of the data space
• next arguments are keyword arguments:
  • at least one keyword argument file_path or file_id, which need to be specified separately from the first argument and contain path relative to the space directory (for file_path) or file ID (for file_id)
  • argument specific to particular operation

Get token scope

Returns a dictionary with a map of data spaces available through specified access token, including some attributes such a list of Oneprovider instances supporting each space:

>>> client.get_token_scope()
{
    'validUntil': None,
    'dataAccessScope': {
        'spaces': {
            '67ae143419b1838715505bbad66096d2chfcf2': {
                'supports': {
                    '71eb92fb45b414fc87d47aab3cf4a6b8ch836c': {
                        'readonly': False
                    }
                },
                'name': 'MyData'
            }
        },
        'readonly': False,
        'providers': {
            '71eb92fb45b414fc87d47aab3cf4a6b8ch836c': {
                'version': '21.02.8',
                'online': True,
                'name': 'oneprovider1',
                'domain': 'oneprovider1.example.com'
            }
        }
    }
}

List spaces

Returns the list of data space names effectively available to the user based on specified access token.

>>> client.list_spaces()
['MyData']

Get space id

Resolves a space name to a an internal Onedata space ID:

>>> client.get_space_id('MyData')
'67ae143419b1838715505bbad66096d2chfcf2'

Get file id

Resolves a path to a directory or file to a unique file ID (file here can refer to file, directory or entire space):

>>> client.get_file_id('MyData')
'000000000058E0AC677569642373706163655F3637616531343334313962313833383731353530356262616436363039366432636866636632233637616531343334313962313833383731353530356262616436363039366432636866636632'

Corresponding API endpoint is lookup_file_id.

Get attributes

This method returns attributes for specified space, directory or file. It accepts the following optional keyword arguments:

• attributes - list of specific attributes to return
• file_path - path to the file relative to the space
• file_id - unique file ID
• provider - specific Oneprovider to use for the operation

For example, to get attributes for MyData space:

>>> client.get_attributes('MyData')
{'fileId': '000000000058E0AC677569642373706163655F3637616531343334313962313833383731353530356262616436363039366432636866636632233637616531343334313962313833383731353530356262616436363039366432636866636632',
 'parentFileId': None,
 'name': 'MyData',
 'posixPermissions': '775',
 'atime': 1748857564,
 'mtime': 1748856735,
 'ctime': 1748856735,
 'type': 'DIR',
 'size': 0,
 'directShareIds': [],
 'index': 'g2gDZAAKbGlzdF9pbmRleG0AAAAmNjdhZTE0MzQxOWIxODM4NzE1NTA1YmJhZDY2MDk2ZDJjaGZjZjJkAAl1bmRlZmluZWQ',
 'displayGid': 0,
 'displayUid': 0,
 'ownerUserId': 'VIRTUAL_SPACE_OWNER_67ae143419b1838715505bbad66096d2chfcf2',
 'originProviderId': '71eb92fb45b414fc87d47aab3cf4a6b8ch836c',
 'hardlinkCount': 1}

To get attributes for a specific file, we need to also specify the path within the space:

>>> client.get_attributes('MyData', file_path='/file.txt')
{'fileId': '000000000052E01B67756964236432666666313638346261626132626630623661333262353638653733333633636866636632233637616531343334313962313833383731353530356262616436363039366432636866636632',
 'parentFileId': '000000000058E0AC677569642373706163655F3637616531343334313962313833383731353530356262616436363039366432636866636632233637616531343334313962313833383731353530356262616436363039366432636866636632',
 'name': 'file.txt',
 'posixPermissions': '664',
 'atime': 1748862683,
 'mtime': 1748862675,
 'ctime': 1748862675,
 'type': 'REG',
 'size': 4,
 'directShareIds': [],
 'index': 'g2gDZAAKbGlzdF9pbmRleG0AAAAIZmlsZS50eHRtAAAAJjcxZWI5MmZiNDViNDE0ZmM4N2Q0N2FhYjNjZjRhNmI4Y2g4MzZj',
 'displayGid': 0,
 'displayUid': 1976898,
 'ownerUserId': '52d4038aab393bc675783924836879e9ch393b',
 'originProviderId': '71eb92fb45b414fc87d47aab3cf4a6b8ch836c',
 'hardlinkCount': 1}

Corresponding API endpoint is get_args.

Set attributes

This method allows to modify selected file or directory attributes, specified by attributes arguments as a dictionary. It accepts the following optional keyword arguments:

• file_path - path to the file relative to the space
• file_id - unique file ID
• provider - specific Oneprovider to use for the operation

For example we can modify file permissions:

>>> client.set_attributes('MyData', file_path='/file.txt', attributes={'mode': '666'})
>>> client.get_attributes('MyData', file_path='/file.txt', attributes=['posixPermissions'])
{'posixPermissions': '666'}

Corresponding API endpoint is set_args.

List children

Returns the list of direct children of a given directory (specified by file_path or file_id argument). It accepts the following optional keyword arguments:

• attributes - list of Onedata file attributes to return (e.g. size)
• limit - maximum number of children to return (default: 1000)
• continuation_token - token for pagination to get next page of results
• file_path - path to the directory relative to the space
• file_id - unique file ID of the directory
• provider - specific Oneprovider to use for the operation

>>> client.list_children('MyData', file_path='dir1', limit=5)
{'nextPageToken': 'g2gEZAAQcGFnaW5hdGlvbl90b2tlbmgDZAAKbGlzdF9pbmRleG0AAAAKZmlsZTEyLnR4dG0AAAAmNzFlYjkyZmI0NWI0MTRmYzg3ZDQ3YWFiM2NmNGE2YjhjaDgzNmNkAAl1bmRlZmluZWRkAARtb3Jl',
 'isLast': False,
 'children': [{'name': 'file0.txt', 'type': 'REG'},
              {'name': 'file1.txt', 'type': 'REG'},
              {'name': 'file10.txt', 'type': 'REG'},
              {'name': 'file11.txt', 'type': 'REG'},
              {'name': 'file12.txt', 'type': 'REG'}]}

To get the next page of result, we need to pass the nextPageToken as continuation_token argument:

>>> client.list_children('MyData', file_path='dir1', limit=5, continuation_token='g2gEZAAQcGFnaW5hdGlvbl90b2tlbmgDZAAKbGlzdF9pbmRleG0AAAAKZmlsZTEyLnR4dG0AAAAmNzFlYjkyZmI0NWI0MTRmYzg3ZDQ3YWFiM2NmNGE2YjhjaDgzNmNkAAl1bmRlZmluZWRkAARtb3Jl')
{'nextPageToken': 'g2gEZAAQcGFnaW5hdGlvbl90b2tlbmgDZAAKbGlzdF9pbmRleG0AAAAKZmlsZTE3LnR4dG0AAAAmNzFlYjkyZmI0NWI0MTRmYzg3ZDQ3YWFiM2NmNGE2YjhjaDgzNmNkAAl1bmRlZmluZWRkAARtb3Jl',
 'isLast': False,
 'children': [{'name': 'file13.txt', 'type': 'REG'},
              {'name': 'file14.txt', 'type': 'REG'},
              {'name': 'file15.txt', 'type': 'REG'},
              {'name': 'file16.txt', 'type': 'REG'},
              {'name': 'file17.txt', 'type': 'REG'}]}

and continue as long as isLast returns False.

Corresponding API endpoint is list_children.

Get file or directory contents

This method allows to read an entire file. It accepts the following optional keyword arguments:

• offset - byte offset to start reading from (default: 0)
• size - number of bytes to read
• file_path - path to the file relative to the space
• file_id - unique file ID
• provider - specific Oneprovider to use for the operation

For example, to read an entire file:

>>> client.get_file_content('MyData', file_path='file.txt')
b'TEST'

or specified range in one request:

>>> client.get_file_content('MyData', file_path='file.txt', offset=2, size=2)
b'ST'

If requested file is a directory, this method returns a TAR archive with its contents. Any nested files or subdirectories, to which the client does not have access (e.g. due to insufficient POSIX permissions or ACLs) are omitted in the resulting archive.

>>> tar_bytes = client.get_file_content('MyData', file_path='dir1')
>>> tar = tarfile.open(fileobj=io.BytesIO(tar_bytes))
>>> files = [entry.name for entry in tar.getmembers()]
>>> files[0:5]
['dir1', 'dir1/file0.txt', 'dir1/file1.txt', 'dir1/file10.txt', 'dir1/file11.txt']

Corresponding API endpoint is download_file_content_by_path.

Iter file content

This method returns a Python iterator, allowing to read file in chunks of specified size. It accepts the following optional keyword arguments:

• chunk_size - size of each chunk in bytes (default: 1048576 = 1 MB)
• file_path - path to the file relative to the space
• file_id - unique file ID
• provider - specific Oneprovider to use for the operation

For example:

>>> stream = client.iter_file_content('MyData', chunk_size=2, file_path='file.txt')
>>> [chunk for chunk in stream]
[b'TE', b'ST']

Corresponding API endpoint is download_file_content_by_path.

Put file content

This method allows to append or modify the contents of a file. The file must exist beforehand. It accepts the following optional keyword arguments:

• offset - byte offset to start writing at
• file_path - path to the file relative to the space
• file_id - unique file ID
• provider - specific Oneprovider to use for the operation

>>> client.create_file('MyData', file_path='file2.txt')
'0000000000527B5767756964236333633832643730316165633034333139666634666161353331363861393465636866636632233637616531343334313962313833383731353530356262616436363039366432636866636632'
>>> client.put_file_content('MyData', b'ABCD', file_path='file2.txt')
>>> client.get_file_content('MyData', file_path='file2.txt')
b'ABCD'

It's also possible to specify an offset after which to write the content in the file:

>>> client.put_file_content('MyData', b'EFGH', file_path='file2.txt', offset=2)
>>> client.get_file_content('MyData', file_path='file2.txt')
b'ABEFGH'

Corresponding API endpoint is update_file_content.

Create file or directory

Creates a file at path specified by file_path argument. Additionally it accepts the following optional keyword arguments:

• file_type - The file type to create, it can be one of:
  • REG - regular file
  • DIR - directory
  • LNK - hard link
  • SYMLNK - symbolic link
• create_parents - If the parent path does not exist and argument is set to true, the operation will attempt to create intermediate parent directories.
• mode - integer argument can be passed to specify POSIX permissions for the file, (e.g. 0665).

If the file already exists, the operation fails with an error.

>>> client.create_file('MyData', file_path='dir3', file_type='DIR')
'000000000052DBCF67756964236462363362636366653466373662336464333334316235396537653636613564636861383337233637616531343334313962313833383731353530356262616436363039366432636866636632'

>>> client.create_file('MyData', file_path='dir3/file3.txt')
'000000000052EEA567756964233236346335393335643762386238363731363261663062393761303363653532636861383337233637616531343334313962313833383731353530356262616436363039366432636866636632'

Corresponding API endpoint is create_file.

Remove

Removes file or directory specified by file_path (or file_id) argument. It accepts the following optional keyword arguments:

• file_path - path to the file relative to the space
• file_id - unique file ID
• provider - specific Oneprovider to use for the operation

In case of a directory, all its children are recursively removed - note that the operation will fail part-way if the client does not have permissions to remove some of the nested files/directories.

>>> client.remove('MyData', file_path='dir3/file3.txt')

Corresponding API endpoint is remove_file_at_path.

Move

This method allows to rename a file or directory within a space. It accepts the following optional keyword arguments:

• provider - specific Oneprovider to use for the operation

>>> client.move('MyData', 'file.txt', 'MyData', 'file_new.txt')

To move file between different spaces, it has to be copied using read and write operations and then removed from the old location (or use OnedataRESTFS).

References

• PyFilesystem2
• OnedataRESTFS
• Onedata access tokens