This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

# baton Python Wrapper

Introduction

Python 3 Wrapper for baton, superseding a [previous implementation in metadata-check] (https://github.com/wtsi-hgi/metadata-check/blob/9cd5c41b0f2e254fc1d6249a14752bd428587bb7/irods_baton/baton_wrapper.py).

The wrapper provides access to most of baton’s functionality.

How to use

Prerequisites

  • Python >= 3.5.2
  • baton >= 0.16.4
  • iRODS >= 4.1.9

Note: Although older version of baton/iRODS will probably work, the library is only aimed at the versions specified above.

Installation

Stable releases can be installed via PyPI:

$ pip3 install baton

Bleeding edge versions can be installed directly from GitHub:

$ pip3 install git+https://github.com/wtsi-hgi/python-baton-wrapper.git@<commit_id_or_branch_or_tag>#egg=baton

To declare this library as a dependency of your project, add it to your requirement.txt file.

API

Setup

To use the iRODS API, you must first define a “connection” to an iRODS server:

from baton.api import connect_to_irods_with_baton, Connection

# Setup connection to iRODS using baton
irods = connect_to_irods_with_baton("/where/baton/binaries/are/installed/", skip_baton_binaries_validation=False) # type: Connection

Data Objects and Collections

The API provides the ability to retrieve models of the data objects and collections stored on an iRODS server. Similarly to the JSON that baton provides, the models do not contain the payloads. They do however provide access to all of the information that baton can retrieve about an entity, including Access Control Lists (ACLs), custom metadata (AVUs), the content of collections and information about data object replicas. All methods provide the option to not load AVUs.

from baton.models import DataObject, Collection, SearchCriterion, ComparisonOperator

# Get models of data objects or collections at the given path(s) in iRODS
irods.data_object.get_by_path("/collection/data_object", load_metadata=False)    # type: DataObject:
irods.collection.get_by_path(["/collection", "/other_collection"])   # type: Sequence[Collection]:

# Setup search for data objects or collections based on their metadata
search_criterion_1 = SearchCriterion("attribute", "match_value", ComparisonOperator.EQUALS)
search_criterion_2 = SearchCriterion("other_attribute", "other_match_value", ComparisonOperator.LESS_THAN)
# Do search to get models of data objects or collections
irods.data_object.get_by_metadata(search_criterion_1, zone="OptionalZoneRestriction")   # type: Sequence[DataObject]
irods.collection.get_by_metadata([search_criterion_1, search_criterion_2], load_metadata=False)   # type: Sequence[Collection]

# Get models of data objects or collections contained within a collection(s)
irods.collection.get_all_in_collection("/collection", load_metadata=False)    # type: Sequence[Collection]
irods.data_object.get_all_in_collection(["/collection", "/other_collection"])   # type: Sequence[DataObject]

Metadata (AVUs)

The API provides the ability to both retrieve and manipulate the custom metadata (AVUs) associated with data objects and collections.

Warning: there is currently no support for reading/writing the unit property of AVUs.

Although the type of metadata is the same for both data objects and collections, due to the way iRODS works, it is necessary to know the type of entity that a path corresponds to in order to retrieve metadata.

from baton.collections import IrodsMetadata

metadata_1 = IrodsMetadata({"key": {"value_1"}})
metadata_2 = IrodsMetadata({"another_key": {"value_1", "value_2"}})

# Metadata (methods available for both `data_object` and `collection`)
irods.data_object.metadata.get_all("/collection/data_object")   # type: IrodsMetadata
irods.collection.metadata.get_all(["/collection", "/other_collection"])   # type: Sequence[IrodsMetadata]

# `metadata_1` is added to the data object with the first path in the list and `metadata_2` is added to the second
irods.data_object.metadata.add(["/collection/data_object", "/other_data_object"], [metadata_1, metadata_2])
irods.collection.metadata.add("/collection", metadata_1)

irods.data_object.metadata.set("/collection/data_object", metadata_1)
# `metadata_1` is added to both collections in the list
irods.collection.metadata.set(["/collection", "/other_collection"], metadata_1)

irods.data_object.metadata.remove(["/collection/data_object", "/other_data_object"], [metadata_1, metadata_2])
irods.collection.metadata.remove("/collection", metadata_1)

irods.data_object.metadata.remove_all("/collection/data_object")
irods.collection.metadata.remove_all(["/collection", "/other_collection"])

Access Control Lists (ACLs)

The API provides the ability to both retrieve and manipulate the access control lists (ACLs) associated with data objects and collections.

from baton.models import AccessControl

# ACLs. Note: it is implied that the owner is in the same zone as the entity to which the access control is applied
acl_examples = [
    AccessControl(User("user_1", "zone_user_is_in"), AccessControl.Level.READ),
    AccessControl(User("group_1", "zone_group_is_in"), AccessControl.Level.WRITE),
    AccessControl("user_1#zone_user_is_in", AccessControl.Level.OWN)
]

irods.data_object.access_control.get_all("/collection/data_object") # type: Set[AccessControl]
irods.collection.access_control.get_all(["/collection", "/another/collection"])  # type: List[Set[AccessControl]]

irods.data_object.access_control.add_or_replace(["/collection/data_object", "/another/data_object"], acl_examples[0])
irods.collection.access_control.add_or_replace("/collection", acl_examples, recursive=True)

irods.data_object.access_control.set("/collection/data_object", acl_examples[1])
irods.collection.access_control.set(["/collection", "/another/collection"], acl_examples[0], recursive=False)

irods.data_object.access_control.revoke(["/collection/data_object", "/another/data_object"], acl_examples)
irods.collection.access_control.revoke("/collection", acl_examples[1], recursive=True)

irods.data_object.access_control.revoke_all(["/collection/data_object", "/another/data_object"])
irods.collection.access_control.revoke_all("/collection", recursive=True)

Custom objects via specific queries

iRODS supports specific queries which return new types of object. In order to use such custom objects in iRODS via this library, a custom model of the object should to be made. Then, a subclass of BatonCustomObjectMapper needs to be defined to specify how a specific query (or number of specific queries) can be used to retrieve from and/or modify the object in iRODS.

The API provides the ability to retrieve the queries that are installed on an iRODS server (ironically, by use of a specific query!):

from baton.models import SpecificQuery

# Get specific queries that have been installed on the iRODS server
irods.specific_query.get_all(zone="OptionalZoneRestriction")  # type: Sequence[SpecificQuery]

JSON Serialization/Deserialization

There are JSON encoders and decoders for nearly all iRODS object models in this library. These can be used to convert models to/from their baton defined JSON representations. All serializers/deserializers extend JSONEncoder and JSONDecoder (most through use of the hgijson library) meaning that they can be used with Python’s built in “json` package <https://docs.python.org/3/library/json.html>`__:

import json
from baton.json import DataObjectJSONEncoder, DataObjectJSONDecoder, CollectionJSONEncoder, CollectionJSONDecoder, IrodsMetadataJSONEncoder, IrodsMetadataJSONDecoder, AccessControlJSONEncoder, AccessControlJSONDecoder

data_object_as_json_string = json.dumps(data_object, cls=DataObjectJSONEncoder)     # type: str
data_object = json.loads(data_object_as_json_string, cls=DataObjectJSONDecoder)     # type: DataObject

collection_as_json_string = json.dumps(collection, cls=CollectionJSONEncoder)   # type: str
collection = json.loads(collection_as_json_string, cls=CollectionJSONDecoder)   # type: Collection

metadata_as_json_string = json.dumps(metadata, cls=IrodsMetadataJSONEncoder)    # type: str
metadata = json.loads(metadata_as_json_string, cls=IrodsMetadataJSONDecoder)    # type: IrodsMetadata

acl_as_json_string = json.dumps(metadata, cls=AccessControlJSONEncoder)     # type: str
acl = json.loads(acl_as_json_string, cls=AccessControlJSONDecoder)  # type: List[AccessControl]

Development

Setup

Install both library dependencies and the dependencies needed for testing:

$ pip3 install -q -r requirements.txt
$ pip3 install -q -r test_requirements.txt

A baton installation is not required.

Some tests use Docker therefore a Docker daemon must be running on the test machine, with the environment variables DOCKER_TLS_VERIFY, DOCKER_HOST and DOCKER_CERT_PATH set.

Testing

Using nosetests, in the project directory, run:

$ nosetests -v --cover-inclusive --tests baton/tests, baton/tests/_baton

To generate a test coverage report with nosetests:

$ nosetests -v --with-coverage --cover-package=baton --cover-inclusive --tests baton/tests, baton/tests/_baton

License

LGPL license.

Copyright (c) 2015, 2016 Genome Research Limited

Release History

Release History

1.0.1

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
baton-1.0.1.tar.gz (36.2 kB) Copy SHA256 Checksum SHA256 Source Oct 21, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting