hiro-graph-client 5.3.0

Hiro Client for Graph REST API of HIRO 7

HIRO Graph API Client

This is a client library to access data of the HIRO Graph.

This library also contains classes for handling the WebSockets event-ws and action-ws API.

To process large batches of data, take a look at "hiro-batch-client". - ( PyPI: hiro-batch-client, GitHub: hiro-batch-client-python)

For more information about HIRO Automation, look at https://www.arago.co/

For more information about the APIs this library covers, see https://developer.hiro.arago.co/7.0/api/

Currently, implemented are

• HiroApp for app
• HiroAuth for auth
• HiroGraph for graph
• HiroIam for iam
• HiroKi for ki
• HiroAuthz for authz
• HiroVariables for variables

and for the websockets

• AbstractEventWebSocketHandler for event-ws
• AbstractActionWebSocketHandler for action-ws

Quickstart

To use this library, you will need an account at https://id.arago.co/ and access to an OAuth Client-Id and Client-Secret to access the HIRO Graph. See also https://developer.hiro.arago.co.

Most of the documentation is done in the sourcecode.

HiroGraph Example

Example to use the straightforward graph api client:

from hiro_graph_client import PasswordAuthTokenApiHandler, HiroGraph

hiro_client: HiroGraph = HiroGraph(
    api_handler=PasswordAuthTokenApiHandler(
        root_url="https://core.arago.co",
        username='',
        password='',
        client_id='',
        client_secret=''
    )
)

# The commands of the Graph API are methods of the class HIROGraph.
# The next line executes a vertex query for instance. 
query_result = hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')

print(query_result)

TokenApiHandler

Authorization against the HIRO Graph is done via tokens. These tokens are handled by classes of type AbstractTokenApiHandler in this library. Each of the Hiro-Client-Object (HiroGraph, , HiroApp, etc.) need to have some kind of TokenApiHandler at construction.

This TokenApiHandler is also responsible to determine the most up-to-date endpoints for the API calls. You can supply a custom list of endpoints by using the dict parameter custom_endpoints= on construction.

A custom list of headers can also be set via the dict parameter headers= in the constructor. These would update the internal headers. Header names can be supplied in any upper/lower-case.

This library supplies the following TokenApiHandlers:

---

FixedTokenApiHandler

A simple TokenApiHandler that is generated with a preset-token at construction. Cannot update its token.

---

EnvironmentTokenApiHandler

A TokenApiHandler that reads an environment variable (default is HIRO_TOKEN) from the runtime environment. Will only update its token when the environment variable changes externally.

---

PasswordAuthTokenApiHandler

This TokenApiHandler logs into the HiroAuth backend and obtains a token from login credentials. This is also the only TokenApiHandler (so far) that automatically tries to renew a token from the backend when it has expired.

---

All code examples in this documentation can use these TokenApiHandlers interchangeably, depending on how such a token is provided.

The HiroGraph example from above with another customized TokenApiHandler:

from hiro_graph_client import EnvironmentTokenApiHandler, HiroGraph

hiro_client: HiroGraph = HiroGraph(
    api_handler=EnvironmentTokenApiHandler(
        root_url="https://core.arago.co"
    )
)

# The commands of the Graph API are methods of the class HIROGraph.
# The next line executes a vertex query for instance. 
query_result = hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')

print(query_result)

Example with additional parameters:

from hiro_graph_client import EnvironmentTokenApiHandler, HiroGraph

hiro_client: HiroGraph = HiroGraph(
    api_handler=EnvironmentTokenApiHandler(
        root_url="https://core.arago.co",
        env_var='_TOKEN',
        headers={
            'X-Custom-Header': 'My custom value'
        },
        custom_endpoints={
            "graph": "/api/graph/7.2",
            "auth": "/api/auth/6.2"
        },
        client_name="HiroGraph (testing)"  # Will be used in the header 'User-Agent'
    )
)

# The commands of the Graph API are methods of the class HIROGraph.
# The next line executes a vertex query for instance. 
query_result = hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')

print(query_result)

Token Handler sharing

When you need to access multiple APIs of HIRO, share the TokenApiHandler between the API objects to avoid unnecessary requests for token- and version-information against HIRO. The TokenApiHandler will share a requests.Session, token- and version-request-handling, and the token string itself between them.

from hiro_graph_client import HiroGraph, HiroApp, PasswordAuthTokenApiHandler

hiro_api_handler = PasswordAuthTokenApiHandler(
    root_url="https://core.arago.co",
    username='',
    password='',
    client_id='',
    client_secret=''
)

hiro_client: HiroGraph = HiroGraph(
    api_handler=hiro_api_handler
)

hiro_app_client: HiroApp = HiroApp(
    api_handler=hiro_api_handler
)

Connection sharing

You can also let TokenApiHandlers share a common connection session instead of letting each of them create their own. This might prove useful in a multithreading environment where tokens have to be set externally or change often (i.e. one token per user per thread). This also ensures, that version-requests happen only once when the connection is initialized.

Use the parameters pool_maxsize and pool_block to further tune the connection parameters for parallel access to the backend. See requests Session Objects and the Python documentation of requests.adapters.HTTPAdapter and GraphConnectionHandler.

from hiro_graph_client import HiroGraph, HiroApp, FixedTokenApiHandler, GraphConnectionHandler

connection_handler = GraphConnectionHandler(
    root_url="https://core.arago.co",
    pool_maxsize=200,                     # Optional: Max pool of cached connections for this connection session
    pool_block=True,                      # Optional: Do not allow more parallel connections than pool_maxsize
    client_name="Your Graph Client 0.0.1" # Optional: Will be used in the header 'User-Agent'
)

# Work with token of user 1

user1_client: HiroGraph = HiroGraph(
    api_handler=FixedTokenApiHandler(
        connection_handler=connection_handler,
        token='token user 1'
    )
)

# Work with token of user 2 (Shared Token Handler)

user2_api_handler = FixedTokenApiHandler(
    connection_handler=connection_handler,
    token='token user 2'
)

user2_client: HiroGraph = HiroGraph(
    api_handler=user2_api_handler
)

hiro_app_client: HiroApp = HiroApp(
    api_handler=user2_api_handler
)

Everything written in Token Handler Sharing still applies.

SSL Configuration

SSL parameters are configured using the class SSLConfig. This class translates the parameters given to the required fields for the requests library of Python (parameters cert and verify there). This configuration is given to the TokenApiHandlers and will be used by the clients attached to it as well.

If this is not set, the default settings of the library requests will be used, which is to verify any server certificates by using system defaults.

Example: Disable verification

from hiro_graph_client import EnvironmentTokenApiHandler, HiroGraph, SSLConfig

hiro_client: HiroGraph = HiroGraph(
    api_handler=EnvironmentTokenApiHandler(
        root_url="https://core.arago.co",
        # Disable any verification.
        ssl_config=SSLConfig(verify=False)
    )
)

query_result = hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')

print(query_result)

Example: Set custom SSL certificates

from hiro_graph_client import EnvironmentTokenApiHandler, HiroGraph, SSLConfig

hiro_client: HiroGraph = HiroGraph(
    api_handler=EnvironmentTokenApiHandler(
        root_url="https://core.arago.co",
        # Set custom certification files. If any of them are omitted, system defaults will be used.
        ssl_config=SSLConfig(
            cert_file="<path to client certificate file>",
            key_file="<path to key file for the client certificate>",
            ca_bundle_file="<path to the ca_bundle to verify the server certificate>"
        )
    )
)

query_result = hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')

print(query_result)

Graph Client "HiroGraph"

The Graph Client is mostly straightforward to use, since all public methods of this class represent an API call in the Graph API. Documentation is available in source code as well. Some calls are a bit more complicated though and explained in more detail below:

Attachments

To upload data to such a vertex, use HiroGraph.post_attachment(data=...). The parameter data= will be given directly to the call of the Python library requests as requests.post(data=...). To stream data, set data to an object of type IO. See the documentation of the Python library requests for more details.

Downloading an attachment is done in chunks, so huge blobs of data in memory can be avoided when streaming this data. Each chunk is 64k by default.

To stream such an attachment to a file, see the example below:

ogit_id = '<ogit/_id of a vertex>'
data_iter = hiro_client.get_attachment(ogit_id)

with io.start("attachment.bin", "wb") as file:
    for chunk in data_iter:
        file.write(chunk)

To read the complete data in memory, see this example:

ogit_id = '<ogit/_id of a vertex>'
data_iter = hiro_client.get_attachment(ogit_id)

attachment = b''.join(data_iter)

WebSockets

This library contains classes that make using HIRO WebSocket protocols easier. They handle authentication, exceptions and much more.

The classes do not handle buffering of messages, so it is the duty of the programmer to ensure, that incoming messages are either handled quickly or being buffered to avoid clogging the websocket. The classes are thread-safe, so it is possible to handle each incoming message asynchronously in its own thread and have those threads send results back if needed (See multithreaded example in Action WebSocket).

Closing WebSockets

To shut these WebSockets (ws) down cleanly, please consider these scenarios:

Default behaviour

The library reacts on KeyboardInterrupt (SIGINT) and closes the WebSocket cleanly with closing message to the sever when such an interrupt is received. If another signal (like SIGTERM) is received, the program will stop immediately without any closing message back to the server.

Signal handling

When installing signal handlers, you need to use ws.signal_stop() to shut the WebSocket down. Do NOT use ws.stop() or the closing process will deadlock. This is because the signal interrupt is executed in the same thread as ws.run_forever().

Example:

import signal

[...]

with ActionWebSocket(api_handler=FixedTokenApiHandler('HIRO_TOKEN')) as ws:
    def signal_handler(signum, handler):
        ws.signal_stop()


    signal.signal(signal.SIGINT, signal_handler)
    signal.signal(signal.SIGTERM, signal_handler)
    signal.signal(signal.SIGHUP, signal_handler)  # This signal might not be available on MS Windows 

    ws.run_forever()

Closing from a separate thread

When closing the WebSocket from another thread than the one running ws.run_forever(), you should use ws.stop(). This ensures, that the shutdown is synchronized and ws.stop() will return after the WebSocket has been closed.

Event WebSocket

This websocket receives notifications about changes to vertices that match a certain filter.

See also API description of event-ws

Example:

import threading

from hiro_graph_client.clientlib import FixedTokenApiHandler
from hiro_graph_client.eventswebsocket import AbstractEventsWebSocketHandler, EventMessage, EventsFilter


class EventsWebSocket(AbstractEventsWebSocketHandler):

    def on_create(self, message: EventMessage):
        """ Vertex has been created """
        print("Create:\n" + str(message))

    def on_update(self, message: EventMessage):
        """ Vertex has been updated """
        print("Update:\n" + str(message))

    def on_delete(self, message: EventMessage):
        """ Vertex has been removed """
        print("Delete:\n" + str(message))


events_filter = EventsFilter(filter_id='testfilter', filter_content="(element.ogit/_type=ogit/MARS/Machine)")

with EventsWebSocket(api_handler=FixedTokenApiHandler('HIRO_TOKEN'),
                     events_filters=[events_filter],
                     query_params={"allscopes": "false", "delta": "false"}) as ws:
    ws.run_forever()  # Use KeyboardInterrupt (Ctrl-C) to exit. 

If you do not set the parameter scope=, the default scope of your account will be used. If you need to set the scope by hand, use the following:

[...]

api_handler = FixedTokenApiHandler('HIRO_TOKEN')

default_scope = api_handler.decode_token()['data']['default-scope']

with EventsWebSocket(api_handler=api_handler,
                     events_filters=[events_filter],
                     scopes=[default_scope],
                     query_params={"allscopes": "false", "delta": "false"}) as ws:
    ws.run_forever()  # Use KeyboardInterrupt (Ctrl-C) to exit. 

Action WebSocket

This websocket receives notifications about actions that have been triggered within a KI. Use this to write your own custom action handler.

See also API description of action-ws

Simple example:

import threading

from hiro_graph_client.actionwebsocket import AbstractActionWebSocketHandler
from hiro_graph_client.clientlib import FixedTokenApiHandler


class ActionWebSocket(AbstractActionWebSocketHandler):

    def on_submit_action(self, action_id: str, capability: str, parameters: dict):
        """ Message *submitAction* has been received """

        # Handle the message
        print(f"ID: {action_id}, Capability: {capability}, Parameters: {str(parameters)}")

        # Send back message *sendActionResult*
        self.send_action_result(action_id, "Everything went fine.")

    def on_config_changed(self):
        """ The configuration of the ActionHandler has changed """
        pass


with ActionWebSocket(api_handler=FixedTokenApiHandler('HIRO_TOKEN')) as ws:
    ws.run_forever()  # Use KeyboardInterrupt (Ctrl-C) to exit. 

Multithreading example using a thread executor:

import threading
import concurrent.futures

from hiro_graph_client.actionwebsocket import AbstractActionWebSocketHandler
from hiro_graph_client.clientlib import FixedTokenApiHandler, AbstractTokenApiHandler


class ActionWebSocket(AbstractActionWebSocketHandler):

    def __init__(self, api_handler: AbstractTokenApiHandler):
        """ Initialize properties """
        super().__init__(api_handler)
        self._executor = None

    def start(self) -> None:
        """ Initialize the executor """
        super().start()
        self._executor = concurrent.futures.ThreadPoolExecutor()

    def stop(self, timeout: int = None) -> None:
        """ Shut the executor down """
        if self._executor:
            self._executor.shutdown()
        self._executor = None
        super().stop(timeout)

    def handle_submit_action(self, action_id: str, capability: str, parameters: dict):
        """ Runs asynchronously in its own thread. """
        print(f"ID: {action_id}, Capability: {capability}, Parameters: {str(parameters)}")
        self.send_action_result(action_id, "Everything went fine.")

    def on_submit_action(self, action_id: str, capability: str, parameters: dict):
        """ Message *submitAction* has been received. Message is handled in thread executor. """
        if not self._executor:
            raise RuntimeError('ActionWebSocket has not been started.')
        self._executor.submit(ActionWebSocket.handle_submit_action, self, action_id, capability, parameters)

    def on_config_changed(self):
        """ The configuration of the ActionHandler has changed """
        pass


with ActionWebSocket(api_handler=FixedTokenApiHandler('HIRO_TOKEN')) as ws:
    ws.run_forever()  # Use KeyboardInterrupt (Ctrl-C) to exit. 

---

CHANGELOG

---

v5.3.0

• Decode any token via static decode_token_ext().

Changes for /api/auth/6.6:

• Ability to revoke a token.
• Recognize access_token and _TOKEN in token results.
• Remove obsolete method fresh() from class TokenInfo because each token refresh issues a new refresh_token by default now.

v5.2.5

• [bugfix] Calculation within clientlib.TokenInfo.expired() now returns False as expected when no expires_at is available for a token. Previous versions would raise an exception.

v5.2.4

• Merge headers in AbstractAPI instead of replacing them.

v5.2.3

• Remove accept_all_certs for good. Directly use the flag in SSLConfig.

v5.2.2

• Debugged python code documentation.
• Fixed copyright notices.

v5.2.1

• Update because of wrong documentation on https://pypi.org/project/hiro-graph-client/

v5.2.0

• Separate connection sessions from token handlers. Introducing class GraphConnectonHandler which can be shared between TokenApiHandlers and API-Classes.
• Updated documentation.

v5.1.0

• Use connection pooling via requests.Session.
  • Set pool_connections=1, pool_maxsize=10 by default (the latter can be changed).
  • Option pool_block to block any connections that would exceed the pool_maxsize.
• Removed obsolete accept_all_certs().
• Fixed pytest in Makefile target.
• Added information about requests.Session and the pool_maxsize to documentation.

v5.0.0

• Externalize batchclient.py as its own project at https://github.com/arago/hiro-batch-client-python.

v4.9.0

• Added batch handling of issues. handle_vertices_combined recognizes _issue_data now for issues directly linked to vertices.
• Updated documentation.

v4.8.0

• Refactoring of websocketlib.py:

  • Removed unnecessary extra thread.
  • Debugged shutdown of WebSockets via signals and threads.
  • Updated documentation:
    • On signal handling.
    • On clean shutdown of WebSockets.
    • Adjusted the example code for WebSockets.

• Allow - or _ as build number separator on test tags, i.e. t4.8.0_0 and t4.8.0-0 are handled identical now.

v4.7.5

• [budgfix] Handle self.submitStore and self.resultStore properly on shutdown in AbstractActionWebSocketHandler. They might not have been initialized at all when startup fails.

v4.7.4

• [bugfix] Catch 401 when using PasswordAuthTokenHandler and refresh_token. Raise TokenUnauthorizedError when this happens to trigger a retry.

v4.7.3

• Refactoring of HiroGraphBatch. Removed obsolete methods.

v4.7.2

• Fixed bad call of patch() methods in IAM API.

v4.7.1

• Updated some code documentation in HiroGraph that became obsolete.

• Updated graph queries in HiroGraph:

  • For vertex and gremlin queries: Skip unnecessary fields in query payload.
  • Add parameter count to vertex query.

• Renamed default _client_name to hiro-graph-client (name of the lib on PyPI).

v4.7.0

• Content- / Media-Type handling

  • Throw WrongContentTypeError when an unexpected (or missing) Content-Type is returned.
  • Check for Content-Type of error results for constructing HTTPError.

• Error handling

  • Separate error message extraction, so it can be overwritten per API.
  • Try to handle different message formats by guessing, where their error message might be.
  • Intercept special error messages for API ki.
  • Flag log_communication_on_error to enable logging of communication when an error is detected in the response.

• Add parameter max_tries to TokenHandlers.

v4.6.2

• Added documentation of new APIs

v4.6.1

• Updated CHANGELOG.md

v4.6.0

• Added the following APIS:
  • KI
  • AuthZ
  • Variables
• Adjust return value typing with API methods that return lists of dicts.

v4.5.2

BUGFIX

• Previous versions incorrectly translated True and False to "True" and "False" whereas "true" and "false" (lowercase) are needed in URL queries. Fixed.

v4.5.1

• Add decode_token() to decode the information embedded in a HIRO token.

v4.5.0

• GitHub repository got renamed from python-hiro-clients to hiro-client-python. No other technical changes.

v4.4.0

• Added IAM client
• Updated Graph client and Auth client
• put_binary is allowed to return simple strings now. (i.e. for avatar image updates).

v4.3.0

• Adding SSL configuration

v4.2.14

• Removed bug with reversed operator in websocketlib.
• Updated installation instructions in README.md.

v4.2.13

• You need to explicitly set query_params={'allscopes': 'true'} if you want to enable it for EventWebSockets. If this is left out of the query_params, it will be added as 'allscopes': 'false'.

v4.2.12

• Use typing to make sure, that query_params= for WebSockets is of type Dict[str, str].
• Set query_params={'allscopes': 'false'} as default for the EventWebSocket.

v4.2.11

• Debugged EventWebSocket handling.
• Abort connection when setting scope or filters failed.

v4.2.10

• Adding scopes to EventWebSockets.

v4.2.9

• Documentation of feature in v4.2.8

v4.2.8

• WebSockets have new option query_params to add arbitrary query parameters to the initial websocket request.

v4.2.7

Changes to AbstractAuthenticatedWebSocketHandler:

• Introducing run_forever() which will return after the reader thread has been joined. This can happen when another thread calls stop() (normal exit) or an internal error occurs, either directly when the connection is attempted, a token got invalid and could not be refreshed, or any other exception that has been thrown in the internal reader thread.

  This ensures, that the main thread does not continue when the websocket reader thread is not there.

• Enable parallel executions of send() across multiple threads.

• Make sure, that only one thread triggers a restart by a call to restart().

• Check for active websocket reader thread via is_active().

• Update examples for websockets in README.md.

Generic

• Update README.md to show usage of client_name.

v4.2.6

• Do not require package uuid - it is already supplied with python

v4.2.5

• Send valid close messages to backend.
• Introduced parameter client_name to give connections a name and also set header User-Agent more easily.

v4.2.4

• Updated CHANGELOG.md.

v4.2.3

• Hardening of clientlib. Removed some None-Value-Errors.

v4.2.2

• Introduce parameter remote_exit_codes to AbstractAuthenticatedWebSocketHandler.

v4.2.1

• Avoid blocking thread in _backoff() by not using sleep() but threading.Condition.wait().

v4.2.0

• Implement websocket protocols
  • event-ws
  • action-ws

v4.1.3

• Use yield from instead of return

v4.1.2

• Removed a bug with double yields on binary data

v4.1.1

• Only log request/responses when logging.DEBUG is enabled

v4.1.0

• Added timeseries handling to command handle_vertices_combined

v4.0.0

• AbstractTokenApiHandler

  • Better token handling.
  • Resolve graph api endpoints via calling /api/version.
    • Ability to customize headers. Headers are handled case-insensitively and are submitted to requests capitalized.
    • Ability to override internal endpoints.

• AbstractIOCarrier works with with statements now.

• Added BasicFileIOCarrier.

• Removed ApiConfig.

• Renamed several internal classes.

• Better error messages.

• HTTP secure protocol logging.

• Fixed timestamp creation for tokens.

• Joe's suggestions - thanks Joe!

v3.1.0

• Separation of APIs in client libraries. Currently, supported APIs are:
  • HiroGraph: https://core.arago.co/help/specs/?url=definitions/graph.yaml
  • HiroAuth: https://core.arago.co/help/specs/?url=definitions/auth.yaml
  • HiroApp: https://core.arago.co/help/specs/?url=definitions/app.yaml
• Use correct headers with binary transfers.
• Added gremlin and multi-id queries to HiroGraph.

v3.0.0

• Renamed classes to match documentation elsewhere (i.e. Graphit -> HiroGraph, GraphitBatch -> HiroGraphBatch).
• Catch token expired error when refresh_token has expired.
• Documentation with examples

v2.4.2

Added VERSION to package_data in setup.py

v2.4.1

Added documentation for PyPI

v2.4.0

Initial release after split from https://github.com/arago/hiro-clients