sequoia-client-sdk 4.4.0

Sequoia Python Client SDK

Python Sequoia Client SDK

A Python Client SDK for interacting with client services.

The central idea is that Client SDK allows python application code to communicate with the Piksel Palette RESTful RESTful services. Users can also search, filter and select their response collections.

Installation

pip install sequoia-client-sdk

Usage

Creating a SequoiaClient

To create the client it is necessary to provide the url for the service registry and named arguments specifying the credentials for the auth_type being used. If no auth_type is specified, then the default CLIENT_GRANT is used:

client = Client("https://registry-sandbox.sequoia.piksel.com/services/testmock",
                grant_client_id="clientId",
                grant_client_secret="clientSecret")

Authentication types

When creating the client, authentication type can be specified using the parameter auth_type:

client = Client("https://registry-sandbox.sequoia.piksel.com/services/testmock",
                auth_type=AuthType.CLIENT_GRANT,
                grant_client_id="clientId",
                grant_client_secret="clientSecret")

The Sequoia RESTful services have an OAuth token-based authorisation model, meaning that the Client SDK must first acquire a time-limited access token before making further requests. CLIENT_GRANT or BYO_TOKEN types should be used.

It is also possible to connect to the client via a proxy using two-way TLS authentication. In this case, MUTUAL auth_type should be used.

There are four authentication types:

CLIENT_GRANT type

This is the default type. With CLIENT_GRANT mode grant_client_id and grant_client_secret parameters are used to get an access token. The access token is refreshed automatically when expired. Optionally, byo_token parameter can be provided when instantiating the client, and will be used until it is expired. Then the access token is refreshed automatically.

client = Client("https://registry-sandbox.sequoia.piksel.com/services/testmock",
                auth_type=AuthType.CLIENT_GRANT,
                grant_client_id="clientId",
                grant_client_secret="clientSecret")

BYO_TOKEN type

With this method byo_token is required. That access token will be used to authenticate requests. The access token will be used along the client life and won’t be refreshed.

NO_AUTH type

Mode used when no authentication is required.

MUTUAL type

Mode used when mutual TLS authentication is required. Paths to local client certificate, client key and a server certificate files must be provided in the client_cert, client_key and server_cert arguments respectively.

client = Client("https://registry-sandbox.sequoia.piksel.com/services/testmock",
                auth_type=AuthType.MUTUAL,
                client_cert="/certs/client_cert.pem",
                client_key="/certs/client_key.pem",
                server_cert="/certs/server_cert.pem",
                ...

Content Type

By default the client sets “Content-Type” and “Accept’ header values of http requests to “application/vnd.piksel+json”. A different content type for these headers can be specified in the content_type parameter when creating a client.

client = Client("https://registry-sandbox.sequoia.piksel.com/services/testmock",
                auth_type=AuthType.MUTUAL,
                client_cert="/certs/client_cert.pem",
                client_key="/certs/client_key.pem",
                server_cert="/certs/server_cert.pem",
                content_type="application/json"
                )

Creating an endpoint

An endpoint defines the resource on which to perform the operations.

profile_endpoint = client.workflow.profiles
content_endpoint = client.metadata.contents

API methods

Read

Retrieves one resource given its reference and owner and returns the response retrieved.

endpoint.read(owner, ref)

Browse

Retrieves the list of resources that matches with the criteria and returns the response.

endpoint.browse(owner, criteria)

Store

Creates one or more resources and returns the response retrieved.

endpoint.store(owner, json)

Criteria API for Requesting Data

The SDK supports a fluent criteria API to abstract client code from the details of the Sequoia query syntax. This API allows to provide filters to retrieve the queried data and a way to request for related resources and its fields:

Criterion

The way to provide the filter to get specific data is by using the criterion this way.

endpoint.browse("testmock",
    Criteria().add_criterion(StringExpressionFactory.field("contentRef").equal_to("testmock:sampleContent"))
)

This alternative way is also supported:

endpoint.browse("testmock",
    Criteria().add(criterion=StringExpressionFactory.field("contentRef").equal_to("testmock:sampleContent"))
)

The following filtering criteria are supported:

equalTo

StringExpressionFactory.field("engine").equal_to("diesel")

Will generate the criteria expression equivalent to: field=diesel (withEngine=diesel)

Inclusion of related documents

The SDK support inclusion of related documents up to 1 level (direct relationships).

Both, direct and indirect relationships, are allowed. In each case resource’s reference are needed to perform the mapping.

Criteria().add_inclusion(Inclusion.resource('assets'))

This alternative way is also supported:

Criteria().add(inclusion=Inclusion.resource('assets'))

Selecting fields

The SDK allows to specify which fields will be present in the response, discarding the rest of them.

For now it can be used only for Inclusions

Criteria().add(inclusion=Inclusion.resource('assets').fields('name','ref'))

Paginating results

Iterator

Browse responses can be paginated. To paginate results, browse response has to be used as an iterator.

for response in endpoint.browse('testmock'):
    resources = response.resources

Not iterator

If browse function is not used as an iterator, only first page is retrieved. i.e:

response = endpoint.browse('testmock')
resources_in_page_1 = response.resources

With continue

Sequoia services allow to paginate using the parameter continue, which will return the link to get the following page in the meta of the response. The browse can be call repeatedly while there are pages to be read. Optionally, you can set the number of items per page.

for response in endpoint.browse('testmock', query_string='continue=true&perPage=2'):
    resources = response.resources

Paginating linked resources

Inclusion

When doing an inclusion, service returns a list of linked resources. Those resources can be paginated. Let’s assume a browse of contents is performed with assets resource as an inclusion. To perform pagination:

for linked_assets in endpoint.browse('testmock').linked('assets'):
    for linked_asset in linked_assets:
        asset_name = linked_asset['name']

If linked response is not used as an iterator, only first page of linked resources is retrieved:

linked_assets =  endpoint.browse('testmock').linked('assets')
for linked_asset in linked_assets.resources:
    asset_name = linked_asset['name']

Retrying requests

When a request is returning a retrievable status code, a retry strategy can be configured with backoff_strategy. By default backoff_strategy is

{'wait_gen': backoff.constant, 'interval': 0, 'max_tries': 10}

We can set a different backoff strategy.

client = Client("https://registry-sandbox.sequoia.piksel.com/services/testmock",
                grant_client_id="clientId",
                grant_client_secret="clientSecret",
                backoff_strategy={'wait_gen': backoff.expo, 'max_tries': 5, 'max_time': 300}
                )

Here an exponential strategy will be used, set in the constructor.

For a more fine grain customization the retry policy can also be set for specific requests by passing the parameter at method level. This is valid for read, browse, get and request methods. The backoff strategy configured in the constructor will be used for all the other queries in which another backoff strategy is not given. Here you can see an example:

client = Client("https://registry-sandbox.sequoia.piksel.com/services/testmock",
                grant_client_id="clientId",
                grant_client_secret="clientSecret",
                backoff_strategy={'wait_gen': backoff.expo, 'max_tries': 5, 'max_time': 300}
                )
assets_endpoint = client.metadata.contents
response1 = assets_endpoint.browse(
    self.owner,
    criteria.Criteria()
        .add_criterion(criteria.StringExpressionFactory.field('ref').equal_to('test:c0007'))
        .add_inclusion(criteria.Inclusion.resource('assets')),
    backoff_strategy={'wait_gen': backoff.constant, 'interval': 0, 'max_tries': 20}
)
response2 = assets_endpoint.browse(
    self.owner,
    criteria.Criteria()
        .add_criterion(criteria.StringExpressionFactory.field('ref').equal_to('test:c02'))
        .add_inclusion(criteria.Inclusion.resource('categories'))
)

For that example, the first browse will retry the query up to 20 times while the second browse will do up to 5 or until the the max time is reached.

Retry when status code

You can also provide a number of HTTP status codes to perform the retry of the query, this is, when the query you are performing returns one of the status codes you’ve specified, the query is automatically retried. The key word you have to use for this is retry_http_status_codes within the backoff_strategy dictionary.

For instance:

client = Client("https://registry-sandbox.sequoia.piksel.com/services/testmock",
                grant_client_id="clientId",
                grant_client_secret="clientSecret",
                backoff_strategy={'wait_gen': backoff.expo, 'max_tries': 5, 'max_time': 300,
                                  'retry_http_status_codes': [404, 409]}
                )

When max_time is set to None or not passed, a default value is automatically set to avoid possible undesired behaviour such as infinite loops. The default value is set to 120 seconds.

For more info about backoff strategies https://github.com/litl/backoff

Retry when empty result

You can also set up the retries policy for the case in that the resources you are querying for are missing in the response. This is useful when you are quite sure the data you are querying will eventually exist in the service even though it doesn’t exist yet.

The way to configure this is by using the parameter retry_when_empty_result in the method you use to query the service, this is valid for read, browse, get and request methods.

The parameter retry_when_empty_result accepts either a boolean value to specify all resources are expected, either they are the main resources or the linked ones, or a dictionary in which you can explicitly specify the type of resources you are expecting to have in the response. In cases these resources are missing the query will be retried.

Let’s see an example:

client = Client("https://registry-sandbox.sequoia.piksel.com/services/testmock",
                grant_client_id="clientId",
                grant_client_secret="clientSecret",
                backoff_strategy={
                    'retry_when_empty_result': True
                    }
                )
assets_endpoint = client.metadata.contents
response = assets_endpoint.browse(
    self.owner,
    criteria.Criteria()
        .add_criterion(criteria.StringExpressionFactory.field('ref').equal_to('test:c0007'))
        .add_inclusion(criteria.Inclusion.resource('categories'))
        .add_inclusion(criteria.Inclusion.resource('assets')),
    retry_when_empty_result=True
)

This way you are asking to retry the query when the response has no data for the main resource and for the inclusions you are querying for.

This is, if your query look like https://metadata-sandbox.sequoia.piksel.com/data/contents?include=assets,categories&owner=test&withRef=test:c0007 the query will be retried until the content test:c0007 is returned and it has at least one asset and one category in the response too. Or the retries reach the limit.

A finer configuration using a dictionary is allowed so you can specify which resources have to be checked this way:

client = Client("https://registry-sandbox.sequoia.piksel.com/services/testmock",
                grant_client_id="clientId",
                grant_client_secret="clientSecret"
                )
assets_endpoint = client.metadata.contents
response = assets_endpoint.browse(
    self.owner,
    criteria.Criteria()
        .add_criterion(criteria.StringExpressionFactory.field('ref').equal_to('test:c0007'))
        .add_inclusion(criteria.Inclusion.resource('categories'))
        .add_inclusion(criteria.Inclusion.resource('assets')),
    retry_when_empty_result={
                        'contents': True,
                        'assets': False,
                        'categories': True
                    }
)

In that example both resources contents and categories are checked to be returned, but not assets.

In case the limit of retries is reached and that condition is not fulfilled the latest response is returned. Bear in mind that the response can very likely have a status code of 200 and a body with data.

Remember, as specified above a max_time is automatically set even though it is not given.

Correlation ID

Every request to Sequoia RESTful services is added with a unique correlation id in the headers.

-- request headers --
    ...
    x-correlation-id: f0fca55f3da85..6336cb20fda36
    ...

The SDK allows to set a correlation id at the client to be added to all the subsequent requests.

client = Client("https://registry-sandbox.sequoia.piksel.com/services/testmock",
                ...
                correlation_id="custom_correlation_id_1234",
                ...
                )

endpoint.browse(owner, criteria)

 -- request headers --
    ...
    x-correlation-id: custom_correlation_id_1234
    ...

It also allows to provide both an user and an application ids so each operation request will be set with an unique generated correlation id having these values as prefix. This correlation id will be shared by all related requests derived by that operation: browse, store, etc (e.g. the subsequents paging requests in a browse operation).

Both parameters user_id and application_id has to be provided, providing just one you won’t have a prefix in the correlation id.

client = Client("https://registry-sandbox.sequoia.piksel.com/services/testmock",
                ...
                user_id="user123",
                application_id="app101",
                ...
                )

endpoint.browse(owner, criteria)

 -- request headers --
    ...
    x-correlation-id: user123/app101/cbd05bd7-3099-4dcb-aeff-806ccec3292a
    ...

endpoint.browse(owner, criteria)

 -- request headers --
    ...
    x-correlation-id: user123/app101/9becd6c7-8ef0-44c4-a240-6c02c583957f
    ...

The parameter correlation_id has precedence over user_id and application_id.

Development

It has been tested for Python 3.5 and 3.6

You can use the included command line tool make to work with this project

Preparing environment

Create new virtualenv

It’s encouraging to create a new virtual environment and install all the dependencies in it. You can use these commands:

mkdir -p ~/.virtualenvs
virtualenv -p python3.6 ~/.virtualenvs/sequoia-python-client-sdk
workon sequoia-python-client-sdk
pip install -r requirements.txt
pip install -r requirements_test.txt

Testing

There are two different ways of running the tests.

Run tests on the current environment

Using pytest option will run all the unit tests over your environment.

make test

Run tests on every compatible python version

While using the option test will set up a virtual environment for the supported version of Python, i.e. 3.5 and 3.6 and will run all the tests on each of them.

make test-all

If you are using pyenv and found issues running this command because tox isn’t able to create the virtualenvs, just add the python versions you have installed to the file .python-version like this:

echo "3.6.9" >> .python-version
echo "3.7.7" >> .python-version
echo "3.8.3" >> .python-version

Lint

To make sure the code fulfills the format run

make lint

History

1.0.0

• First release.

1.1.0 (2017-10-25)

• Upgrade to Python 3.6

1.2.0 (2019-03-06)

• Libraries urllib3 and requests upgraded to solve security issues:

  • CVE-2018-20060

  • CVE-2018-18074

1.2.1 (2019-03-26)

• Load yaml config file for testing in a safer way as specified in PyYAML.

2.0.0 (2019-06-06)

• Removing python 2.7 compatibility.

• Adding backoff to http requests. Configurable backoff from client creation.

• Libraries urllib3 and requests upgraded to solve security issues.

2.1.0 (2019-09-30)

• Modifying setup.cfg to allow different version formats (i.e development versions).

• Paging with continue parameter.

• When token is expired, it is updated automatically with CLIENT_GRANT auth type.

2.1.1 (2019-10-02)

• Token fetching not restarting backoff. Retries continuing its count instead of restarting it when there is a invalid token.

2.2.0 (2020-08-13)

• Allowing to provide correlation_id value when the client is created.

• Caching tokens by grant_client_id and token_url to avoid calling identity in case credentials are cached.

• PageBrowser keeping a response cache to avoid duplicated requests.

• Bug fixed when paging main content. Query params should to be added to next url.

• New AuthType.MUTUAL.

3.0.0 (2020-10-06)

• Removing transaction_id value when the client is created.

• Allowing to provide user_id and application_id values as correlation id prefix.

4.0.0 (2020-10-21)

• Python 3.5 support removed.

• Python 3.7 supported.

• Python 3.8 supported.

• Pagination with continue parameter over linked resources supported.

• Requirements upgraded.

4.0.1 (2020-12-22)

• When token is expired, it is updated automatically with CLIENT_GRANT auth type,

  the 401 response wasn’t managed to do so, only the exception was. Now the 401 response is treated like that.

4.0.2 (2021-03-04)

• Two new methods added to Criterion object so the fluent API is easier to use: add_inclusion and add_criterion.

4.1.0 (2021-04-13)

• New keyword retry_http_status_codes for the backoff_strategy to retry specific http status codes.

• Prospector version upgraded to 1.3.1 so it works with python versions 3.9.4, 3.8.9, 3.7.10, 3.6.13.

• Lint issues solved (OAuth2SessionTokenManagementWrapper request method signature).

• GitHub Actions configured to run lint and unit tests.

4.2.0 (2021-04-13)

• Python 3.9 supported.

• Requirements upgraded.

• Drop the use of some libraries: jsonpickle, twine.

• Tox is installed in the Makefile when used.

4.3.0 (2021-04-27)

• New parameter retry_when_empty_result for the read, browse, get and request methods to retry the query when resources are missing in the response.

4.4.0 (2021-05-03)

• The backoff_strategy can be specified in the read, browse, get and request methods so it can be different from the one passed in the constructor.

• Set up the logger name to allow a better logging configuration