datahub-api-connector 1.5

This package simplifies the calls to Data Hub's API.

No magic. You need to follow the Data Hub Swagger Documentation for correct formatting of your requests

Be aware that this is a new package, certainly missing many features. Feel free to contribute.

You first need to create an instance of the ApiConnector class with following parameters:

ATTENTION: this package can only be used from Data Hub version 7.0 (July 1, 2025) onward, as it uses the authentication to the the new technology (Keycloak). For previous versions, the opinum-api-connector package must be used (https://github.com/opinum/opinum-api-connector) instead.

VERSION 1.5 Account scoping fix.

• A token renewed with its refresh token now sends the account_id again. It was only sent on the first token request, so a token renewed in the middle of a run could come back scoped to another tenant: the calls then read and wrote the wrong account's data with a token that looked perfectly valid.
• As a safety net, a renewed token that claims another account than account_id is discarded and a brand new token is requested.
• New token_claims and token_account_id properties, to check which account the current token is really scoped to.

VERSION 1.4 Retry release.

• Server-side failures are now retried instead of being raised on the first attempt. Until now only connection errors and read timeouts were, while raise_for_status() turns a 500 into an HTTPError that no clause caught, so a single transient failure aborted the call. See the new retry_on_status and retry_unsafe_methods parameters.
• The wait between two attempts now doubles each time, capped at 60 seconds, with a small jitter so that the threads sharing an instance stop retrying in lockstep. A Retry-After response header is honoured.
• Retry and failure logs now name the method, the URL and the beginning of the response body, where the API states the actual cause. It was discarded before.
• The module no longer calls logging.basicConfig() nor changes the root logger level, which reset the logging an application had already configured for itself.

VERSION 1.3 Sturdiness and correctness release.

• The token lifetime now follows the real expiry returned by the authentication server (with a 120 seconds safety margin) instead of a fixed 3 minutes limit, and is renewed with its refresh token when possible.
• All calls now go through a single persistent connection pool (see the new pool_size parameter) instead of opening a new connection each time, which exhausted the available sockets on long multi-threaded runs.
• Fixed a retried call sending a doubly encoded body (rejected with a 400), and a token renewed between two attempts not being used. Read timeouts are now retried as well, and the last failed attempt no longer waits before raising.
• Fixed operation_timeout_sec being ignored on push_data when operation_id was given too, and file names containing special characters on send_file_to_storage.
• multi_thread_request_on_path now really uses threads (it raised a pickling error before) and splits the calls on max_parameter_entities as documented, instead of on max_futures. Expect different call sizes than in 1.2.
• ApiConnector can be closed, and used as a context manager, to release its connections.

VERSION 1.2 Fixed operation_id on PUSH calls, and added operation_timeout_sec. Also added the possibility to request a count of items on some calls with the IncludeItemsCount optional parameter.

VERSION 1.1 Improved sturdiness. Added thread lock on token requests, and a default timeout of 10 seconds on all requests (incliding token)

environment

a dictionary of environment variables

if None, ApiConnector uses your environment variables (os.environ)

Mandatory environment variables are:

• DATAHUB_USERNAME: the Data Hub user.
  TAKE CARE: if this user has access to multiple tenants and if you do not specify a tenant id, ApiConnector will use the last tenant used.
• DATAHUB_PASSWORD: the password for the user
• DATAHUB_CLIENT_ID: the client id for accessing the API
• DATAHUB_CLIENT_SECRET the corresponding secret

Optional environment variables are:

• DATAHUB_API_URL: another API URL than the Europe SaaS one (https://api.opinum.com)
• DATAHUB_AUTH_URL: another authentication URL than the Europe SaaS one (https://auth.opinum.com)
• DATAHUB_PUSH_URL: another push URL than the Europe SaaS one (https://push.opinum.com)
• DATAHUB_SCOPE: the scope of you session (default: "datahub-api")
  if you want to push data, the scope should be "datahub-api push-data"

account_id

one of the tenant ids available for the Data Hub user (default: None)

it is sent on every token request, including renewals, so an instance stays on its tenant for its whole lifetime. token_account_id tells which account the current token claims to be scoped to (None when the token does not say).

retries_when_connection_failure

number of extra attempts when no 200 or 204 return code (default: 0, maximum: 5)

the wait between two attempts doubles each time (seconds_between_retries, then twice that, and so on, capped at 60 seconds) and carries a small random jitter, so that several threads sharing the instance do not all retry at the very same moment. A Retry-After response header takes precedence over that wait.

request_timeout

timeout value in seconds on all requests (including fetch token) (default: 10)

retry_on_status

HTTP statuses retried instead of being raised straight away (default: 429, 500, 502, 503, 504)

501 is absent on purpose, and so are the 4xx other than 429: they would fail identically on a second attempt. Pass None to disable status retries.

retry_unsafe_methods

also apply retry_on_status to post, put, patch and delete (default: False)

by default only get is retried on those statuses. Replaying a call that changes something, when the server may have applied it before failing, would duplicate the change. Set this to True when your write calls are safe to replay, or when the call only reads despite being a post (the query-by-body POST /data is one).

log_level

sets log level for the module (default: INFO)

only this package's logger is touched; the root logger of the calling application is left as that application configured it

pool_size

size of the connection pool shared by all calls of the instance (default: 32)

it must be at least as large as the number of threads sharing the instance (the workers parameter of multi_thread_request_on_path, default: 16)

Once you have your ApiConnector instance, you may use the class methods

• get
• post
• patch
• put
• delete
• send_file_to_storage

All keyword arguments will be converted to path parameters in the API call with one important exception, the data keyword referring to the body of your call.

There are two other class methods for data pushing because we have another API for this

• push_data
• push_dataframe_data

There is a little bit of magic with the method multi_thread_request_on_path that splits a list of parameters Allowing to make parallel calls.