datahub-api-connector 1.7

This package simplifies the calls to Data Hub's API.

No magic. You need to follow the Data Hub Swagger Documentation for correct formatting of your requests

Be aware that this is a new package, certainly missing many features. Feel free to contribute.

You first need to create an instance of the ApiConnector class with following parameters:

ATTENTION: this package can only be used from Data Hub version 7.0 (July 1, 2025) onward, as it uses the authentication to the the new technology (Keycloak). For previous versions, the opinum-api-connector package must be used (https://github.com/opinum/opinum-api-connector) instead.

VERSION 1.7 Connection lifetime fix, following up on the pool introduced in 1.3.

• The authentication session of every token request is now closed. One OAuth2Session is built per token request, and none of them were closed, so each held a connection to the authentication server until the garbage collector got to it. On a long run over many accounts those piled up next to the API connections.
• Documented the lifetime of an instance, which was the missing piece: it owns a connection pool, so it is meant to be built once, reused, and closed. Building one per unit of work leaks its pool until the object is collected, and caching one per account without ever closing them exhausts the sockets the operating system can hand out (WSAENOBUFS / ERR_NO_BUFFER_SPACE on Windows) on a long run. To work through many accounts, reuse one instance per thread and assign account_id.

VERSION 1.6 Retry hardening release. The retries of 1.4 were reachable only when retries_when_connection_failure was set, and several failures escaped them entirely.

• Retries are now on by default (3 extra attempts). retry_on_status was configured out of the box but unreachable: the default attempt budget was a single attempt, so a transient 500 on a get was raised without ever being retried. Pass retries_when_connection_failure=0 for the previous behaviour.
• A failure of the authentication server no longer aborts the call before a single attempt. The authentication library parses the token response without checking its status first, so a 5xx or a gateway's error page reached the caller as an oauthlib error that no clause caught. Those are now retried; a rejected credential or scope is still raised at once.
• A transient authentication failure no longer makes the constructor raise.
• A token the server refuses with a 401, although it still looked valid on this side (revoked, clock skew, account switched server-side), is now renewed once and the call replayed. It used to fail for good.
• Assigning a new account_id on an existing instance now takes effect immediately. The token of the previous tenant stayed in use until it expired, so up to an hour of calls read and wrote the wrong account. A token claiming another account than account_id is never sent.
• An authentication outage no longer becomes a stampede: the threads sharing an instance used to ask for a refresh and a full re-issue each, on every one of their attempts (16 threads produced 128 token requests, now 2). A thread retrying after its own backoff is never suppressed.
• 408 and 425 joined the retried statuses, and a response cut short or with a corrupt encoding (ChunkedEncodingError, ContentDecodingError) is retried instead of raised: neither derives from requests' HTTPError, so both fell through every clause.
• multi_thread_request_on_path no longer discards the calls of a group that already succeeded when one of them fails. They are yielded first, then the failure is raised. See the new raise_on_error parameter to keep going instead.

VERSION 1.5 Account scoping fix.

• A token renewed with its refresh token now sends the account_id again. It was only sent on the first token request, so a token renewed in the middle of a run could come back scoped to another tenant: the calls then read and wrote the wrong account's data with a token that looked perfectly valid.
• As a safety net, a renewed token that claims another account than account_id is discarded and a brand new token is requested.
• New token_claims and token_account_id properties, to check which account the current token is really scoped to.

VERSION 1.4 Retry release.

• Server-side failures are now retried instead of being raised on the first attempt. Until now only connection errors and read timeouts were, while raise_for_status() turns a 500 into an HTTPError that no clause caught, so a single transient failure aborted the call. See the new retry_on_status and retry_unsafe_methods parameters.
• The wait between two attempts now doubles each time, capped at 60 seconds, with a small jitter so that the threads sharing an instance stop retrying in lockstep. A Retry-After response header is honoured.
• Retry and failure logs now name the method, the URL and the beginning of the response body, where the API states the actual cause. It was discarded before.
• The module no longer calls logging.basicConfig() nor changes the root logger level, which reset the logging an application had already configured for itself.

VERSION 1.3 Sturdiness and correctness release.

• The token lifetime now follows the real expiry returned by the authentication server (with a 120 seconds safety margin) instead of a fixed 3 minutes limit, and is renewed with its refresh token when possible.
• All calls now go through a single persistent connection pool (see the new pool_size parameter) instead of opening a new connection each time, which exhausted the available sockets on long multi-threaded runs.
• Fixed a retried call sending a doubly encoded body (rejected with a 400), and a token renewed between two attempts not being used. Read timeouts are now retried as well, and the last failed attempt no longer waits before raising.
• Fixed operation_timeout_sec being ignored on push_data when operation_id was given too, and file names containing special characters on send_file_to_storage.
• multi_thread_request_on_path now really uses threads (it raised a pickling error before) and splits the calls on max_parameter_entities as documented, instead of on max_futures. Expect different call sizes than in 1.2.
• ApiConnector can be closed, and used as a context manager, to release its connections.

VERSION 1.2 Fixed operation_id on PUSH calls, and added operation_timeout_sec. Also added the possibility to request a count of items on some calls with the IncludeItemsCount optional parameter.

VERSION 1.1 Improved sturdiness. Added thread lock on token requests, and a default timeout of 10 seconds on all requests (incliding token)

environment

a dictionary of environment variables

if None, ApiConnector uses your environment variables (os.environ)

Mandatory environment variables are:

• DATAHUB_USERNAME: the Data Hub user.
  TAKE CARE: if this user has access to multiple tenants and if you do not specify a tenant id, ApiConnector will use the last tenant used.
• DATAHUB_PASSWORD: the password for the user
• DATAHUB_CLIENT_ID: the client id for accessing the API
• DATAHUB_CLIENT_SECRET the corresponding secret

Optional environment variables are:

• DATAHUB_API_URL: another API URL than the Europe SaaS one (https://api.opinum.com)
• DATAHUB_AUTH_URL: another authentication URL than the Europe SaaS one (https://auth.opinum.com)
• DATAHUB_PUSH_URL: another push URL than the Europe SaaS one (https://push.opinum.com)
• DATAHUB_SCOPE: the scope of you session (default: "datahub-api")
  if you want to push data, the scope should be "datahub-api push-data"

account_id

one of the tenant ids available for the Data Hub user (default: None)

it is sent on every token request, including renewals, so an instance stays on its tenant for its whole lifetime. token_account_id tells which account the current token claims to be scoped to (None when the token does not say).

it can also be assigned on an existing instance to switch tenant. The current token belongs to the previous one, so it is dropped and the next call gets a token for the new account. A token claiming an account other than this one is never sent.

retries_when_connection_failure

number of extra attempts when no 200 or 204 return code (default: 3, maximum: 5)

pass 0 for a single attempt, which was the default until 1.6. Note that a connection failure or a timeout is retried whatever the method is, unlike the statuses of retry_on_status: that is what this parameter has always meant, and push_data relies on it.

the wait between two attempts doubles each time (seconds_between_retries, then twice that, and so on, capped at 60 seconds) and carries a small random jitter, so that several threads sharing the instance do not all retry at the very same moment. A Retry-After response header takes precedence over that wait.

a failure of the authentication server counts as a transient failure too, since getting a token has no effect on the resource. A rejected credential or scope is raised straight away. A call refused with a 401 is a case apart: the token is renewed and the call replayed once, without using up one of these attempts.

request_timeout

timeout value in seconds on all requests (including fetch token) (default: 10)

retry_on_status

HTTP statuses retried instead of being raised straight away (default: 408, 425, 429, 500, 502, 503, 504)

501 is absent on purpose, and so are the 4xx other than those listed: they would fail identically on a second attempt. Pass None to disable status retries.

401 is not part of this and cannot be: it is handled apart, by renewing the token and replaying the call once.

retry_unsafe_methods

also apply retry_on_status to post, put, patch and delete (default: False)

by default only get is retried on those statuses. Replaying a call that changes something, when the server may have applied it before failing, would duplicate the change. Set this to True when your write calls are safe to replay, or when the call only reads despite being a post (the query-by-body POST /data is one).

log_level

sets log level for the module (default: INFO)

only this package's logger is touched; the root logger of the calling application is left as that application configured it

pool_size

size of the connection pool shared by all calls of the instance (default: 32)

it must be at least as large as the number of threads sharing the instance (the workers parameter of multi_thread_request_on_path, default: 16)

because it owns that pool, an instance is meant to be long-lived: build a few, reuse them, and close them (ApiConnector is a context manager). One instance per unit of work leaks its pool until the object is collected, and a cache of them never closed runs the machine out of sockets. To work through many accounts, reuse one instance per thread and assign account_id rather than building one per account

Once you have your ApiConnector instance, you may use the class methods

• get
• post
• patch
• put
• delete
• send_file_to_storage

All keyword arguments will be converted to path parameters in the API call with one important exception, the data keyword referring to the body of your call.

There are two other class methods for data pushing because we have another API for this

• push_data
• push_dataframe_data

There is a little bit of magic with the method multi_thread_request_on_path that splits a list of parameters Allowing to make parallel calls.

The calls are run in groups (max_futures). When one of them fails, the ones of its group that already succeeded are yielded first, then the failure is raised, so the work the API has already done is not thrown away. Pass raise_on_error=False to get everything that worked and have the failures logged only.