TetraScience Python Connector SDK

Version

v2.4.0

Table of Contents

• Summary
• Usage
  • Connector Class
    • Creating and running a connector
    • Starting the connector and running
      • Configuring TdpApi
  • Proxy support
  • Initialization of TdpApi
    • Connector example
  • SQL Connector
  • Commands
    • Lifecycle methods and hooks
    • Additional Connector commands
    • Custom commands
  • Polling
• Logging
  • Logger usage
  • Setting log levels
  • CloudWatch logging and standalone connector support
• Product telemetry
  • Emitting events
  • Counting
  • Configuration
  • What the SDK never sends
• OpenAPI code generation
  • Using the generated API client
    • Example usage
    • Retrieving connector data with filtering
      • Using TdpApi (async)
      • Using TdpApiSync (synchronous)
      • Using Connector class methods
• Local testing with standalone connectors
  • Prerequisites
  • Setup process
    • Step 1: build your local connector image
    • Step 2: use the standalone installer
    • Step 3: environment configuration
    • Step 4: running your local connector
    • Alternative: running containerless locally
  • Important notes
• Running tests
• Changelog
  • v2.3.0
  • v2.2.0
  • v2.1.0
  • v2.0.0
  • v1.0.1
  • v1.0.0
  • v0.9.0
  • v0.8.0
  • v0.7.0
  • v0.6.0
  • v0.5.0
  • v0.4.0
  • v0.3.0
  • v0.2.0
  • v0.1.0

Summary

The TetraScience Python Connectors SDK provides utilities and APIs for building TetraScience pluggable connectors in Python. Connectors are containerized applications used for transferring data between the Tetra Data Platform and other systems. Some examples of existing connectors:

• The S3 connector receives file events from an S3 bucket via an SQS queue and pulls the corresponding objects into Tetra
• The Kepware KEPServerEX connector pulls tags from KEPServerEX over MQTT and writes corresponding JSON files to Tetra
• The LabX connector can connect to multiple LabX instances and retrieve completed tasks. The LabX connector was written using this Python SDK

Usage

Connector Class

The Connector class is the core component of the SDK. It provides methods and hooks to manage the lifecycle of a connector, handle commands, perform periodic tasks, and interact with Tetra.

Creating and running a connector

To create a Connector instance, you need to provide a TdpApi instance and optional ConnectorOptions. The TdpApi instance is the class that interacts with Tetra

from ts_sdk_connectors_python.connector import Connector, ConnectorOptions
from ts_sdk_connectors_python.tdp_api import TdpApi

tdp_api = TdpApi()
connector = Connector(tdp_api=tdp_api, options=ConnectorOptions())

Starting the connector and running

async def main():
    tdp_api = TdpApi()
    connector = Connector(tdp_api=tdp_api, options=ConnectorOptions())
    await connector.start()
    while True:
        await asyncio.sleep(1)

asyncio.run(main())

Configuring TdpApi

Required configuration values for TdpApi can be provided either as instance args or pulled from environment variables. If any arguments are not provided, they are pulled from the environment variables.

# manually provide configuration values
tdp_api = TdpApi(
    aws_region="us-east-1",
    org_slug="tetrascience-yourorg",
    hub_id="your-hub-id",
    connector_id="your-connector-id",
    datalake_bucket="your-datalake-bucket",
    stream_bucket="your-stream-bucket",
    tdp_certificate_key="your-tdp-certificate-key",
    jwt_token_parameter="your-jwt-token-parameter",
    tdp_endpoint="https://api.tetrascience.com",
    outbound_command_queue="your-outbound-command-queue",
    kms_key_id="your-kms-key-id",
    artifact_type="connector",
    connector_token="your-connector-token",
    local_certificate_pem_location="path/to/your/certificate.pem"
)

# Automatically pull all args from environment variables
tdp_api = TdpApi()

# Some arguments provided and remaining args pulls from environment variables
tdp_api = TdpApi(datalake_bucket="your-datalake-bucket")

The following environment variables are used by the TdpApiConfig class to configure the TetraScience Data Platform API client. Note that not all of them are necessarily relevant:

Variable Name	Description
AWS_REGION	The AWS region to use.
ORG_SLUG	The organization slug for the TetraScience Data Platform.
HUB_ID	The hub ID for the connector.
CONNECTOR_ID	The unique identifier for the connector.
DATALAKE_BUCKET	The name of the datalake bucket.
STREAM_BUCKET	The name of the stream bucket.
TDP_CERTIFICATE_KEY	The key for the Tetra certificate.
JWT_TOKEN_PARAMETER	Name of the SSM parameter that contains the JWT token. Used Used by non-standalone connectors
TDP_ENDPOINT	The base URL for the TetraScience Data Platform API.
OUTBOUND_COMMAND_QUEUE	The queue name for outbound commands.
KMS_KEY_ID	The KMS key ID.
ARTIFACT_TYPE	The type of artifact (e.g., connector, data-app).
CONNECTOR_TOKEN	The JWT authentication token for the connector. Used for standalone connectors to request initial AWS credentials
LOCAL_CERTIFICATE_PEM_LOCATION	The local certificate PEM file location.

Proxy support

In addition to the above environment variables, the connector uses proxy settings determined from the environment variables HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. For connectors on a Hub, the connector sets these environment variables based on the Hub's proxy settings. For standalone connectors, the standalone installer will set lowercase versions of these variables. The connector checks the environment, and in the case where lowercase versions exist but uppercase ones don't, it copies the lowercase values over to uppercase.

IAM proxy (network-restricted environments)

Set IAM_PROXY to route the SDK's AWS calls (SSM, S3) through a forward/IAM proxy in environments that block direct outbound access. Two caveats apply to a fully locked-down setup:

• Set HTTPS_PROXY as well. IAM_PROXY covers the AWS SDK (aioboto3) path only. The initial request that fetches AWS credentials from the Tetra platform uses an HTTP client that honors HTTPS_PROXY, not IAM_PROXY — so with only IAM_PROXY set, that first call goes direct and will be blocked. Set both.
• TDP_CERTIFICATE_BUNDLE applies to direct connections only. When traffic is tunneled through an HTTPS proxy, the upstream certificate is not verified against this bundle, so it has no effect on the proxied path. It is honored only for direct (non-proxied) connections.

Initialization of TdpApi

TdpApi must initialize the AWS and HTTP clients before it can communicate with the connector's AWS services and connector endpoints in Tetra.

tdp_api = TdpApi()
await tdp_api.init_client(proxy_url = "127.0.0.1:3128")  # your proxy URL here, if needed
files = tdp_api.get_connector_files(...)
...

Connector example

Here is an example of a custom connector that prints "Hello World" on a scheduled interval:

from typing import Optional
from ts_sdk_connectors_python.connector import Connector, ConnectorOptions
from ts_sdk_connectors_python.custom_commands import register_command
from ts_sdk_connectors_python.tdp_api import TdpApi
from ts_sdk_connectors_python.utils import Poll

class BasicScheduledConnector(Connector):
    """Prints hello world on a scheduled interval"""

    def __init__(
        self,
        tdp_api: TdpApi,
        schedule_interval: int,
        options: Optional[ConnectorOptions] = None,
    ):
        super().__init__(tdp_api=tdp_api, options=options)
        self.poll: Optional[Poll] = None
        self.schedule_interval = schedule_interval

    async def on_start(self):
        await super().on_start()
        self._start_polling()

    async def on_stop(self):
        await super().on_stop()
        self._stop_polling()

    @register_command("TetraScience.Connector.PollingExample.SetScheduleInterval")
    async def set_schedule_interval(self, schedule_interval: str):
        self.schedule_interval = float(schedule_interval)
        self._stop_polling()
        self._start_polling()

    def _start_polling(self):
        if not self.poll:
            self.poll = Poll(self.execute_on_schedule, self.schedule_interval)
            self.poll.start()

    def _stop_polling(self):
        if self.poll:
            self.poll.stop()
            self.poll = None

    async def execute_on_schedule(self):
        print("HELLO WORLD")


# Usage
import asyncio

async def main():
    tdp_api = TdpApi()
    await api.init_client()
    connector = BasicScheduledConnector(tdp_api=tdp_api, schedule_interval=5)
    await connector.start()
    await asyncio.sleep(10)
    await connector.shutdown()

asyncio.run(main())

This example demonstrates how to create a custom connector that prints "Hello World" every 5 seconds and allows the schedule interval to be updated via a custom command. For more information on registering commands, see below.

SQL Connector

The SDK provides a SqlConnector abstract base class that simplifies building SQL-based connectors using an inheritance pattern. This allows you to easily connect to any SQL database (PostgreSQL, MySQL, SQL Server, Oracle, etc.) and incrementally retrieve data using watermarks.

For detailed documentation on the SQL Connector, including:

• Quick Start guide
• Multi-query support with examples
• Configuration parameters and timeout configuration
• Watermark strategies with examples
• Best practices (read-only grants, query performance, indexes)
• Error codes for manifest.json
• Troubleshooting

See the SQL Connector Documentation.

For technical design details (synchronous SQLAlchemy rationale, connection pooling, runtime configuration updates), see SQL Connector Design.

Commands

Tetra communicates with connectors via the command service. The data acquisition service in Tetra uses a set of commands we refer to as "lifecycle commands". The Connector class implements a command listener and has several methods that are invoked when lifecycle commands come in.

Lifecycle methods and hooks

Starting and Initializing methods

• start: Starts the connector and its main activities.
  • Triggers:
    • None. Almost all connectors will call this from main.py, the default entrypoint of the container
  • Default implementation:
    • calls on_initializing hook
    • loads connector details (does not call on_connector_updated)
    • starts metrics collection, heartbeat, and command listener tasks
    • if the connector's operating status is RUNNING, calls on_start hook
    • calls on_initialized hook
• on_initializing: A developer-defined hook that is called at the beginning of the default implementation of Connector.start
  • Triggers:
    • None. In default implementation, called once by Connector.start
  • Default implementation:
    • None
• on_initialized: A developer-defined hook that is called at the end of the default implementation of Connector.start
  • Triggers:
    • None. In default implementation, called once by Connector.start
  • Default implementation:
    • None
• on_start: A developer-defined hook that runs when the connector's operating status is set to RUNNING
  • Triggers (any of the following):
    • A command with action TetraScience.Connector.Start is received
      • this corresponds to setting the connector operating status toRUNNING
    • During Connector.start if the connector's operating status is RUNNING
      • this typically happens when a disabled connector is "enabled as RUNNING"
  • Default implementation:
    • reloads connector config, which subsequently calls on_connector_updated

Running methods

• on_connector_updated: A developer-defined hook that gets called when the connector details are updated. Because this corresponds to config changes and is also triggered indirectly by on_start, it is the most common place to initialize resources for the connector to work with third-party systems. Since it is also triggered by on_stop, checking that the connector's operating status is RUNNING before starting any data ingestion is important
  • Triggers (any of the following):
    • A command with action TetraScience.Connector.UpdateConfig is received.
      • sent by the data acquisition service after valid configuration is applied
    • A command with action TetraScience.Connector.Start is received.
      • invoked by base implementation of Connector.on_start
    • A command with action TetraScience.Connector.Stop is received.
      • invoked by base implementation of Connector.on_stop
  • Default Implementation:
    • None
• validate_config: A developer defined method that determines if a given connector config is valid:
  • Triggers:
    • A command with action TetraScience.Connector.ValidateConfig is received
      • sent by the platform when a user attempts to save connector config
  • Default implementation:
    • always returns {"valid": true}
• test_config: A developer-defined method that live-tests a given connector config — going beyond the static checks in validate_config (for example, actually connecting to the configured third-party system):
  • Triggers:
    • A command with action TetraScience.Connector.TestConfig is received
      • In TDP v4.7.0 and later, this command is sent when a user clicks the Test button on the connector's configuration page. On earlier platforms the command is never sent.
  • Behavior:
    • The SDK runs validate-then-test in one process: it first runs validate_config, and only if that passes does it run the live test_config probe.
  • Default implementation:
    • Raises an UNKNOWN_COMMAND response, which the platform reports as "unsupported". Behavior change on upgrade: a connector with no test_config override previously showed "Configuration test passed" (the platform fell back to validate_config); after upgrading to an SDK that implements this command, the Test button reports "This connector does not support configuration testing." Implement test_config to keep a passing test.
  • The ts_sdk_connectors_python.testing module provides a helper to exercise this same path (validate-then-test) from your own unit tests.

Idle and Disable/Shutdown methods

• shutdown: A method called when the connector and its container will be stopped
  • Triggers:
    • A command with action TetraScience.Connector.Shutdown is received.
      • sent by the platform when a user disables the connector
  • Default implementation:
    • calls on_shutdown
    • stops metrics collection, heartbeat, and command listener tasks
• on_shutdown: A developer-defined hook that runs when the connector is stopping. Connector specific cleanup can usually be implemented here without overriding shutdown
  • Triggers:
    • A command with action TetraScience.Connector.Shutdown is received.
      • sent by the platform when a user disables the connector
  • Default Implementation:
    • None
• on_stop: A developer-defined hook that runs when the connector's operating status is set to IDLE
  • Triggers (any of the following):
    • A command with action TetraScience.Connector.Stop is received.
      • this corresponds to setting the connector operating status to IDLE
  • Default Implementation:
    • reloads connector config, which subsequently calls on_connector_updated

Additional Connector commands

The Connector class also supports some other commands. These can be sent using the commands API

Action Name	Connector Method Called	Description	Triggers	Side-Effects
TetraScience.Connector.ListCustomCommands	handle_get_available_custom_commands	This method returns a list of available custom commands registered to the connector.
TetraScience.Connector.SetLogLevel	set_log_level	This method is used to set the log level of the connector dynamically. Supported levels DEBUG, INFO, WARNING, ERROR, CRITICAL.	User sends a command to set the log level	Adjusts the verbosity of logs without restarting the connector
custom commands	specified by @register_command decorators	Custom commands registered using the @register_command decorator.

Custom commands

Custom commands are user-defined commands that can be registered by developers to extend the functionality of a connector. This can be useful for implementing capabilities that you want the connector to have, but only to use on demand. One example would be ingesting historical data from a given time window for a connector that typically only receives new data. Another example might be requests for the connector to send information to a third-party system. This gives Tetra pipelines a way to call upon the connector to act on their behalf.

Connector implements a command listener that both listens to all the previously mentioned standard commands, and also checks a connector-specific registry for custom commands. Custom commands are registered using the @register_command decorator. The decorator takes a string argument that corresponds to the action of the command. The convention for action names is TetraScience.Connector.<ConnectorName>.<CustomActionName>, as in the following example:

from ts_sdk_connectors_python.custom_commands import register_command
from ts_sdk_connectors_python.connector import Connector

class MyConnector(Connector):
    @register_command("TetraScience.Connector.ExampleConnector.MyCustomAction")
    def my_custom_action(self, body: dict):
        print(f"Action called with body: {body}")
        return None

Commands contains further technical details on custom command and command registration.

The return type of the custom command method should be either None, a dictionary, or a string that can be converted into a dictionary. Specifically, you can refer to the ts_sdk_connectors_python.models.CommandResponseBody type for more details.

Polling

The SDK provides a Poll class that allows repeated execution of a target function at a specified interval. This is useful for tasks that need to be performed periodically, such as checking the status of a resource or polling an API.

For more details on how to use the Poll class, refer to the Polling Documentation.

Logging

The SDK provides a logger to facilitate structured logging. This logger supports multi-threading, asynchronous operations, logger inheritance, and upload to CloudWatch.

Log messages are in JSON format for uptake into CloudWatch. The following are examples of log messages.

{"level":"debug","message":"Loading TDP certificates from local volume /etc/tetra/tdp-cert-chain.pem","extra":{"context":"ts_sdk_connectors_python.AuthenticatedClientCreator"}}
{"level":"info","message":"TDP certificates loaded from local volume /etc/tetra/tdp-cert-chain.pem","extra":{"context":"ts_sdk_connectors_python.AuthenticatedClientCreator"}}
{"level":"info","message":"Client initialized","extra":{"context":"ts_sdk_connectors_python.tdp_api_base","orgSlug":"tetrascience-yourorg","connectorId":"3eca48c9-3eb2-4414-a491-a8dda151da50"}}
{"level":"info","message":"Starting metrics task: cpu_usage_metrics_provider","extra":{"context":"ts_sdk_connectors_python.metrics"}}
{"level":"info","message":"Starting metrics task: memory_used_metrics_provider","extra":{"context":"ts_sdk_connectors_python.metrics"}}

Logger usage

The CloudWatch logger supports logger inheritance, allowing you to create child loggers that inherit the configuration and context of their parent loggers. This is useful for organizing log messages by component or module.

The get_logger method will return a logger that inherits from the connector SDK's root logger. Simply give the logger a useful name and begin using the logger. A new logger name will created by adding new suffix (see below). Providing the extra argument will add additional information present in all log messages with the logger. The extra argument can also be provided to any of the log methods (info, debug, warning, error, critical).

Example usage:

from ts_sdk_connectors_python.logger import get_logger
# Create a parent logger
parent_logger = get_logger("parent_logger", extra={"foo": "bar"})

assert parent_logger.name == 'ts_sdk_connectors_python.parent_logger'

parent_logger.info('my message', extra={'baz': 'bazoo'})

# expected log message
# note that 'foo' and 'baz' are included as 'extra'
# note that the logger name is also given in the 'extra.context'
{"level":"info","message":"my message","extra":{"context":"ts_sdk_connectors_python.parent_logger","foo":"bar","baz":"bazoo"}}

The following methods are provided to create logs at various levels:

logger.debug('Use this for detailed debug information. This is the lowest level and by default not
 emitted by the logger')
logger.info('Use this for general info. This is the default level for connectors')
logger.warning('Use this for warnings')
logger.error('Use this for errors. Note the exc_info which can provide stack trace info', exc_info=True)
logger.critical('Use this for critical errors that cause failure')

You may also create child loggers by using the get_child method, which will just add another suffix to an existing logger and merge provided extra.

# Create a child logger that inherits from the parent logger
child_logger = parent_logger.get_child("child", extra={"baz": "qux"})
assert child_logger.name == 'ts_sdk_connectors_python.parent_logger.child_logger'

# Log messages using the child logger
child_logger.info("This is a message from the child logger")

# note the extra is merged with the parent extra
{"level":"info","message":"my message","extra":{"context":"ts_sdk_connectors_python.parent_logger.child_logger","foo":"bar","baz":"qux"}}

Setting log levels

To reduce the volume of logs, you can set the log level. The supported levels are NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL. This can be done via the set_root_connector_sdk_log_level method:

from ts_sdk_connectors_python.logger import set_root_connector_sdk_log_level

set_root_connector_sdk_log_level("DEBUG")

By default, all loggers made by get_logger will have an NOTSET log level, meaning they all inherit their effective log level from the root connector SDK. It is therefore not recommended to set the log level for any child loggers. Instead, use the set_root_connector_sdk_log_level method.

Connector also implements the command TetraScience.Connector.SetLogLevel which allows you to set the log level of the connector dynamically. Here is an example command request:

{
  "payload": {
    "level": "DEBUG"
  },
  "action": "TetraScience.Connector.SetLogLevel",
  "targetId": "your-connector-id"
}

CloudWatch logging and standalone connector support

If the connector is in standalone mode (meaning CONNECTOR_TOKEN env var is set), logs will get uploaded to AWS CloudWatch in addition to getting logged to console. Logging and CloudWatch reporting occurs on a separate processing thread apart from the main connector processing thread. Uploading to CloudWatch occurs in batches and whenever a batch size is hit or on a set interval. Relevant envvars related to these settings can be found in constants.py.

The CloudWatchReporter class is responsible for managing the buffering and flushing of log events to AWS CloudWatch. It handles the following tasks:

• Buffering log events.
• Flushing buffered log events to CloudWatch based on certain conditions (e.g., buffer size limit, flush interval).
• Managing the CloudWatch log stream and log group.
• Handling errors during the flushing process.

Flushing log events to AWS CloudWatch occurs for a number of reasons:

• The buffer reaches its size limit.
• The flush interval is reached.
• The flush limit is reached.
• The connector is started.
• The connector is stopped.
• An explicit flush is triggered.

Product telemetry

ts_sdk_connectors_python.telemetry emits product events — what users do in your data app — to the TDP telemetry pipeline. This is separate from the CloudWatch logging above: logs are for debugging your app, product events are for usage analytics and dashboards.

It is a thin layer over stock OpenTelemetry (opentelemetry-sdk + BatchLogRecordProcessor + the OTLP/HTTP log exporter). Events are batched in the background and posted to the TDP gateway as OTLP/protobuf, using the stock exporter unmodified — the gateway decodes protobuf directly, so this distro no longer carries a JSON shim.

The same contract is implemented in JavaScript for React data apps (@tetrascience-npm/request), so a Python-emitted and a JS-emitted event for the same action land as the same record.

Emitting events

from ts_sdk_connectors_python.telemetry import track_error, track_event

# Event names follow `App[:Page][:Object]:Action`
track_event("Sandbox:Report:Exported", {"report.format": "pdf"})

# Attributes are for filtering and grouping; the optional third argument is a
# free-form payload serialized into the record body.
track_event("Sandbox:Query:Run", {"query.source": "sql"}, {"row_count": 42})

try:
    export_report()
except Exception as error:
    track_error(error, {"report.format": "pdf"})

Telemetry is fire-and-forget. If the endpoint is unset, unreachable, or the exporter fails, nothing is raised into your app — failures are logged at debug level only. You never need a try around a track_event call.

Counting

There is no separate counter API, and you do not need one. One track_event call is one record is one increment: the collector's count connector matches every product event and derives a CloudWatch metric grouped by event name, so anything you emit is already counted.

track_event("Sandbox:Report:Exported")
track_event("Sandbox:Report:Exported", {"report.format": "pdf"})

Attributes are optional — pass them when they are worth filtering on, omit them for a bare tally. To record a magnitude, emit the event that many times rather than passing the number as an attribute; a dimension is not something most backends can sum. If you need a value rather than a count, that is gauge / histogram / up_down_counter, which are OTel metrics rather than product events.

(A product-event counter(name) once shipped alongside track_event, but it was removed at contract 4.0.0: the count connector keys off the signal type, so it counted both identically, and two spellings invited emitting one action under both — which made COUNT(name) count neither the actions nor the increments. The counter() the package still exports is a different thing — the OTel metrics counter (counter(name, delta, attributes), a monotonic total), not a product-event helper.)

Configuration

Everything is read from the environment the container runtime injects; no setup call is required.

Variable	Set by	Purpose
ARTIFACT_NAMESPACE	data-acquisition	service.namespace
ARTIFACT_SLUG	data-acquisition	ts.artifact.slug, and service.name (ts-data-app-<slug>)
ARTIFACT_VERSION	data-acquisition	service.version
ORG_SLUG	data-acquisition	ts.org producer hint (the gateway re-stamps the authenticated org)
TDP_ENDPOINT	data-acquisition	Gateway base URL; the SDK appends /v1/otlp/logs
OTEL_COLLECTION_ENABLED	data-acquisition	Platform kill switch; false disables emission
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT	data-acquisition	Stock OTel fallback (internal collector — no identity stamping)
OTEL_EXPORTER_OTLP_ENDPOINT	data-acquisition	Stock OTel fallback; the SDK appends /v1/logs
TS_TELEMETRY_ENABLED	app author	false turns emission off; overrides OTEL_COLLECTION_ENABLED
TS_TELEMETRY_DEBUG	app author	true also prints every record to stdout. Off by default
TS_TELEMETRY_LOGS_ENDPOINT	app author	Explicit OTLP logs URL; wins over everything above
TS_TELEMETRY_METRICS_ENABLED	app author	Turn on OTel metrics export. Off by default
TS_TELEMETRY_TRACING_ENABLED	app author	Turn on OTel tracing export. Off by default
TS_TELEMETRY_METRICS_ENDPOINT	app author	Explicit OTLP metrics URL
TS_TELEMETRY_TRACES_ENDPOINT	app author	Explicit OTLP traces URL

Enable precedence — the two switches are not independent. TS_TELEMETRY_ENABLED is read FIRST and short-circuits, so it overrides OTEL_COLLECTION_ENABLED rather than combining with it:

TS_TELEMETRY_ENABLED	OTEL_COLLECTION_ENABLED	Result
unset	unset	enabled — deployed containers emit with no configuration
unset	false	disabled (the platform veto works)
false	anything	disabled
true	false	enabled — the platform veto is defeated

That last row is the trap: setting the app flag true anywhere durable — a Dockerfile ENV, a process supervisor, an entrypoint — makes data-acquisition's kill switch permanently unreachable. Leave both unset and let the task environment decide; the default is already on.

When emission is disabled, or when no endpoint resolves, no exporter is built, no background thread runs, and no socket is opened.

To override anything (an explicit endpoint, an auth-token provider, a session id), call init_telemetry(...) once at startup:

from ts_sdk_connectors_python.telemetry import init_telemetry

init_telemetry(
    auth_token_provider=lambda: current_user_token(),
    debug=True,
)

What the SDK never sends

• No user identity. The SDK never emits a user id, email, or name. Identity is stamped server-side at the gateway from the authenticated principal, so it cannot be spoofed by app code. Attributes that look like user identity (user_id, ts.user.*, enduser.id, ...) are stripped before emission.
• No secrets. Attribute and body keys matching authorization, token, api_key, secret, password, credential, cookie, session, or email are stripped client-side, recursively for bodies.
• Credential-shaped text in exceptions is redacted, but that is not a licence to put it there. track_error and span.record_error pass the exception message and traceback, and a span's status description, through redact_secrets before truncating them. It replaces JWTs, Bearer/Basic credentials, password= / "client_secret": "…" / x-api-key: … style assignments, AWS access key ids, URL userinfo (postgres://user:pass@host), PEM private keys and CryptoJS ciphertext with ****. This is pattern matching, so a bare secret with no recognisable shape, as in raise RuntimeError(f"lookup failed for {api_key}"), ships verbatim. Never put a credential in an exception message. Attribute values are not inspected. The patterns come from the shared contract and redact byte-for-byte the same as the JS distro.

OpenAPI code generation

This project uses code generation to build client libraries for interacting with the data acquisition service in Tetra based on the OpenAPI specification of the service. The generated code is placed in the ts_sdk_connectors_python/openapi_codegen directory.

Typical users of the SDK will not need to generate this code. For Tetra developers, details are available here

Using the generated API client

The generated API client is used within the TdpApi class to interact with the Tetra REST API. The TdpApi class provides asynchronous methods, while the TdpApiSync class provides synchronous methods.

Example usage

from ts_sdk_connectors_python.tdp_api import TdpApi
from ts_sdk_connectors_python.openapi_codegen.connectors_api_client.models import SaveConnectorValueRequest

async def main():
    api = TdpApi()
    await api.init_client()

    connector_id = "your_connector_id"
    raw_data = [
        SaveConnectorValueRequest(
            key="a-string",
            value={"some_json_field": "some_secret_value"},
            secure=True
        )
    ]

    response = await api.save_connector_data(connector_id, raw_data)
    print(response)

# Run the main function in an async event loop
import asyncio
asyncio.run(main())

For synchronous usage, use the TdpApiSync class:

from ts_sdk_connectors_python.tdp_api_sync import TdpApiSync
from ts_sdk_connectors_python.openapi_codegen.connectors_api_client.models import SaveConnectorValueRequest

def main():
    api = TdpApiSync()
    api.init_client()

    connector_id = "your_connector_id"
    raw_data = [
        SaveConnectorValueRequest(
            key="a-string",
            value={"some_json_field": "some_secret_value"},
            secure=True
        )
    ]

    response = api.save_connector_data(connector_id, raw_data)
    print(response)

    # retrieve parsed DTO object, if available
    print(response.parsed)

# Run the main function
main()

Retrieving connector data with filtering

The Python SDK supports server-side filtering when retrieving connector data. This allows you to efficiently retrieve only the data you need by specifying keys at the API level.

Using TdpApi (async)

from ts_sdk_connectors_python.tdp_api import TdpApi

async def main():
    api = TdpApi()
    await api.init_client()

    connector_id = "your_connector_id"

    # Get all connector data (no filtering)
    all_data = await api.get_connector_data(connector_id)
    print(f"All data: {len(all_data.parsed.values)} items")

    # Get specific keys only (server-side filtering)
    filtered_data = await api.get_connector_data(
        connector_id,
        keys="key1,key2,key3"  # Comma-separated list of keys
    )
    print(f"Filtered data: {len(filtered_data.parsed.values)} items")

asyncio.run(main())

Using TdpApiSync (synchronous)

from ts_sdk_connectors_python.tdp_api_sync import TdpApiSync

def main():
    api = TdpApiSync()
    api.init_client()

    connector_id = "your_connector_id"

    # Get specific keys only (server-side filtering)
    filtered_data = api.get_connector_data(
        connector_id,
        keys="key1,key2"
    )
    print(f"Filtered data: {len(filtered_data.parsed.values)} items")

main()

Using Connector class methods

The Connector class provides convenient methods that automatically use server-side filtering:

from ts_sdk_connectors_python.connector import Connector
from ts_sdk_connectors_python.tdp_api import TdpApi

async def main():
    api = TdpApi()
    await api.init_client()

    connector = Connector(api)

    # Get specific values (uses server-side filtering automatically)
    values = await connector.get_values(["key1", "key2"])
    print(f"Retrieved {len(values)} values")

    # Get single value
    single_value = await connector.get_value("key1")
    if single_value:
        print(f"Value for key1: {single_value.value}")

    # Direct access to connector data with filtering
    data = await connector.get_values(
        keys=["key1", "key2"],
    )
    print(f"Direct access: {len(data)} items")

asyncio.run(main())

Refer to the TdpApi and TdpApiSync class methods for more details on available API interactions.

Local testing with standalone connectors

For local development and testing, you can use standalone connectors to test your connector implementation against Tetra resources while running your code locally. This approach allows you to:

• Test your connector logic without deploying to a Hub
• Debug and iterate quickly during development
• Validate your connector against real Tetra services

Prerequisites

1. Build your local Docker image: Ensure you have built a local Docker image of your connector
2. Access to Tetra environment: You need access to a Tetra organization and appropriate permissions
3. Standalone installer: Access to the TetraScience standalone connector installer

Setup process

Step 1: build your local connector image

First, build your connector as a Docker image locally. This typically involves:

# Example build command (adjust based on your connector's Dockerfile)
docker build -t my-connector:local .

Step 2: use the standalone installer

1. Run the standalone connector installer provided by TetraScience

2. When prompted for the connector image during installation, provide your local image name instead of an official image:

   # Instead of using an official image like:
   # tetrascience/my-connector:v1.0.0
   
   # Use your local build:
   my-connector:local
   

3. The installer will:

   • Set up the necessary Tetra resources (tokens, certificates, etc.)
   • Configure environment variables for standalone operation
   • Create the appropriate Docker run configuration pointing to your local image

Step 3: environment configuration

The standalone installer will configure the following key environment variables:

• CONNECTOR_TOKEN: Authentication token for standalone deployment
• TDP_ENDPOINT: Tetra API endpoint
• ORG_SLUG: Your organization identifier
• CONNECTOR_ID: The connector instance ID
• Other Tetra-specific configuration as needed

Step 4: running your local connector

After setup, your connector will run using your local Docker image but connect to real Tetra services for authentication, data storage, and command processing.

Alternative: running containerless locally

It is also possible for testing to run the connector locally without Docker:

• Skip Step 1 above
• Get the connector token and standalone installer in Step 2, but do not run it
• From the installer.sh file, you can extract values for the environment variables (other than CONNECTOR_TOKEN, which you can get as part of Step 2) and export them. The necessary environment variables are those mentioned in Configuring TdpApi
• In Step 4, run the entrypoint of the connector directly

Important notes

• Ensure your local Docker image is built before running the standalone installer
• The standalone installer handles all Tetra resource provisioning and configuration
• Your local connector will have the same capabilities as a Hub-deployed connector
• Use this method for development and testing; production deployments should use Hub or standalone deployment of the official image

Running tests

Unit tests and integration tests are located in the __tests__/unit and __tests__/integration directories, respectively. To run unit tests, execute:

poetry run pytest

To run integration tests against the Tetra API, export the connector environment variables for tests and run:

poetry run pytest --integration

It is easiest to run the integration tests by following the procedure described in Local testing with standalone connectors.

Changelog

v2.4.0

Features:

• Add a standard testConfig command so a connector's configuration can be live-tested (validate-then-test): the SDK runs configuration validation, then the connector's testConfig probe. In TDP v4.7.0 and later, the Test button on the connector configuration page sends this command; on earlier platforms it is never sent. Behavior change: after this upgrade a connector that has not implemented test_config reports as unsupported on the Test button (previously the platform fell back to validate_config and it showed as passed) — implement test_config to keep a passing test.
• Add data-app OpenTelemetry support (track_event / track_error, traces, and metrics) for Streamlit/FastAPI data apps. The SDK emits telemetry only when your app calls track_event/track_error; once it does, emission is on by default and can be turned off with TS_TELEMETRY_ENABLED=false. Traces and metrics are off by default (enable with TS_TELEMETRY_TRACING_ENABLED / TS_TELEMETRY_METRICS_ENABLED). The TDP telemetry ingest route is available in TDP v4.7.0 and later, and the org must have the DATA_APPS_EVENT_ANALYTICS feature flag — without it the gateway returns 403; on older platforms there is no ingest route (404). Either way emission degrades gracefully: each batch gets the error and is dropped, one error line per batch, with no effect on connector behavior.

Logging & security hardening:

• Keep sensitive values out of logs: command handling and HTTP client logs now emit only non-sensitive identifiers; request/response bodies, URLs (including presigned S3 URLs), and headers are no longer logged.
• Stop error paths from echoing raw request or exception content.
• Redact secrets from telemetry exception messages and stacktraces.
• Escape SQS-unsafe characters (U+FFFE/U+FFFF, C0 controls, lone surrogates) in command responses so messages deliver reliably and round-trip losslessly.
• Raise security floors so downstream connectors inherit patched versions: anyio (>= 4.14.2), urllib3 (>= 2.8.0), cryptography (>= 50.0.0), joserfc (>= 1.6.8).
• Move openapi-python-client (the codegen generator) to the dev group and declare attrs, httpx, and python-dateutil as runtime dependencies, so connectors no longer pull the generator's vulnerable dependency tree into their runtime image.

Fixes:

• Return type-only errors on getConfig fetch failures (validate and test paths) so raw error content is not surfaced.

Testing & known limitations:

• Add a live-TLS test proving TDP_CERTIFICATE_BUNDLE verification is genuinely active on direct connections. The v2.3.0 proxy-tunnel limitation still applies: the bundle has no effect when traffic is tunneled through an HTTPS proxy (see "IAM proxy").

Upgrade notes (bump the SDK to >=2.4.0):

• Dependencies: openapi-python-client moved to the dev group, so its transitive packages (jinja2, typer, ruamel-yaml, shellingham, ruff) leave the runtime tree, and attrs / httpx / python-dateutil are now declared as direct runtime deps. The new security floors also cap major versions. Re-lock your connector, and if you import any of the dropped packages directly, declare them in your own dependencies.
• testConfig: a connector with no test_config override now reports as unsupported on the config Test button (it previously showed as passed). Implement test_config to keep a passing test. No other connector code changes are required.

v2.3.0

• Add IAM/forward-proxy support: when IAM_PROXY is set, AWS SDK (SSM/S3) calls route through the proxy instead of bypassing it
• Add custom CA bundle support for the Tetra platform's own certificate: when TDP_CERTIFICATE_BUNDLE is set, it is passed as the AWS client verify so a self-signed/org-issued CA that signs the Tetra platform certificate is trusted (distinct from the external "org certs" fetched elsewhere in the SDK, e.g. create_httpx_client)
• Add IAM_PROXY and TDP_CERTIFICATE_BUNDLE environment variables (behavior unchanged when unset)
• Pin urllib3 (>= 2.7.0) and aiohttp (>= 3.14.3) security floors at the SDK level so downstream connectors inherit patched versions

Known limitations:

• IAM_PROXY routes the AWS SDK (aioboto3) path only. The initial Tetra platform credential fetch uses a separate HTTP client that honors HTTPS_PROXY; set HTTPS_PROXY in addition to IAM_PROXY for environments that require all egress through the proxy.
• TDP_CERTIFICATE_BUNDLE is honored only on direct connections. Through an HTTPS proxy tunnel the upstream certificate is not verified against the bundle, so it has no effect on the proxied path.

v2.2.0

• Fix SqlConnector bug with wrong DTO type for file metadata
• Fix SqlConnector bug where unique_external_id and filepath were omitted when saving file
• Fix SqlConnector bug where source type was not correctly applied
• Add multi-query support for SqlConnector
• Allow multi-query SqlConnector to handle partial watermarks when some, but not all, queries return data
• Update SqlConnector to use ERROR status for open failed files to allow error_message to display
• Add option in SqlConnector to skip uploading files when no queries yield data

v2.1.0

• Adds support to create SQL connectors
• Updates supported Python versions to 3.11+ (including 3.12, 3.13, 3.14)
• Updates openapi-python-client to v0.27.1
• Updates black to v25.11.0
• Updates pylint to v3.3.0

v2.0.0

• Breaking: change connector helper methods to throw ConnectorError on unexpected API behavior instead of returning None
• For standalone connectors, capture logs before CloudWatch reporter initialization in buffer to be uploaded later
• Sort log events by timestamp before sending to CloudWatch
• Add filtering to Connector.get_values
• Add option to disable TLS verification to TdpApi.create_httpx_instance

v1.0.1

• Add automatic batching to Connector.get_files and Connector.save_files
• Add necessary S3 metadata to support destination_id for file uploads

v1.0.0

• Fix bug where proxy settings were loaded at wrong time from Hub
• Add synchronous init_client method for TdpApiSync

v0.9.0

• Add enum of health status to models.py
• Add ability to read connector manifest file
• Add support for user agent strings
• Refactor SDK to use AWS class
• Update aioboto3 to use upstream fix
• Add health reporting to CloudWatch logger
• Move some methods from Node SDK TdpClient to Python Connector class
• Fix crash loops for connectors unable to start in RUNNING

v0.8.0

• Add CI Pipeline to release PR builds to JFrog
• Add consistent AWS sessions through AWS class; fix cloud/hub/standalone deployment issues in v0.7.0
• ELN-661: Update Poll class to add default error handling and logging features

v0.7.0

• Update README to include logger practices
• Add HTTP request timeouts and SQS request timeouts
• Fix bugs causing tdpApi.upload_file to fail with error when using additional checksums
• Fix bug where the SDK ignore envar AWS_REGION in client init
• Support missing ConnectorFileDto.errorCount

v0.6.0

• Fix a 400 Bad Request error caused by the Tetra API client having an Authorization header that conflicted with the S3 presigned URL authentication

v0.5.0

• Fix bug where Connector.start fails when given an uninitialized TdpApi
• Improve logging in connector.start()

v0.4.0

• Implement CloudWatchReporter and logger to provide consistent logging by the SDK for local and cloudwatch logs
• Fix bugs in parsing ConnectorFileDto objects which formerly resulted in raised exceptions
• Introduce partial standalone deployment support for AWS and logger initialization

v0.3.0

• Add support to fetch connector JWT from AWS, allowing cloud connector deployment
• Use type SaveConnectorFilesRequest the signature of TdpApi.update_connector_files() and TdpApiSync.update_connector_files()
• Make CommandResponse.status optional to help with parsing messages from the command queue

v0.2.0

• Add upload_file method to TdpApi, TdpApiSync and Connector classes
• Bug fix for command request and response data validation
• Bug fix for parsing the incoming command body for validate_config_by_version

v0.1.0

• Initial version