etlgen-client

Python SDK and command-line client for Fibtec's ETL Integrator: describe a source and a destination, and get back the Python that connects them, as an ETL script (optionally with tests), a REST or GraphQL connector, or a webhook handler. Nothing you upload is stored by the service, and this package stores and logs nothing either.

Status: alpha (0.1.x). Python 3.11+. One runtime dependency, httpx.

pip install etlgen-client

Before you start: authentication

Read this first, because it decides whether this release fits your use.

The API authenticates with a Cognito ID token that expires after one hour. There are no API keys yet. So you give the client either a token, or a function that returns a fresh one:

from etlgen_client import Client

client = Client(token=my_id_token, base_url="https://<your-api-id>.execute-api.<region>.amazonaws.com")

• It must be the ID token, not the access token: the API reads your organisation from the token's email claim.

• base_url is required. It is the API address your administrator gets from the EtlGenApi stack's ApiUrl output. It must be https (plain http is only accepted for localhost).

• To try it quickly, copy the ID token from your signed-in portal session (browser developer tools, Application, Local Storage, the entry whose name ends in .idToken). It is valid for an hour.

• For a script that keeps running, pass a function. This example uses pycognito, a third-party library (not a dependency of this package) that does Cognito's SRP password login and renews tokens. It works for password accounts, not Google or Apple sign-in:

  from pycognito import Cognito
  
  user = Cognito(user_pool_id="<pool id>", client_id="<app client id>", username="you@company.com")
  user.authenticate(password="...")
  
  def fresh_id_token() -> str:
      user.check_token()        # renews the tokens when they have expired
      return user.id_token
  
  client = Client(token=fresh_id_token, base_url="https://...")
  

  The function is called once per request.

This is not yet suited to unattended CI or server use, which needs a long-lived service credential the API does not have. Customer API keys are the planned follow-up; when they ship, this package gets an api_key= argument and nothing else changes.

Quick start

from pathlib import Path

from etlgen_client import Client, EtlGenInsufficientCreditError, EtlGenTooLargeError

source = {"format": "xsd", "content": Path("customer.xsd").read_text()}
destination = {"format": "json_schema", "content": Path("customer.schema.json").read_text()}

with Client(token=fresh_id_token, base_url=BASE_URL) as client:
    quote = client.generate.estimate(source, destination)      # free, writes nothing
    print(quote["tier"], quote["price_expected_gbp_pence"], "pence")

    try:
        result = client.generate.etl(source, destination, include_tests=True)
    except EtlGenTooLargeError as exc:
        print(f"{exc.field_count} fields, limit {exc.max_field_count}: split it up")
    except EtlGenInsufficientCreditError as exc:
        print(f"top up first: this job costs {exc.price_gbp_pence} pence")
    else:
        Path("etl_job.py").write_text(result["script"])
        if result["tests"]:
            Path("test_etl_job.py").write_text(result["tests"])
        print(result["unresolved_fields"], result["triangle"]["tes"])

The first argument of each call is the dict the API takes: {"format": ..., "content": ...} for a schema pair, an OpenAPI document (or a GraphQL introspection document) for a connector, a declared webhook surface for a webhook handler. The server validates them.

Methods

Why the generation calls are never retried. A paid-plan job deducts credit before it runs, and the API has no idempotency key, so retrying a call whose outcome is unknown could charge you twice. Reads and estimate are retried on connection errors, timeouts and 5xx (three retries, with backoff). A timeout on a generation call raises the underlying httpx exception.

The default timeout is 60 seconds, above API Gateway's own 30-second limit.

Errors

Every API error is an EtlGenError subclass carrying response_status and response_body. A network failure with no response raises the httpx exception.

The two 422 subclasses are also EtlGenValidationError.

Command line

pip install etlgen-client installs an etlgen command.

export ETLGEN_TOKEN=...                       # a Cognito ID token (prefer this over --token)
export ETLGEN_BASE_URL=https://<api-id>.execute-api.<region>.amazonaws.com

etlgen billing get
etlgen generate estimate --params params.json
etlgen generate etl --params params.json --out-dir ./generated
etlgen triangle get --params-json '{"job_id": "..."}'
etlgen list-functions --domain generate

params.json is a JSON object of the method's keyword arguments. Any string that starts with @file: is replaced by that file's contents, so a schema never has to be pasted into JSON:

{
  "source":      {"format": "xsd",         "content": "@file:customer.xsd"},
  "destination": {"format": "json_schema", "content": "@file:customer.schema.json"},
  "include_tests": true
}

A method with required arguments, called with neither --params nor --params-json, prints its signature and documentation instead.

--out-dir DIR (for etl, connector and webhook) writes the generated files under DIR (etl_job.py and test_etl_job.py for an ETL job, the portal's own names), overwriting existing files, and prints a summary without the file bodies. A path in the result that would escape DIR is refused.

Exit codes: 0 ok, 1 other error or bad input (including a network failure), 2 usage, 3 auth, 4 validation (including too large and content safety), 5 insufficient credit, 6 free-plan limit or quota, 7 rate limited, 8 not found.

A content-safety failure (exit 4) prints the check's reason on its own line: either that the script embeds a value from your sample data, or that the check itself gave no usable verdict. The work was done either way and is not refunded.

A failure with no response at all (connection refused, DNS, a timeout) prints one error: line, not a traceback. If it happened on generate etl, connector or webhook after the request may have been sent, a second line says the job may still have run and been charged: those calls are never retried, so run etlgen billing get before trying again.

Licence

Apache-2.0. Copyright Fibtec Limited.