External Systems

About Foundry Sources

The External Systems library is Python SDK built as an interface to reference Foundry Sources from code.

Installation

You can install the Python package using pip:

pip install external-systems

Basic Source Usage

HTTP Client

For REST based sources, a preconfigured HTTP client is provided built on top of the Python requests library. For on-prem systems using Agent Proxy the client will be pre-configured with the corresponding proxy

from external_systems.sources import Source, HttpsConnection
from requests import Session

my_source: Source = ...

https_connection: HttpsConnection = my_source.get_https_connection()

source_url: str = https_connection.url
http_client: Session = https_connection.get_client()

response = http_client.get(source_url + "/api/v1/example/", timeout=10)

Secrets

Source secrets can be referenced using get_secret("<secret_name>") on the source.

from external_systems.sources import Source

my_source: Source = ...

some_secret: str = my_source.get_secret("SECRET_NAME")

For sources using session credentials we support credentials generation and refresh management. This can be done by using get_session_credentials which supports S3 and BigQuery sources.

Session credentials may not be available in all Foundry runtime environments

from external_systems.sources import Source, Refreshable, SourceCredentials, AwsCredentials

s3_source: Source = ...

refreshable_credentials: Refreshable[SourceCredentials] = s3_source.get_session_credentials()

session_credentials: SourceCredentials = refreshable_credentials.get()

if not isinstance(session_credentials, AwsCredentials):
    raise ...

On-prem Connectivity with Foundry Agent Proxy

Socket

For non-HTTP connections to external systems that require connections through Foundry's agent proxy, a pre-configured socket is provided.

On-Prem SFTP Server Example

For this example we'll be using the fabric library

import fabric

from external_systems.sources import Source
from socket import socket

SFTP_HOST = <sftp_host>
SFTP_PORT = <sftp_port>

on_prem_proxied_source: Source = ...

username: str = on_prem_sftp_server.get_secret("username")
password: str = on_prem_sftp_server.get_secret("password")

proxy_socket: socket = source.create_socket(SFTP_HOST, SFTP_PORT)

with fabric.Connection(
    SFTP_HOST,
    user=username,
    port=SFTP_PORT,
    connect_kwargs={
        "password": password,
        "sock": proxy_socket,
    },
) as conn:
    sftp = conn.sftp()
    file_list = sftp.listdir(".")

Authenticated Proxy URI

For more granular use cases a pre-authenticated proxy URI is provided to allow connections to on-prem external systems.

Example

We'll be using the httpx library.

import httpx

from external_systems.sources import Source

on_prem_system: Source = ...

authenticated_proxy_uri: str = on_prem_system.get_https_proxy_uri()

with httpx.Client(proxy=authenticated_proxy_uri) as client:
    ...