warpzone-sdk 19.0.0

The main objective of this package is to centralize logic used to interact with Azure Functions, Azure Service Bus and Azure Table Storage

WarpZone SDK

This package contains tools used in the WarpZone project. These tool include:

• Client for Storage
• Client for Servicebus
• Function wrapper

---

Client for Storage

Blob storage

WarpzoneBlobClient client is used for uploading to and downloading from Azure Storage Blob Service.

---

Client for Servicebus

Due to limitations on message sizes, we use different methods for sending events and data using Azure Service Bus.

Events

We use the Service Bus for transmitting event messages. By an event, we mean a JSON formatted message, containing information about an event occuring in one part of the system, which needs to trigger another part of the system (such as an Azure Function trigger).

WarpzoneEventClient client is used for sending and receiving events.

--

Data

We do not use the Service Bus for transmitting data directly. Instead, we use a claim-check pattern, were we store the data using Storage Blob, and transmit an event about the details of this stored data.

WarpzoneDataClient client is used for sending and receiving data in this way. The following diagram shows how the process works:

1. Data is uploaded
2. Event containing the blob location is send
3. Event is received
4. Data is downloaded using the blob location contained in the event

The transmitted event has the following format:

{
  "container_name": "<container-name>",
  "blob_name": "<blob-name>",
  "timestamp": "<%Y-%m-%dT%H:%M:%S%z>"
}

The data will be stored with

• <container-name> = <topic-name>
• <blob-name> = <subject>/year=<%Y>/month=<%m>/day=<%d>/hour=<%H>/<message-id>.<extension>

---

Function Wrapper

For executing logic, we use a framework built on top of Azure Functions. The following diagram shows how the framework works:

1. The function is triggered by a trigger object (e.g. a timer or a message being received)
2. Possible dependency objects are initialized (potentially using information from the trigger). These are used to integrate with external systems (e.g. a database client).
3. Using the trigger and dependencies as inputs, the function outputs and an output object (e.g. a message being sent).

The reason we have used our own framework instead of Azure Functions directly, is that we want to use our own objects as triggers, dependencies and outputs, instead of the built-in bindings. For example, as explained above, we have created our own abstraction of a message for transmitting data (warpzone.DataMessage); so we would like to use this, instead of the built-in binding azure.function.ServiceBusMessage.

Since it is not yet possible to define custom bindings in Python, we have defined our own wrapping logic, to handle the conversion between our own objects and the built-in bindings. The following diagram shows how the wrapping logic works:

1. Azure trigger binding is converted to trigger object
2. Either
   • (a) Output object is converted to Azure output binding.
   • (b) Use custom output logic, when no suitable output binding exists (e.g. we use the Azure Service Bus SDK instead of the Service Bus output binding, since this is recommended)
3. All logs and traces are sent to App Insights automatically.

--

Examples

Azure Function with data messages as trigger and output:

# function.json
{
  "scriptFile": "__init__.py",
  "entryPoint": "main",
  "bindings": [
    {
      "name": "msg",
      "type": "serviceBusTrigger",
      "direction": "in",
      "connection": "...",
      "topicName": "...",
      "subscriptionName": "..."
    }
  ]
}

import warpzone as wz

def do_nothing(data_msg: wz.DataMessage) -> wz.DataMessage:
    return data_msg

main = wz.functionize(
    f=do_nothing,
    trigger=wz.triggers.DataMessageTrigger(binding_name="msg"),
    output=wz.outputs.DataMessageOutput(wz.Topic.UNIFORM)
)

Azure Function with HTTP messages as trigger and output:

# function.json
{
    "scriptFile": "__init__.py",
    "entryPoint": "main",
    "bindings": [
      {
        "authLevel": "anonymous",
        "name": "req",
        "type": "httpTrigger",
        "direction": "in"
      },
      {
        "type": "http",
        "direction": "out",
        "name": "$return"
      }
    ]
  }

# __init__.py
import warpzone as wz
import azure.functions as func

def return_ok(req: func.HttpRequest) -> func.HttpResponse:
    return func.HttpResponse("OK")

main = wz.functionize(
    f=return_ok,
    trigger=wz.triggers.HttpTrigger(binding_name="req"),
    output=wz.outputs.HttpOutput()
)

Azure Function using dependencies:

import warpzone as wz

def do_nothing(
  data_msg: wz.DataMessage,
  db: wz.WarpzoneDatabaseClient,
) -> wz.DataMessage:
    return data_msg

main = wz.functionize(
    f=do_nothing,
    trigger=wz.triggers.DataMessageTrigger(binding_name="msg"),
    output=wz.outputs.DataMessageOutput(wz.Topic.UNIFORM),
    dependencies=[wz.dependencies.DeltaDatabaseDependency()],
)