sap-ecs-azure-log-forwarder 1.2.3

A package to consume events from an Azure storage queue, process log files, and forward them to a HTTP endpoint or file.

Azure Log Forwarder

Azure Log Forwarder continuously watches an Azure Storage Queue for blob creation messages, downloads the referenced log files, and delivers their contents according to the output mode you choose.

Features

• Monitors an Azure Storage Queue for blob creation messages.
• Processes logs and forwards them to a configurable HTTP endpoint, writes them to local files, or streams them to the console for ad-hoc inspection.
• Supports both mutual TLS and standard HTTP connections.
• Offers flexible HTTP authentication via bearer tokens or API keys.
• Fully configurable through environment variables or a .env file in the working directory.
• Provides an inactivity timeout to gracefully terminate when the queue is empty.

Prerequisites

• Python 3.10 or higher.
• An Azure Storage account with a configured storage queue.
• A SAS token with appropriate permissions to access the storage account and queue.

Installation

With direct internet access:

pip install sap-ecs-azure-log-forwarder

or

pip install sap-ecs-azure-log-forwarder==<version>

Without direct internet access:

1. Download the wheel
   On a machine with internet access, visit the PyPI files page for sap-ecs-azure-log-forwarder and download the .whl file matching your Python version (e.g., sap_ecs_azure_log_forwarder-1.0.2-py3-none-any.whl).
2. Transfer the file
   Copy the downloaded wheel to the target (offline) machine using SCP, or another secure method.
3. Install from file
   On the offline machine, run:

   pip install /path/to/sap_ecs_azure_log_forwarder-<version>-py3-none-any.whl
   

Configuration

Configure the forwarder by setting environment variables in your shell or in a .env file in the current working directory. The most common settings are listed below.

• SAS_TOKEN: SAS token for the LogServ Storage Account. (Required)

• STORAGE_ACCOUNT_NAME: Name of your LogServ Storage Account which the SAS token was generated for. (Required)

• STORAGE_QUEUE_NAME: Name of the Azure Storage Queue that the storage Account sends messages to. (Required)

• OUTPUT_METHOD: Method to forward logs (http, files, or console). (Required)

  • http: Forward logs to an HTTP endpoint.
  • files: Write logs to files in the specified output directory.
  • console: Print each log line to stdout without forwarding or writing it elsewhere. All other informational/debug logging is suppressed in this mode.

• TIMEOUT_DURATION: Time in seconds to wait for messages before exiting. (Optional)

• LOGSERV_LOG_INCLUDE_FILTERS: Comma-separated list of log type filters. Only messages whose subject paths contain at least one of these filters will be processed (e.g., hana, hanaaudit, linux). (Optional)

• LOGSERV_LOG_EXCLUDE_FILTERS: Comma-separated list of log type exclude filters. Messages whose subject paths containing any of these filters will be skipped. (Optional)

• HTTP_ENDPOINT: HTTP endpoint to forward logs to. (Required if OUTPUT_METHOD=http)

• TLS_CERT_PATH: Path to the TLS certificate for mutual TLS connections. (Optional)

• TLS_KEY_PATH: Path to the TLS key for mutual TLS connections. (Optional)

• AUTH_METHOD: Authentication method. (Required if OUTPUT_METHOD=http)

  • token: Use a bearer/OAuth token for authentication.
  • api_key: Use an API key for authentication.

• AUTH_TOKEN: Bearer/OAuth token for HTTP endpoint authentication. (Required if AUTH_METHOD=token)

• API_KEY: API key for HTTP endpoint authentication. (Required if AUTH_METHOD=api_key)

• CA_CERT_PATH: Path to the CA bundle certificate path, in case of a HTTPS endpoint using self-signed certificate ( Optional )

• DISABLE_SSL_VERIFY: [FOR DEBUGGING PURPOSES ONLY] - Disable the SSL verification, value could be yes, 1, true ( Optional )

• OUTPUT_DIR: Output directory to write log files to. (Required if OUTPUT_METHOD=files)

• COMPRESS_OUTPUT_FILE: Whether to gzip-compress output files. (Defaults to true)

  • true: Compress output files using gzip.
  • false: Do not compress output files.

• LOG_LEVEL: Log level for the forwarder application. (DEBUG, INFO, WARNING, ERROR, CRITICAL) (Defaults to INFO)

Example of setting environment variables in a shell:

export SAS_TOKEN="logserv_storage_account_SAS_token"
export STORAGE_ACCOUNT_NAME="logserv_storage_account_name"
export STORAGE_QUEUE_NAME="logserv_queue_name"
export TIMEOUT_DURATION=120  # Timeout after 120 seconds of inactivity. DO NOT set for indefinite runs.
export LOGSERV_LOG_INCLUDE_FILTERS="hana"
export LOGSERV_LOG_EXCLUDE_FILTERS="tmp"

# For HTTP output
export OUTPUT_METHOD="http"
export HTTP_ENDPOINT="https://your-http-endpoint.com"
export TLS_CERT_PATH="/path/to/your/tls_cert.pem"
export TLS_KEY_PATH="/path/to/your/tls_key.pem"
export AUTH_METHOD="token"
export AUTH_TOKEN="your_token"
export AUTH_METHOD="api_key"
export API_KEY="your_api_key"
export CA_CERT_PATH="/path/to/your/ca-bundle.crt"
export DISABLE_SSL_VERIFY="false"

# For file output
export OUTPUT_METHOD="files"
export OUTPUT_DIR="/path/to/your/output/directory/"
export COMPRESS_OUTPUT_FILE="false"

# For console output
# export OUTPUT_METHOD="console"

# Log level
export LOG_LEVEL="DEBUG" # Options: DEBUG, INFO, WARNING, ERROR, CRITICAL. Default is INFO.

Example of setting environment variables in a local .env file:

SAS_TOKEN="logserv_storage_account_SAS_token"
STORAGE_ACCOUNT_NAME="logserv_storage_account_name"
STORAGE_QUEUE_NAME="logserv_queue_name"
TIMEOUT_DURATION=120  # Timeout after 120 seconds of inactivity. DO NOT set for indefinite runs.
LOGSERV_LOG_INCLUDE_FILTERS="hana"
LOGSERV_LOG_EXCLUDE_FILTERS="tmp"

# For HTTP output
OUTPUT_METHOD="http"
HTTP_ENDPOINT="https://your-http-endpoint.com"
TLS_CERT_PATH="/path/to/your/tls_cert.pem"
TLS_KEY_PATH="/path/to/your/tls_key.pem"
AUTH_METHOD="token"
AUTH_TOKEN="your_token"
AUTH_METHOD="api_key"
API_KEY="your_api_key"
CA_CERT_PATH="/path/to/your/ca-bundle.crt"
DISABLE_SSL_VERIFY="false"

# For file output
OUTPUT_METHOD="files"
OUTPUT_DIR="/path/to/your/output/directory/"
COMPRESS_OUTPUT_FILE="false"

# For console output
# OUTPUT_METHOD="console"

# Log level
LOG_LEVEL="DEBUG" # Options: DEBUG, INFO, WARNING, ERROR, CRITICAL. Default is INFO.

Usage

To run the Azure Log Forwarder, use the following command:

sap-ecs-azure-log-forwarder

This starts the process of consuming messages from the storage queue, processing log files, and forwarding them according to the configured output mode. The program will exit if no messages are found within the specified timeout duration; otherwise it runs until you stop it.

Examples

Forwarding specific log types

To forward only specific log types, set LOGSERV_LOG_INCLUDE_FILTERS to a comma-separated list of filters.

For example, to forward HANA Audit logs, set the following environment variable:

In your shell:

export LOGSERV_LOG_INCLUDE_FILTERS="hanaaudit"

Or in your .env file:

LOGSERV_LOG_INCLUDE_FILTERS="hanaaudit"

Only files matching at least one of these filters will be processed and forwarded. If you want to forward all log types, simply omit the LOGSERV_LOG_INCLUDE_FILTERS variable or set it to an empty string. Please reach out to your SAP contact for the correct container structure or run the forwarder without the filter to see the available log types and their structure.

Excluding specific log types

To skip specific log types, set LOGSERV_LOG_EXCLUDE_FILTERS to a comma-separated list of filters.

In your shell:

export LOGSERV_LOG_EXCLUDE_FILTERS="tmp"

Or in your .env file:

LOGSERV_LOG_EXCLUDE_FILTERS="tmp"

Things to remember

• If TIMEOUT_DURATION is not set, the program will run indefinitely.
• Exclude filters are applied before include filters: any message matching an exclude filter will be skipped, even if it matches a include filter. In other words, if you set both LOGSERV_LOG_INCLUDE_FILTERS and LOGSERV_LOG_EXCLUDE_FILTERS, the exclude filters will take precedence over the include filters.

License

This application and its source code are licensed under the terms of the SAP Developer License Agreement. See the LICENSE file for more information.

Release Notes

1.0.1

• First proper release!

1.0.2

• Updated README with diagrams and instructions on installing the package without access to the internet.

1.0.3

• Added a new configuration option (LOG_LEVEL) to set the log level for the application.
• Updated README with instructions on setting the log level.

1.0.5

• Added a new configuration option (LOGSERV_LOG_FILTER) to filter log types.
• Added a new configuration option (COMPRESS_OUTPUT_FILE) to set the compression option for output files.
• Updated README with instructions on setting the log type filter and compression option.
• Skipped version 1.0.4 to resolve vulnerability issues in the dependencies.

1.0.6

• Changed LOGSERV_LOG_FILTER to LOGSERV_LOG_INCLUDE_FILTERS and made it a comma-separated list of filters.

1.0.7

• Increased the visibility timeout for the queue message to 1 minute instead of 5 seconds to allow for longer processing times.

1.1.0

• Added a new configuration option (LOGSERV_LOG_EXCLUDE_FILTERS) to specify negative filters for message subjects.
• Renamed LOGSERV_LOG_FILTERS to LOGSERV_LOG_INCLUDE_FILTERS for clarity.
• Skipped version 1.0.8 and 1.0.9 to resolve vulnerability issues in the dependencies.

1.1.1

• Added OUTPUT_METHOD=console to stream log contents directly to stdout.
• Modified queue message deletion to cope with Azure's MessageNotFound race condition.

1.2.0

• Include the CA_CERT_PATH and DISABLE_SSL_VERIFY options

1.2.1

• Bugfixes

1.2.2

• Bugfixes