airflow-memray 0.1.1

Memory profiling for Airflow with Memray.

Airflow Memray

Memory profiling for Airflow with Memray.

Configuration Reference

This section contains the list of all the available Airflow Memray configurations that you can set in airflow.cfg file or using environment variables.

base_folder

The base folder under which Airflow Memray will store profiling results. Possible values can be anything what is supported by Airflow Object Storage.

If it refers to a local file system path, then it must be accessible by the task and the webserver.

Default: "file:///tmp/airflow/memray"

Environment Variable: AIRFLOW__MEMRAY__BASE_FOLDER

storage_conn_id

The Airflow Connection to use if base_folder is set to a remote cloud storage location.

Default: None

Environment Variable: AIRFLOW__MEMRAY__STORAGE_CONN_ID

tasks

The tasks to be profiled as a comma separated list of wildcard pattern as implemented by the fnmatch module. The pattern are applied against the full task ID in the form <dag_id>.<task_id>.

Set it to "*" to profile all tasks.

Default: ""

Environment Variable: AIRFLOW__MEMRAY__TASKS

Airflow Summit 2024

I have given a presentation about this package at Airflow Summit 2024.

You can visit the official page of the presentation or directly watch video on YouTube by clicking on the following picture: