adls-pandas-utils

Package that simplifies working with parquet files stored in Azure Data Lake Storage Gen2 (ADLS Gen2) from pandas. Developed by Datalier for use in data pipelines (bronze/silver processing and Power BI source data).

Authors: Niels Teunissen, Luuk Tijssen | Datalier

Installation

Install the package from the Azure DevOps Artifacts feed:

pip install adls-pandas-utils --index-url https://pkgs.dev.azure.com/<organization>/_packaging/<feed-name>/pypi/simple/

Requires Python 3.12+. Main dependencies: pandas, pyarrow, azure-storage-blob, azure-storage-file-datalake.

Modules

blob_helpers documentation

Public functions

combine_files_in_directory_as_dataframe(path_to_files, container_name, access_token, sa_name) -> pd.DataFrame | None

Reads all parquet files in a directory within an ADLS Gen2 container and combines them into a single DataFrame (pd.concat, index is reset).

Returns: a single combined pd.DataFrame, or None when there are no files or reading fails (the error is printed, not raised).

from adls_pandas_utils.blob_helpers import combine_files_in_directory_as_dataframe

df = combine_files_in_directory_as_dataframe(
    path_to_files="silver/new/",
    container_name="datalake",
    access_token=token,
    sa_name="mystorageaccount",
)

Internal functions (_ prefix)

These functions are used internally, but can also be called directly from pipelines.

_dataframe_to_adls_gen2_parquet_file(df, access_token, destination_container, blob_directory, blob_name, sa_name) -> dict

Writes a DataFrame as a parquet file to ADLS Gen2. An existing blob with the same name is deleted first (overwrite behavior).

Before writing, the DataFrame is preprocessed so the parquet file stays Power BI-compatible under the pyarrow engine (see the pipeline below).

Returns: metadata dict from upload_blob (e.g. etag, last_modified).

Note: during upload the file is temporarily written to the local working directory under blob_name and removed afterwards. Timestamps are written as microseconds (coerce_timestamps="us", truncation allowed).

_read_parquet_file_as_dataframe(file_path, container_system) -> pd.DataFrame

Downloads a single parquet file from ADLS Gen2 and reads it into a DataFrame (pyarrow engine, via an in-memory buffer).

_load_file_as_dataframe(file_path, container_system) -> pd.DataFrame

Wrapper around _read_parquet_file_as_dataframe that accepts either a str path or a path item from FileSystemClient.get_paths() (ItemPaged).

Preprocessing pipeline (Power BI compatibility)

When writing via _dataframe_to_adls_gen2_parquet_file, columns are coerced in this order so pyarrow can serialize them and Power BI's parquet connector interprets them correctly:

1. _coerce_bool_object_cols — object columns containing only True/False/None → nullable boolean (parquet BOOLEAN). Runs before the numeric coercion, otherwise booleans would silently become 1/0.
2. _coerce_date_object_cols — date-like object/string columns → datetime64. Recognizes datetime.date/datetime.datetime instances as well as date strings (YYYY-MM-DD, YYYY/MM/DD, DD-MM-YYYY, DD/MM/YYYY, ISO timestamps). Ambiguous formats are interpreted dayfirst (Dutch/EU); blank strings and unparseable values become NaT.
3. _coerce_numeric_object_cols — object columns with Decimal/mixed-numeric values or numeric strings → numeric (pd.to_numeric). Prevents parquet DECIMAL columns from landing in Power BI as text.
4. _datetime_to_us_naive — tz-aware datetimes → UTC without timezone, downcast to microseconds (datetime64[us]). Power BI's parquet connector fails on tz-aware TIMESTAMP and on TIMESTAMP_NANOS.
5. _object_to_str — remaining object columns (including all-None) → StringDtype (parquet STRING), because pyarrow cannot infer a type for mixed or empty object columns.

Build & publish

See how-to-build-publish.md. In short:

py -m build
py -m twine upload --repository pypi dist/*

Always bump the version in pyproject.toml first.

Tests

pytest --cov=src/adls_pandas_utils --cov-report=term-missing