Skip to main content

adls-pandas-utils

Package that simplifies working with parquet files stored in Azure Data Lake Storage Gen2 (ADLS Gen2) from pandas. Developed by Datalier for use in data pipelines (bronze/silver processing and Power BI source data).

Authors: Niels Teunissen, Luuk Tijssen | Datalier

Installation

Install the package from the Azure DevOps Artifacts feed:

pip install adls-pandas-utils --index-url https://pkgs.dev.azure.com/<organization>/_packaging/<feed-name>/pypi/simple/

Requires Python 3.12+. Main dependencies: pandas, pyarrow, azure-storage-blob, azure-storage-file-datalake.

Modules

Module Purpose
blob_helpers Reading and writing parquet files in ADLS Gen2 as pandas DataFrames (documented in this README)
full_incr_load Full/incremental load patterns (load silver curated data, combine new + existing, move processed files)
utils BearerTokenCredential (wraps an access token as a TokenCredential) and strtobool

blob_helpers documentation

Public functions

combine_files_in_directory_as_dataframe(path_to_files, container_name, access_token, sa_name) -> pd.DataFrame | None

Reads all parquet files in a directory within an ADLS Gen2 container and combines them into a single DataFrame (pd.concat, index is reset).

Parameter Type Description
path_to_files str Path within the container where the parquet files are stored
container_name str Name of the container (file system) in ADLS Gen2
access_token str Service principal access token for the storage account
sa_name str Name of the storage account

Returns: a single combined pd.DataFrame, or None when there are no files or reading fails (the error is printed, not raised).

from adls_pandas_utils.blob_helpers import combine_files_in_directory_as_dataframe

df = combine_files_in_directory_as_dataframe(
    path_to_files="silver/new/",
    container_name="datalake",
    access_token=token,
    sa_name="mystorageaccount",
)

Internal functions (_ prefix)

These functions are used internally, but can also be called directly from pipelines.

_dataframe_to_adls_gen2_parquet_file(df, access_token, destination_container, blob_directory, blob_name, sa_name) -> dict

Writes a DataFrame as a parquet file to ADLS Gen2. An existing blob with the same name is deleted first (overwrite behavior).

Before writing, the DataFrame is preprocessed so the parquet file stays Power BI-compatible under the pyarrow engine (see the pipeline below).

Parameter Type Description
df pd.DataFrame The DataFrame to upload
access_token str Service principal access token
destination_container str Destination container in ADLS Gen2
blob_directory str Directory path within the container (including trailing /)
blob_name str File name of the blob
sa_name str Name of the storage account

Returns: metadata dict from upload_blob (e.g. etag, last_modified).

Note: during upload the file is temporarily written to the local working directory under blob_name and removed afterwards. Timestamps are written as microseconds (coerce_timestamps="us", truncation allowed).

_read_parquet_file_as_dataframe(file_path, container_system) -> pd.DataFrame

Downloads a single parquet file from ADLS Gen2 and reads it into a DataFrame (pyarrow engine, via an in-memory buffer).

Parameter Type Description
file_path str Path to the parquet file within the container
container_system FileSystemClient ADLS Gen2 file system client

_load_file_as_dataframe(file_path, container_system) -> pd.DataFrame

Wrapper around _read_parquet_file_as_dataframe that accepts either a str path or a path item from FileSystemClient.get_paths() (ItemPaged).

Preprocessing pipeline (Power BI compatibility)

When writing via _dataframe_to_adls_gen2_parquet_file, columns are coerced in this order so pyarrow can serialize them and Power BI's parquet connector interprets them correctly:

  1. _coerce_bool_object_cols — object columns containing only True/False/None → nullable boolean (parquet BOOLEAN). Runs before the numeric coercion, otherwise booleans would silently become 1/0.
  2. _coerce_date_object_cols — date-like object/string columns → datetime64. Recognizes datetime.date/datetime.datetime instances as well as date strings (YYYY-MM-DD, YYYY/MM/DD, DD-MM-YYYY, DD/MM/YYYY, ISO timestamps). Ambiguous formats are interpreted dayfirst (Dutch/EU); blank strings and unparseable values become NaT.
  3. _coerce_numeric_object_cols — object columns with Decimal/mixed-numeric values or numeric strings → numeric (pd.to_numeric). Prevents parquet DECIMAL columns from landing in Power BI as text.
  4. _datetime_to_us_naive — tz-aware datetimes → UTC without timezone, downcast to microseconds (datetime64[us]). Power BI's parquet connector fails on tz-aware TIMESTAMP and on TIMESTAMP_NANOS.
  5. _object_to_str — remaining object columns (including all-None) → StringDtype (parquet STRING), because pyarrow cannot infer a type for mixed or empty object columns.

Build & publish

See how-to-build-publish.md. In short:

py -m build
py -m twine upload --repository pypi dist/*

Always bump the version in pyproject.toml first.

Tests

pytest --cov=src/adls_pandas_utils --cov-report=term-missing

Metadata

Release files for adls-pandas-utils 0.0.15

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for adls-pandas-utils 0.0.15
File Size Uploaded Provenance Actions
adls_pandas_utils-0.0.15.tar.gz 18.0 kB Without provenance Details for adls_pandas_utils-0.0.15.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for adls-pandas-utils 0.0.15
File Interpreter ABI Platform Actions
adls_pandas_utils-0.0.15-py3-none-any.whl Python 3 none any Details for adls_pandas_utils-0.0.15-py3-none-any.whl

Total release size: 29.9 kB

Release files / adls_pandas_utils-0.0.15.tar.gz

Download URL adls_pandas_utils-0.0.15.tar.gz
Size 18.0 kB
Tags Source
SHA-256 checksum
How to use checksums
e0193d5205e3a503d5b07c2c693a5490130b07b5c846b326c7447ed91f01441e
BLAKE2b-256 checksum
How to use checksums
9e55904a90b1e39200278bf17a0dd5d01366c5bd99459e544a237832c409d343
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.3

Release files / adls_pandas_utils-0.0.15-py3-none-any.whl

Download URL adls_pandas_utils-0.0.15-py3-none-any.whl
Size 12.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
be546aa051791a9426cb8871bfad6673ca20b077587448bdc8bf7b2c20cf182b
BLAKE2b-256 checksum
How to use checksums
836ad052c920d9b95020faca6e3b9ca4bfbf7945c21be69ab5b80c9dfad676df
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.3

Security for adls-pandas-utils 0.0.15

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page