adls-pandas-utils
Package that simplifies working with parquet files stored in Azure Data Lake Storage Gen2 (ADLS Gen2) from pandas. Developed by Datalier for use in data pipelines (bronze/silver processing and Power BI source data).
Authors: Niels Teunissen, Luuk Tijssen | Datalier
Installation
Install the package from the Azure DevOps Artifacts feed:
pip install adls-pandas-utils --index-url https://pkgs.dev.azure.com/<organization>/_packaging/<feed-name>/pypi/simple/
Requires Python 3.12+. Main dependencies: pandas, pyarrow, azure-storage-blob, azure-storage-file-datalake.
Modules
| Module | Purpose |
|---|---|
blob_helpers |
Reading and writing parquet files in ADLS Gen2 as pandas DataFrames (documented in this README) |
full_incr_load |
Full/incremental load patterns (load silver curated data, combine new + existing, move processed files) |
utils |
BearerTokenCredential (wraps an access token as a TokenCredential) and strtobool |
blob_helpers documentation
Public functions
combine_files_in_directory_as_dataframe(path_to_files, container_name, access_token, sa_name) -> pd.DataFrame | None
Reads all parquet files in a directory within an ADLS Gen2 container and combines them into a single DataFrame (pd.concat, index is reset).
| Parameter | Type | Description |
|---|---|---|
path_to_files |
str |
Path within the container where the parquet files are stored |
container_name |
str |
Name of the container (file system) in ADLS Gen2 |
access_token |
str |
Service principal access token for the storage account |
sa_name |
str |
Name of the storage account |
Returns: a single combined pd.DataFrame, or None when there are no files or reading fails (the error is printed, not raised).
from adls_pandas_utils.blob_helpers import combine_files_in_directory_as_dataframe
df = combine_files_in_directory_as_dataframe(
path_to_files="silver/new/",
container_name="datalake",
access_token=token,
sa_name="mystorageaccount",
)
Internal functions (_ prefix)
These functions are used internally, but can also be called directly from pipelines.
_dataframe_to_adls_gen2_parquet_file(df, access_token, destination_container, blob_directory, blob_name, sa_name) -> dict
Writes a DataFrame as a parquet file to ADLS Gen2. An existing blob with the same name is deleted first (overwrite behavior).
Before writing, the DataFrame is preprocessed so the parquet file stays Power BI-compatible under the pyarrow engine (see the pipeline below).
| Parameter | Type | Description |
|---|---|---|
df |
pd.DataFrame |
The DataFrame to upload |
access_token |
str |
Service principal access token |
destination_container |
str |
Destination container in ADLS Gen2 |
blob_directory |
str |
Directory path within the container (including trailing /) |
blob_name |
str |
File name of the blob |
sa_name |
str |
Name of the storage account |
Returns: metadata dict from upload_blob (e.g. etag, last_modified).
Note: during upload the file is temporarily written to the local working directory under blob_name and removed afterwards. Timestamps are written as microseconds (coerce_timestamps="us", truncation allowed).
_read_parquet_file_as_dataframe(file_path, container_system) -> pd.DataFrame
Downloads a single parquet file from ADLS Gen2 and reads it into a DataFrame (pyarrow engine, via an in-memory buffer).
| Parameter | Type | Description |
|---|---|---|
file_path |
str |
Path to the parquet file within the container |
container_system |
FileSystemClient |
ADLS Gen2 file system client |
_load_file_as_dataframe(file_path, container_system) -> pd.DataFrame
Wrapper around _read_parquet_file_as_dataframe that accepts either a str path or a path item from FileSystemClient.get_paths() (ItemPaged).
Preprocessing pipeline (Power BI compatibility)
When writing via _dataframe_to_adls_gen2_parquet_file, columns are coerced in this order so pyarrow can serialize them and Power BI's parquet connector interprets them correctly:
_coerce_bool_object_cols— object columns containing onlyTrue/False/None→ nullableboolean(parquet BOOLEAN). Runs before the numeric coercion, otherwise booleans would silently become 1/0._coerce_date_object_cols— date-like object/string columns →datetime64. Recognizesdatetime.date/datetime.datetimeinstances as well as date strings (YYYY-MM-DD,YYYY/MM/DD,DD-MM-YYYY,DD/MM/YYYY, ISO timestamps). Ambiguous formats are interpreted dayfirst (Dutch/EU); blank strings and unparseable values becomeNaT._coerce_numeric_object_cols— object columns withDecimal/mixed-numeric values or numeric strings → numeric (pd.to_numeric). Prevents parquet DECIMAL columns from landing in Power BI as text._datetime_to_us_naive— tz-aware datetimes → UTC without timezone, downcast to microseconds (datetime64[us]). Power BI's parquet connector fails on tz-aware TIMESTAMP and on TIMESTAMP_NANOS._object_to_str— remaining object columns (including all-None) →StringDtype(parquet STRING), because pyarrow cannot infer a type for mixed or empty object columns.
Build & publish
See how-to-build-publish.md. In short:
py -m build
py -m twine upload --repository pypi dist/*
Always bump the version in pyproject.toml first.
Tests
pytest --cov=src/adls_pandas_utils --cov-report=term-missing
Metadata
Release files for adls-pandas-utils 0.0.15
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| adls_pandas_utils-0.0.15.tar.gz | 18.0 kB | Without provenance | Details for adls_pandas_utils-0.0.15.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| adls_pandas_utils-0.0.15-py3-none-any.whl | Python 3 | none | any | Details for adls_pandas_utils-0.0.15-py3-none-any.whl |
Total release size: 29.9 kB
Release files / adls_pandas_utils-0.0.15.tar.gz
| Download URL | adls_pandas_utils-0.0.15.tar.gz |
|---|---|
| Size | 18.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e0193d5205e3a503d5b07c2c693a5490130b07b5c846b326c7447ed91f01441e
|
|
BLAKE2b-256 checksum How to use checksums |
9e55904a90b1e39200278bf17a0dd5d01366c5bd99459e544a237832c409d343
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
Release files / adls_pandas_utils-0.0.15-py3-none-any.whl
| Download URL | adls_pandas_utils-0.0.15-py3-none-any.whl |
|---|---|
| Size | 12.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
be546aa051791a9426cb8871bfad6673ca20b077587448bdc8bf7b2c20cf182b
|
|
BLAKE2b-256 checksum How to use checksums |
836ad052c920d9b95020faca6e3b9ca4bfbf7945c21be69ab5b80c9dfad676df
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
Security for adls-pandas-utils 0.0.15
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.