ka-uts-dfr 4.1.2.250609

Dataframe (Pandas, Polars) Utilities

ka_uts_dfr

Overview

Dataframe (Pandas, Polars) Utilities

Installation

Package ka_uts_dfr can be installed from PyPI or Anaconda.

To install with pip:

$ python -m pip install ka_uts_dfr

To install with conda:

$ conda install -c conda-forge ka_uts_dfr

This requires that the readme extra is installed:

$ python -m pip install ka_uts_dfr[readme]

Package Modules

Classification

The Modules of Package ka_uts_dfr could be classified into the following module classes:

1. Modules for pandas dataframe

2. Modules for polars dataframe

Modules for Pandas Dataframe

Modules for Pandas Dataframe

Name	Type
pddf.py	Pandas Dataframe

pddf.py

The Module pddf.py contains a single static classes PdDf.

pddf.py Class: PdDf

The static Class PdDf is used to manage Pandas Dataframes; it contains the subsequent methods.

PdDf Methods

Methods of static class PdDf

Name	Description
sh_d_aod	show dictionary of array of dictionaries.
sh_d_pddf	show dictionary of pandas dataframes.
pivot_table	create pandas dataframe pivot table. The pivot rules are defined by a pivot dictionary.
filter	Filter pandas dataframe. The filteris defined by filter dictionary
set_ix_drop_col_filter	set index and drop column filter
format-leading_zeros	format pandas dataframe columns with leading zeros
format-as-date	format pandas dataframe columns as date

PdDf Method: sh_d_aod

Parameter

Parameter of PdDf method sh_d_aod

Name	Type	Description
df	TyPdDf	Pandas Datafame
key	str	Keyword arguments

Return Value

Return Value of PdDf method sh_d_aod

Name	Type	Description
d_aod	TyDoAoD	dictionary of array of dictionaries

PdDf Method: sh_d_pddf

Parameter

Parameter of PdDf method sh_d_pddf

Name	Type	Description
cls	class	current class
df	TyPdDf	Pandas Datafame
key	str	keyword arguments

Return Value

Return Value of PdDf method sh_d_pddf

Name	Type	Description
d_df	TyDoPdDf	dictionary of pandas dataframes

PdDf Method: pivot_table

Parameter

Parameter of PdDf method pivot_table

Name	Type	Description
cls	class	current class
df	TyPdDf	pandas datafame
d_pv	TyDic	pivot table definition dictionary

Return Value

Return Value of PdDf method pivot_table

Name	Type	Description
dfpv	TyPdDf	pandas dataframe pivot table

PdDf Method: filter

Parameter

Parameter of PdDf method filter

Name	Type	Description
cls	class	current class
df	TyPdDf	pandas datafame
d_filter	TyDic	filter definition dictionary
relation	TyStr	filter relation

Return Value

Return Value of PdDf method filter

Name	Type	Description
df_new	TyPdDf	filtered pandas datafame

PdDf Method: set_ix_drop_col_filter

Parameter

Parameter of PdDf method set_ix_drop_col_filter

Name	Type	Description
cls	class	current class
df	TyPdDf	pandas datafame
d_filter	TyDic	filter definition dictionary
relation	str	filter relation

Return Value

Return Value of PdDf method set_ix_drop_col_filter

Name	Type	Description
df_new	TyPdDf	filtered pandas datafame

PdDf Module: format_leading_zeros

Parameter

Parameter of PdDf method format_leading_zeros

Name	Type	Description
cls	class	current class
df	TyPdDf	pandas datafame
d_filter	TyDic	filter definition dictionary
relation	str	filter relation

Return Value

Return Value of PdDf method format_leading_zeros

Name	Type	Description
df_new	TyPdDf	filtered pandas datafame

PdDf Method: format_as_date

Parameter

Parameter of PdDf method format_as_date

Name	Type	Description
cls	class	current class
df	TyPdDf	pandas datafame
d_filter	TyDic	filter definition dictionary
relation	str	filter relation

Return Value

Return Values of PdDf methodR ormat_as_date

Name	Type	Description
df_new	TyPdDf	filtered pandas datafame

Modules for Polars Dataframe

Modules for Polars Dataframe

Module	Classes
Name|Type	Name	Type	Description
pldf	Polars Dataframe		PdDf	Static	Manage Polars Dataframes

pldf.py

The Module pldf contains a single static class PLDF.

PlDf

The static Class PlDf contains the subsequent methods.

PlDf Methods

pldf Methods

Name	Description
filter	Filter polars dataframe using the given statement.
pivot	Create polars dataframe pivot table. The pivot rules are defined by the given pivot dictionary.
pivot_filter	Filter polars dataframe using the given statement and create polars dataframe pivot table from filtered dataframe. The pivot rules are defined by the given pivot dictionary.
to_aod	create pandas dataframe pivot table. The pivot rules are defined by pivot dictionary
to_doa	create pandas dataframe pivot table. The pivot rules are defined by pivot dictionary

PlDf Method: filter

Parameter

Parameter of PlDf method filter

Name	Type	Description
cls	class	current class
df	TyPdDf	polars datafame
stmt	TyStmt	filter statement

Return Value

Return Value of PlDf method filter

Name	Type	Description
df_new	TyPlDf	filtered polars datafame

PlDf Method: pivot

Parameter

Parameter of P.Df method pivot

Name	Type	Description
cls	class	current class
df	TyPlDf	polars datafame
d_pv	TyDic	pivot table definition dictionary

Return Value

Return value of PdDf method pivot

Name	Type	Description
dfpv	TyPlDf	polars dataframe pivot table

PlDf Method: pivot_filter

Parameter

Parameter of PdDf method pivot_filter

Name	Type	Description
cls	class	current class
df	TyPlDf	polars datafame
d_pv	TyDic	pivot table definition dictionary
stmt	TyStmt	filter statement

Return Value

Return value of PlDf method pivot_gilter

Name	Type	Description
dfpv	TyPlDf	polars dataframe pivot table

PlDf Method: to_aod

Parameter

Parameter of PdDf method to_aod

Name	Type	Description
df	TyPlDf	polars datafame

Return Value

Return value of PlDf method to_aod

Name	Type	Description
aod	TyAoD	Array of Dictionaries

PlDf Method: to_doa

Parameter

Parameter of PdDf method to_doa

Name	Type	Description
df	TyPlDf	polars datafame

Return Value

Return value of PlDf method to_doa

Name	Type	Description
doa	TyDoA	Dictionary of Arrays

Appendix

Package Logging

Description

The Standard or user specifig logging is carried out by the log.py module of the logging package ka_uts_log using the standard- or user-configuration files in the logging package configuration directory:

• <logging package directory>/cfg/ka_std_log.yml,

• <logging package directory>/cfg/ka_usr_log.yml.

The Logging configuration of the logging package could be overriden by yaml files with the same names in the application package- or application data-configuration directories:

• <application package directory>/cfg

• <application data directory>/cfg.

Log message types

Logging defines log file path names for the following log message types: .

1. debug

2. info

3. warning

4. error

5. critical

Log types and Log directories

Single or multiple Application log directories can be used for each message type:

Log types and directoriesg

Log type	Log directory
long	short	multiple	single
debug	dbqs	dbqs	logs
info	infs	infs	logs
warning	wrns	wrns	logs
error	errs	errs	logs
critical	crts	crts	logs

Application parameter for logging

Application parameter used in log naming

Name	Decription	Values	Example
dir_dat	Application data directory		/otev/data
tenant	Application tenant name		UMH
package	Application package name		otev_xls_srr
cmd	Application command		evupreg
pid	Process ID		681025
log_ts_type	Timestamp type used in logging files|ts, dt	ts, dt’	ts
log_sw_single_dir	Enable single log directory or multiple log directories	True, False	True

Log files naming

Naming Conventions

Naming conventions for logging file paths

Type	Directory	File
debug	/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>	<Log type>_<ts>_<pid>.log
info	/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>	<Log type>_<ts>_<pid>.log
warning	/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>	<Log type>_<ts>_<pid>.log
error	/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>	<Log type>_<ts>_<pid>.log
critical	/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>	<Log type>_<ts>_<pid>.log

Naming Examples

Naming examples for logging file paths

Type	Directory	File
debug	/data/otev/umh/RUN/otev_xls_srr/evupreg/logs	debs_1737118199_9470.log
info	/data/otev/umh/RUN/otev_xls_srr/evupreg/logs	infs_1737118199_9470.log
warning	/data/otev/umh/RUN/otev_xls_srr/evupreg/logs	wrns_1737118199_9470.log
error	/data/otev/umh/RUN/otev_xls_srr/evupreg/logs	errs_1737118199_9470.log
critical	/data/otev/umh/RUN/otev_xls_srr/evupreg/logs	crts_1737118199_9470.log

Python Terminology

Python Packages

Overview

Python Packages Overview

Name	Definition
Python package	Python packages are directories that contains the special module __init__.py and other modules, packages files or directories.
Python sub-package	Python sub-packages are python packages which are contained in another pyhon package.
Python package sub-directory	directory contained in a python package.
Python package special sub-directory	Python package sub-directories with a special meaning like data or cfg

Examples

Python Package sub-directory-Examples

Name	Description
bin	Directory for package scripts.
cfg	Directory for package configuration files.
data	Directory for package data files.
service	Directory for systemd service scripts.

Python package files

Overview

Python package overview files

Name	Definition
Python package files	Files within a python package.
Special python package files	Package files which are not modules and used as python and used as python marker files like __init__.py.
Python package module	Files with suffix .py; they could be empty or contain python code; other modules can be imported into a module.
Special python package module	Modules like __init__.py or main.py with special names and functionality.

Examples

Python package examples files

Name	Type	Description
py.typed	Type checking marker file	The py.typed file is a marker file used in Python packages to indicate that the package supports type checking. This is a part of the PEP 561 standard, which provides a standardized way to package and distribute type information in Python.
__init__.py	Package directory marker file	The dunder (double underscore) module __init__.py is used to execute initialisation code or mark the directory it contains as a package. The Module enforces explicit imports and thus clear namespace use and call them with the dot notation.
__main__.py	entry point for the package	The dunder module __main__.py serves as an entry point for the package. The module is executed when the package is called by the interpreter with the command python -m <package name>.
__version__.py	Version file	The dunder module __version__.py consist of assignment statements used in Versioning.

Python methods

Overview

Python methods overview

Name	Description
Python method	Python functions defined in python modules.
Special python method	Python functions with special names and functionalities.
Python class	Classes defined in python modules.
Python class method	Python methods defined in python classes

Examples

Python methods examples

Name	Type	Description
__init__	class object constructor method	The special method __init__ is called when an instance (object) of a class is created; instance attributes can be defined and initalized in the method.

Table of Contents

Table of Content

• ka_uts_dfr

  • Overview

  • Installation

  • Package Modules

  • Modules for Pandas Dataframe

    • pddf.py

      • pddf.py Class: PdDf

        • PdDf Methods

        • PdDf Method: sh_d_aod

          • Parameter

          • Return Value

        • PdDf Method: sh_d_pddf

          • Parameter

          • Return Value

        • PdDf Method: pivot_table

          • Parameter

          • Return Value

        • PdDf Method: filter

          • Parameter

          • Return Value

        • PdDf Method: set_ix_drop_col_filter

          • Parameter

          • Return Value

        • PdDf Module: format_leading_zeros

          • Parameter

          • Return Value

        • PdDf Method: format_as_date

          • Parameter

          • Return Value

  • Modules for Polars Dataframe

    • pldf.py

      • PlDf

        • PlDf Methods

        • PlDf Method: filter

          • Parameter

          • Return Value

        • PlDf Method: pivot

          • Parameter

          • Return Value

        • PlDf Method: pivot_filter

          • Parameter

          • Return Value

        • PlDf Method: to_aod

          • Parameter

          • Return Value

        • PlDf Method: to_doa

          • Parameter

          • Return Value

• Appendix

  • Package Logging

    • Description

    • Log message types

      • Log types and Log directories

      • Application parameter for logging

      • Log files naming

        • Naming Conventions

        • Naming Examples

  • Python Terminology

    • Python Packages

      • Overview

      • Examples

    • Python package files

      • Overview

      • Examples

    • Python methods

      • Overview

      • Examples

• Table of Contents