ka-uts-prc 4.1.0.250525

Process and Thread Management Utilities

ka_uts_prc

Overview

Utilities for Process and Thread Management

Installation

The package ka_uts_prc can be installed from PyPI or Anaconda.

To install with pip:

$ python -m pip install ka_uts_prc

To install with conda:

$ conda install -c conda-forge ka_uts_prc

This requires that the readme extra is installed:

$ python -m pip install ka_uts_prc[readme]

Modules

The Package ka_uts_prc contains the subsequent modules.

Modules of package ka_uts_prc*

Module
Name	Description
prc.py	Management of “array of arrays”.
run.py	Array of Dictionaries

Processing Module with classes

Module	Static Classes
Name	Type	Name	Description
prc	Task runner for iterator	Ra	Process Task decorated by the ray remote decorator for all iterator items.
		Co	Process Task using concurrent futures with process- or thread-pool.
		Mp	Process Tasks using multiprocessing with process- or thread-pool
		Sp	Process Task sequentially for all iterator items.
		Prc	Create iterator for path-templates optionally modified by transformer functions (if defined).
			Process Tasks by selecting the run-, process- or thread-function of one of the Processing classes Ra, Co, Mp, Sp with the run_method using the function table of run methods: d_run_method_2_fnc.

Special-Modules without classes

Module	Type	Statements
__init__	Package directory marker	Module with no Statements; the module is used to mark the directory it contains as a package. A dummy Module enforces explicit imports and thus clear namespace use and call them with the dot dot notation
__version__	Version	Assignment Statements for Version System Variables

Processing Modul: prc_by_dir

The Modul prc_by_dir contains the subsequent static classes with static variables and methods

Classes of module prc_by_dir

Static Classes
Name	Description
Ra	Execute tasks using Ray
Co	Execute tasks using Concurrent futures
Mp	Execute tasks using multiprocessing
Sp	Execute tasks using single processing
Prc	Run tasks with selected method

static Class: Ra

static Methods

Ra static Methods of prc

Static Method	Parameter: P / Return Value: RV
Name	Description	Name	Type	P/RV	Description
process	Run tasks in parallel by executing the task decorated with with the ray remote decorator for every iterator item	cls	class	P	current class
		kwargs	T_Dic	P	keyword arguments
			None	RV	no return value

static Class: Co

static Methods

Co static Methods of prc

Static Method	Parameter: P / Return Value: RV
Name	Description	Name	Type	P/RV	Description
process	Run tasks in parallel by applying the map method of the concurrent futures process pool executor to the task and the iterator.	cls	class	P	current class
		kwargs	T_Dic	P	keyword arguments
			None	RV	no return value
thread	Run tasks in parallel by applying the map method of the concurrent futures thread pool executor to the task and the iterator.	cls	class	P	current class
		kwargs	T_Dic	P	keyword arguments
			None	RV	no return value

static Class: Mp

static Methods

Mp static Methods of prc

Static Method	Parameter: P / Return Value: RV
Name	Description	Name	Type	P/RV	Description
process	Run tasks in parallel by applying the map method of the multiprocessing process pool to the task and the iterator.	cls	class	P	current class
		kwargs	T_Dic	P	keyword arguments
			None	RV	no return value
thread	Run tasks in parallel by applying the map method of the multiprocessing thread pool to the task and the iterator.	cls	class	P	current class
		kwargs	T_Dic	P	keyword arguments
			None	RV	no return value

static Class: Sp

static Methods

Sp static Methods of prc

Static Method	Parameter: P / Return Value: RV
Name	Description	Name	Type	P/RV	Description
run	Run tasks sequentially for all items of the iterator	cls	class	P	current class
		kwargs	T_Dic	P	keyword arguments
			None	RV	no return value

static Class: Prc

static Variables

Prc static Variables of prc

Static Variable
Name	Type	Description
d_run_method_2_fnc		T_Dic	Dictionary to translate run methods into run functions

static Methods

Prc static Methods of prc

Static Method	Parameter: P / Return Value: RV
Name	Description	Name	Type	P/RV	Description
do	Select executer with run method and run method function table.	cls	class	P	current class
	Create iterator using the Task function: yield_dl_template.	kwargs	T_Dic	P	keyword arguments
	Run the Task function: task with the executor.		None	RV	no return value

Special Modul: __version__

The Modul __version__ contains no classes, but assignment statements for system variables used by versioning.

__version_ System Variables

System Variable	Example
__title__	‘ka_uts_prc’
__description__	‘Processing Utilities.’
__url__	‘https://ka-com.readthedocs.io/en/latest’
__version___	‘1.0.0.24.08.24’
__build__	0x022200
__author_email_	‘Bernd Stroehle’
__license__	‘Apache-2.0’
__copyright__	‘Copyright 2024 Bernd Stroehle
__cake__	u’u2728 U0001f370 u2728’

Module Data

System Variable	Description
log.main.tenant.yml __copyright__	‘ka_uts_prc’ ‘Copyright 2023 bs29’

Appendix

Package Logging

Description

The Standard or user specifig logging is carried out by the log.py module of the logging package ka_uts_log using the standard- or user-configuration files in the logging package configuration directory:

• <logging package directory>/cfg/ka_std_log.yml,

• <logging package directory>/cfg/ka_usr_log.yml.

The Logging configuration of the logging package could be overriden by yaml files with the same names in the application package- or application data-configuration directories:

• <application package directory>/cfg

• <application data directory>/cfg.

Log message types

Logging defines log file path names for the following log message types: .

1. debug

2. info

3. warning

4. error

5. critical

Log types and Log directories

Single or multiple Application log directories can be used for each message type:

Log types and directoriesg

Log type	Log directory
long	short	multiple	single
debug	dbqs	dbqs	logs
info	infs	infs	logs
warning	wrns	wrns	logs
error	errs	errs	logs
critical	crts	crts	logs

Application parameter for logging

Application parameter used in log naming

Name	Decription	Values	Example
dir_dat	Application data directory		/otev/data
tenant	Application tenant name		UMH
package	Application package name		otev_xls_srr
cmd	Application command		evupreg
pid	Process ID		681025
log_ts_type	Timestamp type used in logging files|ts, dt	ts, dt’	ts
log_sw_single_dir	Enable single log directory or multiple log directories	True, False	True

Log files naming

Naming Conventions

Naming conventions for logging file paths

Type	Directory	File
debug	/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>	<Log type>_<ts>_<pid>.log
info	/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>	<Log type>_<ts>_<pid>.log
warning	/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>	<Log type>_<ts>_<pid>.log
error	/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>	<Log type>_<ts>_<pid>.log
critical	/<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory>	<Log type>_<ts>_<pid>.log

Naming Examples

Naming examples for logging file paths

Type	Directory	File
debug	/data/otev/umh/RUN/otev_xls_srr/evupreg/logs	debs_1737118199_9470.log
info	/data/otev/umh/RUN/otev_xls_srr/evupreg/logs	infs_1737118199_9470.log
warning	/data/otev/umh/RUN/otev_xls_srr/evupreg/logs	wrns_1737118199_9470.log
error	/data/otev/umh/RUN/otev_xls_srr/evupreg/logs	errs_1737118199_9470.log
critical	/data/otev/umh/RUN/otev_xls_srr/evupreg/logs	crts_1737118199_9470.log

Python Terminology

Python Packages

Overview

Python Packages Overview

Name	Definition
Python package	Python packages are directories that contains the special module __init__.py and other modules, packages files or directories.
Python sub-package	Python sub-packages are python packages which are contained in another pyhon package.
Python package sub-directory	directory contained in a python package.
Python package special sub-directory	Python package sub-directories with a special meaning like data or cfg

Examples

Python Package sub-directory-Examples

Name	Description
bin	Directory for package scripts.
cfg	Directory for package configuration files.
data	Directory for package data files.
service	Directory for systemd service scripts.

Python package files

Overview

Python package overview files

Name	Definition
Python package files	Files within a python package.
Special python package files	Package files which are not modules and used as python and used as python marker files like __init__.py.
Python package module	Files with suffix .py; they could be empty or contain python code; other modules can be imported into a module.
Special python package module	Modules like __init__.py or main.py with special names and functionality.

Examples

Python package examples files

Name	Type	Description
py.typed	Type checking marker file	The py.typed file is a marker file used in Python packages to indicate that the package supports type checking. This is a part of the PEP 561 standard, which provides a standardized way to package and distribute type information in Python.
__init__.py	Package directory marker file	The dunder (double underscore) module __init__.py is used to execute initialisation code or mark the directory it contains as a package. The Module enforces explicit imports and thus clear namespace use and call them with the dot notation.
__main__.py	entry point for the package	The dunder module __main__.py serves as an entry point for the package. The module is executed when the package is called by the interpreter with the command python -m <package name>.
__version__.py	Version file	The dunder module __version__.py consist of assignment statements used in Versioning.

Python methods

Overview

Python methods overview

Name	Description
Python method	Python functions defined in python modules.
Special python method	Python functions with special names and functionalities.
Python class	Classes defined in python modules.
Python class method	Python methods defined in python classes

Examples

Python methods examples

Name	Type	Description
__init__	class object constructor method	The special method __init__ is called when an instance (object) of a class is created; instance attributes can be defined and initalized in the method.

Table of Contents

Table of Content

• ka_uts_prc

  • Overview

  • Installation

  • Modules

    • Processing Modul: prc_by_dir

      • static Class: Ra

      • static Class: Co

      • static Class: Mp

      • static Class: Sp

      • static Class: Prc

        • static Variables

        • static Methods

    • Special Modul: __version__

    • Module Data

• Appendix

  • Package Logging

    • Description

    • Log message types

      • Log types and Log directories

      • Application parameter for logging

      • Log files naming

        • Naming Conventions

        • Naming Examples

  • Python Terminology

    • Python Packages

      • Overview

      • Examples

    • Python package files

      • Overview

      • Examples

    • Python methods

      • Overview

      • Examples

• Table of Contents