lognflow 0.10.1

Log and Flow tracking made easy with Python

lognflow

Log and Flow tracking made easy with Python. You can install it by::

pip install lognflow

A simple program to use it would be similar to the following::

from lognflow import lognflow
import numpy as np

data = np.random.rand(100)

logger = lognflow(r'c:\all_logs\\')
logger('This is a test for lognflow and log_single')
logger.log_single('data', data)

The logviewer is also very useful::

from lognflow import logviewer
logged = logviewer(r'c:\all_logs\some_log\\')
data = logged.get_single('data')

The printprogress makes a pretty nice progress bar::

from lognflow import printprogress
N = 100
pbar = printprogress(N)
for _ in range(N):
	# do_something()
	pbar()

There is also a conviniant way to use multiprocessing in Python. You wish to provide a function name, iterable inputs and shared inputs. Then ask to run the function over the iterable inputs using multiprcessing. Then The multiprocessor is for you. The following is a cross-correlation of two masked verctors using a given ststistical function::

from lognflow import multiprocessor

def masked_cross_correlation(inputs_to_iter_sliced, inputs_to_share):
	""" Calculate the correlation of two masked vectors, then use a given
		statsitcal function to get the stat of the correlation.
	"""
    vec1, vec2 = inputs_to_iter_sliced
    mask, statistics_func = inputs_to_share
    vec1 = vec1[mask==1]
    vec2 = vec2[mask==1]
    vec1 -= vec1.mean()
    vec1_std = vec1.std()
    if vec1_std > 0:
        vec1 /= vec1_std
    vec2 -= vec2.mean()
    vec2_std = vec2.std()
    if vec2_std > 0:
        vec2 /= vec2_std
    correlation = vec1 * vec2
    to_return = statistics_func(correlation)
    return(to_return)
    

if __name__ == '__main__':
    data_shape = (1000, 2000)
    data1 = np.random.randn(*data_shape)
    data2 = 2 + 5 * np.random.randn(*data_shape)
    mask = (2*np.random.rand(data_shape[1])).astype('int')
    statistics_func = np.median
    
    inputs_to_iter = (data1, data2)
    inputs_to_share = (mask, statistics_func)
    ccorr = multiprocessor(masked_cross_correlation, 
    					   inputs_to_iter, 
    					   inputs_to_share)
    print(f'ccorr: {ccorr}')

In this package we use a folder on the HDD to generate files and folders in typical formats such as numpy npy and npz, png, ... to log. A log viewer is also availble to turn an already logged flow into variables. Obviously, it will read the folders and map them for you, which is something you could spend hours to do by yourself. Also there is the nicest progress bar, that you can easily understand and use or implement yourself when you have the time.

Looking at most logging packages online, you see that you need to spend a lot of time learning how to use them and realizing how they work. Especially when you have to deal with http servers and ... which will be a huge pain when working for companies who have their own HPC.

This is why lognflow is handy and straight forward.

Many tests are avialable in the tests directory.

• Free software: GNU General Public License v3
• Documentation: https://lognflow.readthedocs.io.

Features

• lognflow puts all the logs into a directory on your pc
• lognflow makes it easy to log text or simple plots.
• logviewer makes it easy to load variables or directories
• printprogress is one of the best progress bars in Python.

Credits ^^^^^^^^

This package was created with Cookiecutter_ and the audreyr/cookiecutter-pypackage_ project template.

.. _Cookiecutter: https://github.com/audreyr/cookiecutter .. _audreyr/cookiecutter-pypackage: https://github.com/audreyr/cookiecutter-pypackage

History

0.1.0 (2022-11-16)

• First release on PyPI.

0.3.0 (2022-12-19)

• Very consistent and easy-to-handle interface

0.3.2 (2022-12-27)

• log your dictionary.

0.5.2 (2023-01-11)

• logger call takes txt only for the main_log
• main_log name can have no time_stamp
• All logs can be force_flush = True.
• del is well implemented.

0.5.3 (2023-03-19)

• use logger.log_single('aName', aDict) to save a dictionary using numpy.savez
• time_in_file_name has been changed to time_tag, sorry there! but early stages.
• time_tag is True in the constructor, but could always be set to False
• All tests passed!

0.6.0 (2023-03-22)

• This is a stable release
• All text files are handled by the with statement
• Renaming bug is fixed
• all tests run properly.
• lognflow has all that logviewer has. We will check if dir exists at every use

0.6.1 (2023-03-22)

• rename had a bug that is fixed

0.6.2 (2023-03-25)

• made it possible to flush_all()
• We support Python 3.7+ because of dataclasses
• printprogress now can disable printing anything and return ETA at the call

0.6.3 (2023-04-01)

• lognflow class does all logviewer does. Maybe it is time to remove logviewer.

0.6.4 (2023-04-06)

• Better documentation and examples for readme
• get_var is added to lognflow to get buffered variables logged by log_var

0.6.5 (2023-04-26)

• Fixed a bug in the docs to allow sphinx compile it.
• log_var will log only the valid time stamps.
• added end keyword argument to log_text

0.6.6 (2023-04-27)

• Better documentation
• added tifffile imread to logviewer and imwrite to lognflow

0.6.7 (2023-04-27)

• A bug in tifffile support was fixed

0.6.8 (2023-05-04)

• Fixing readme for PyPI.
• removed marker from log_plot. user marker and linestyle keyword arguments.
• printprogress returns proper ETA every time if print_function is set to None::

0.7.0 (2023-05-15)

• logviewer returns data by log_sigle if the full name is mentioned.

0.7.1 (2023-05-22)

• printprogress supports lognflow.
• bugs fixed in lognflow.
• For now I guess lognflow and logviewer could be separate.

0.7.2 (2023-05-25)

• bug fixed in logviewer
• text_to_object added to logviewer to read dict or list logged via log_single
• test pass for logviewer including the test for text_to_object

0.7.3 (2023-06-01)

• bug fixed in logviewer in the use of suffix in get_stack_of_files
• log_imshow takes colorbar and remove_axis_ticks flags.
• every lognflow instance has a logviewer pointing to its log_dir called logged.

0.7.4 (2023-06-26)

• critical bug fixed in log_imshow

0.7.5 (2023-06-27)

• Added complex numbers to log_imshow

0.7.6 (2023-07-17)

• printprogress can handle up to 99 days
• log_text takes any save_as
• If variable name has escape key is alright
• If variable name is splitable, we replace them with _

0.8.0 (2023-07-25)

• logger.save and savez are set to be identical to log_single.
• logged.load is set to be identical to get_single.
• utils.py is added to contain all misc functions.
• replace_all added to utils

0.8.1 (2023-07-26)

• a bug fixed in log_var

0.8.2 (2023-08-02)

• the word save_as is now replaced with suffix as is in pathlib
• all loggers can take the suffix as the extension in the parameter_name

0.8.3 (2023-08-02)

• critical bug fixed in log_var to support v0.8.2

0.8.4 (2023-08-03)

• variable names that are pecular will always be fixed first.
• suffix can be read form the file name.
• time_tag will always accompany file name unless stated otherwised.

0.8.5 (2023-08-04)

• Some functions can go to utils and be mentioned in the init
• a bug was fixed in printprogress.

0.8.6 (2023-08-04)

• plt_utils was not added tp 0.8.5

0.9.0 (2023-08-09)

• copy() is now possible from a file or a variable name into another
• default suffix in get_flist is *
• logviewer.get_stack_of_files is only useful for reading data.
• more tests are added.
• moved multichannel_to_frame to utils

0.9.1 (2023-08-25)

• bug removed from plt_utils numbers_as_images_4D.
• bug removed from printprogress when number of steps is very small.

0.10.0 (2023-09-01)

• I added multiprocessor to lognflow
• bug fixed in logviewer

0.10.1 (2023-09-12)

• multi_channel_by_subplots bug fixed for non-square shape
• default colormap is viridis everywhere
• multiprocessor heavily debugged and made a lot easier to use
• better tests added for multiprocessor