halib 0.2.63

Small library for common tasks

📦 Helper Package for Coding and Automation Changelog

v0.2.x (Experiment & Core Updates)

v0.2.63

• ✨ New Feature: halib.utils.txt_merge - a versatile utility for merging multiple text files into one, supporting both explicit file lists and glob patterns with exclusion options, all configurable via YAML (especially useful for bypassing file upload limits in AI chat tools like Perplexity, Claude, or ChatGPT.).

v0.2.62

• ✨ New Feature: halib.utils.md_op and halib.utils.pdcite_op - added support for processing markdown files, and .tex files generated by pandoc for \\citeproc corrections.

v0.2.60

• ✨ New Feature: halib.utils.csv_ops - add arrowup/down to header (e.g metric); also add latex code to auto hightlight (by font style and color italic/bold/underline/color for columns with arrow

• ✨ New Feature: Added halib.utils.tex_op for handling LaTeX operations; currently support flattening nested LaTeX structures into a single line (with all resources files like images, csv, etc. copied to a specified output directory), that can be used as standalone LaTeX code for building pdf using Tex Studio or Overleaf.

v0.2.55

• 🚀 Improvement:: enhance zprofiler allow to inject pre-calculated time for a context (ctx) or a step.

v0.2.49

• ✨ New Feature:: introduce utils/csv_op.py = a YAML-driven declarative CSV transformation pipeline supporting operations like filter, rename, convert, limit, sort, and drop with flexible output path handling.

v0.2.43

• 🚀 Improvement:: enhance log_func decorator with additional logging options and better formatting (in Rich table).

• ✨ New Feature:: introduce utils.PlotlyUtils with parallel coordinates plot and data table support

• 🚀 Improvement:: move wandb_op.py to utils and add scripts folder

• ✨ New Feature:: add common.common.log_func as decorator to log function entry, exit, with execution time and arguments.

• 🚀 Improvement:: enhance utils.dict.DictUtils with deep_exclude/include and prune function

v0.2.28

• ✨ New Feature: Implement utils.slack.SlackUtils class for managing Slack channel message deletion

• ✨ New Feature: Added utils.dict.DictUtils for advanced dictionary manipulations (merging, filtering, transforming).

• ✨ New Feature: Added common.common.pprint_stack_trace to print stack traces with optional custom messages and force stop capability.

• 🚀 Improvement: exp.perf.profiler - allow to export report dict as csv files for further analysis

v0.2.19

• ✨ New Feature: Added exp.core.param_gen to facilitate fast generation of parameter combination sweeps (grid search) using YAML configurations.

v0.2.17

• 🚀 Improvement: Updated exp.perf.profiler with an enabled flag for dynamic toggling.
• 🚀 Improvement: Added a measure context manager to simplify performance measuring of code blocks.

v0.2.13

• ♻️ Refactor: Major reorganization of packages. Renamed research package to exp (Experiment Management).
• 🚀 Improvement: Updated exp/perfcalc.py to allow saving computed performance metrics to CSV without explicitly calling calc_perfs.

v0.2.1

• ✨ New Feature: Added eval_exp method to exp/base_exp for running evaluations (e.g., model testing) after experiments conclude.

---

v0.1.9x (Visualization & Generators)

v0.1.99

• ✨ New Feature: Added gen_ipynb_name to filetype/ipynb. Generates filenames based on the current notebook name with optional timestamps.

v0.1.96

• ✨ New Feature: Added PlotHelper class in research/plot for plotting training history and image grids (dataset samples or model outputs).

v0.1.91

• ✨ New Feature: Added ParamGen class to research/param_gen for parsing YAML files into parameter lists for hyperparameter searches.

v0.1.90

• ✨ New Feature: Added zProfiler class to research/profiler for measuring context/step execution time, supporting dynamic color scales in plots.

---

v0.1.5x - v0.1.7x (Infrastructure & Utilities)

v0.1.77

• ✨ New Feature: Added BaseExp class in research/base_exp to handle common experiment tasks (performance calculation, result saving).

v0.1.67

• 🔧 Maintenance: Switched to uv for virtual environment management.
• 🚀 Improvement: Updated research/perfcalc to support both torchmetrics and custom metrics.

v0.1.61

• ✨ New Feature: Added VideoUtils (util/video) for common video handling tasks.
• ✨ New Feature: Added GPUMonitor (util/gpu_mon) for tracking GPU usage and performance.

v0.1.59

• 🔨 Architecture: Added util/perfcalc abstract base class. This requires implementation of specific performance calculation logic.

v0.1.55

• ✨ New Feature: Added util/dataclass_util for dynamic creation of dataclass objects from dictionaries or YAML (supports nested structures).

v0.1.52

• ✨ New Feature: Added research/perftb module for managing experiment performance tables (filtering by dataset, metric, etc.).

v0.1.50

• ✨ New Feature: Added pprint_local_path to print clickable file URIs for local paths.
• ✨ New Feature: Added research package containing benchquery for dataframe benchmarking.
• ✨ New Feature: Added wandb module for offline syncing and batch clearing of Weights & Biases runs.

---

v0.1.4x (Display & formatting)

v0.1.47

• ✨ New Feature: Added pprint_box to print objects or strings inside a decorative box frame.

v0.1.46

• 🐛 Fix: Filtered UserWarning: Unable to import Axes3D.
• 🚀 Improvement: Added auto_wrap_text to fn_display_df to prevent long text overflow in tables.

v0.1.42

• ✨ New Feature: Added rich_color.py wrapper for basic color lists.

v0.1.41

• ✨ New Feature: Added rich_color.py to support rich color information (palettes, strings) using the rich library.

v0.1.40

• 🚀 Improvement: Updated csvfile.py to use itables and pygwalker for interactive dataframe display in Jupyter notebooks.

---

v0.1.3x (Data & Loading)

v0.1.38

• ✨ New Feature: Added torchloader.py to search for optimal DataLoader configurations (num_workers, batch_size, pin_memory).

v0.1.37

• ✨ New Feature: Added dataset.py for splitting classification datasets into train/val/test sets.

v0.1.33

• ✨ New Feature: Added plot.py for plotting Deep Learning training history (accuracy/loss) using seaborn and matplotlib.

---

v0.1.0 - v0.1.2x (Early Utilities)

v0.1.29

• 🐛 Fix: Pinned kaleido==0.1.* for tele_noti as version 0.2.* caused image generation hangs.

v0.1.24

• ♻️ Refactor: Renamed sys module to system to avoid conflicts with Python's built-in sys.
• ✨ New Feature: Added tele_noti module for Telegram notifications regarding training progress.

v0.1.22

• ✨ New Feature: Added cuda.py to check CUDA availability for both PyTorch and TensorFlow.

v0.1.21

• ✨ New Feature: Added YAML inheritance and overriding support using networkx and omegaconf.

v0.1.15

• ✨ New Feature: Added common logging library and @console_log decorator for function tracing.

v0.1.10

• 🐛 Fix: Fixed typo is_exit -> is_exist in filesys.
• 🚀 Improvement: Updated gdrive to support uploading to specific folders and returning direct shareable links.

v0.1.9

• 🔧 Maintenance: Added requirements.txt.

v0.1.6 - v0.1.8

• 🚀 Performance: Optimized table insertion by using an in-memory row_pool_dict before committing to the DataFrame.
• ✨ New Feature: Added DFCreator for manipulating DataFrames (create, insert, display, save).

v0.1.4 - v0.1.5

• ✨ New Feature: Added cmd module.
• ✨ New Feature: Support for creating Bitbucket Projects from templates.

v0.1.2

• ✨ New Feature: Added support for uploading local files to Google Drive.