CoreUtilities 0.0.81

A comprehensive collection of Python utility functions for data science, file operations, and general-purpose programming

CoreUtils-Python

A comprehensive collection of Python utility functions and modules for data science, file operations, serialization, encryption, and general-purpose programming tasks.

Table of Contents

• Overview
• Installation
• Quick Start
• Module Documentation
  • Core Utilities
  • Data Processing
  • Security & Encryption
  • File Operations
  • Testing
• Running Tests
• Requirements
• Contributing
• License

Overview

CoreUtils-Python is a modular collection of well-documented, tested utility functions designed to streamline common programming tasks across data science, system operations, and application development.

Key Features:

• 🔧 Comprehensive Utilities - Functions, lists, strings, numbers, dictionaries
• 📊 Data Processing - pandas, NumPy, Polars, PyArrow integration
• 🔒 Security - Encryption, signing, secure serialization, CSV-compatible integrity
• 🧪 Well Tested - 418+ unit tests with pytest
• 📝 Documented - NumPy-style docstrings throughout
• ⚡ Performance - Optimized for large-scale data operations

Installation

Basic Installation

# Clone the repository
git clone https://github.com/Ruppert20/CoreUtils-Python.git
cd CoreUtils-Python

# Install dependencies
pip install -r requirements.txt

Requirements

• Python 3.12+
• numpy >= 2.3.0
• pandas >= 2.3.0
• PyYAML >= 6.0.2
• cryptography >= 45.0.7
• tqdm >= 4.67.0

Optional Dependencies

# Install with optional dependencies
pip install "CoreUtilities[optional]"

# Install with development tools
pip install "CoreUtilities[dev]"

• [dev]: black >= 24.0.0, mypy >= 1.8.0, flake8 >= 7.0.0, pytest >= 7.4.0, pytest-cov >= 4.1.0
• [optional]: polars >= 1.33.0, pyarrow >= 21.0.0

Quick Start

# Import utilities
from src.generics import notnull, coalesce
from src.lists import chunk_list, flatten_list
from src.strings import convert_identifier_case
from src.numerics import extract_num, isfloat
from src.signature import SignedFile
from datetime import datetime

# Use null checking
if notnull(value):
    process(value)

# Coalesce values
result = coalesce(None, '', default_value)

# Chunk data for batch processing
for chunk in chunk_list(large_list, 100):
    process_batch(chunk)

# Convert naming conventions
camel = convert_identifier_case('user_name', 'camelCase')

# Write signed file with header metadata
header = {"version": "1.0", "created": datetime.now(), "author": "alice"}
SignedFile.write("data.bin", {"key": "value"}, header=header)

# Write CSV with integrity signature (pandas-compatible)
csv_data = b"name,age\nAlice,30\nBob,25\n"
SignedFile.write("data.csv", csv_data, signature_as_comment=True)

# Read back with verification and header
data, meta = SignedFile.read("data.bin", return_header=True)
print(f"Created by {meta['author']} on {meta['created']}")

Module Documentation

Core Utilities

generics.py

Generic utility functions for null handling and object operations.

Key Functions:

• notnull(v) - Comprehensive null checking (None, empty containers, pd.NA, np.nan)
• isnull(v) - Inverse of notnull
• coalesce(*values) - Return first non-null value
• get_name(obj) - Extract object name

📝 Code | 🧪 Tests | 📖 Documentation

---

functions.py

Function utilities including dynamic loading, introspection, and debugging.

Key Functions:

• get_func(func_path) - Dynamically load functions from string paths
• filter_kwargs(func, kwargs) - Filter kwargs to match function parameters
• get_function_signature(func) - Extract comprehensive function metadata
• inspect_class(cls) - Extract class properties and methods
• is_pickleable(obj) - Check if object can be pickled

📝 Code | 🧪 Tests | 📖 Documentation

---

lists.py

List manipulation utilities for chunking, intersection, and flattening.

Key Functions:

• convert_list_to_string(lst, encapsulate=False) - Convert list to comma-separated string
• chunk_list(lst, n) - Split list into equal-sized chunks
• list_intersection(lst1, lst2) - Find common elements preserving order
• flatten_list(nested) - Recursively flatten nested lists

📝 Code | 🧪 Tests | 📖 Documentation

---

strings.py

String manipulation including case conversion, cleaning, and parsing.

Key Functions:

• remove_illegal_characters(s, case='snake_case') - Clean strings for identifiers
• convert_identifier_case(id, target_format) - Convert between naming conventions
• snake_to_camel_case(s) - Convert snake_case to camelCase
• camel_to_snake_case(s) - Convert camelCase to snake_case
• get_file_name_components(path) - Parse file paths into components
• tokenize_id(id_str, token_index) - Split and extract tokens from IDs

📝 Code | 🧪 Tests | 📖 Documentation

---

numerics.py

Numerical operations, extraction, and validation.

Key Functions:

• extract_num(input_str, return_pos=0) - Extract numbers from strings
• isfloat(value) - Check if value can be converted to float
• convert_to_comma_seperated_integer_list(val) - Convert to comma-separated integers

📝 Code | 🧪 Tests | 📖 Documentation

---

dictionaries.py

Dictionary utilities for pandas aggregation operations.

Key Functions:

• create_aggregation_dict(col_action_dict, start_col, end_col) - Create pandas groupby aggregation dictionaries

📝 Code | 🧪 Tests | 📖 Documentation

---

git.py

Git repository metadata extraction.

Key Functions:

• get_git_metadata() - Extract comprehensive git repository information

📝 Code | 📖 Documentation

---

Data Processing

core_types.py

Cross-library type classification and detection system.

Key Features:

• CoreDataType enum - Universal type classification
• Type detection from objects and strings
• Support for pandas, NumPy, Polars, PyArrow
• String representation parsing (JSON, XML, UUID, dates)

📝 Code | 📖 Documentation

---

iterables.py

Memory profiling and object analysis utilities.

Key Functions:

• deep_stats(obj) - Calculate deep memory size with cycle detection
• find_large_objects(obj, threshold_kb) - Identify memory-intensive objects

📝 Code | 📖 Documentation

---

serialization.py

Extended serialization with multi-format support (JSON, YAML, CBOR, Pickle).

Key Features:

• XSer class - Destination-aware serialization
• Automatic fallback chain: Structured → CBOR → Pickle
• NumPy array support
• HDF5 and Parquet metadata support

📝 Code | 📖 Documentation

---

enhanced_logging.py

Advanced logging with emoji support, progress bars, and structured output.

Key Features:

• Enhanced logger with emoji integration
• Progress bar support
• Structured logging for metrics
• Context managers for scoped logging

📝 Code | 📖 Documentation

---

parrallelization.py

Parallel processing utilities with comprehensive error handling.

Key Features:

• ParallelProcessor class
• Support for serial, thread-based, and process-based execution
• Metrics collection and reporting
• Integration with enhanced logging

📝 Code | 📖 Documentation

---

Security & Encryption

encrypt.py

Encryption utilities using Fernet symmetric encryption.

Key Features:

• Encryptor class for data encryption/decryption
• CryptoYAML for encrypted YAML configuration files
• Key generation and management

📝 Code | 🧪 Tests | 📖 Documentation

---

signature.py

Atomic file writing with cryptographic integrity verification, encryption, and metadata support.

Key Features:

• SignedFile class for signed file operations
• SHA-256/HMAC-SHA256 signatures with integrity verification
• Optional Fernet encryption with authenticated HMAC
• Python object serialization (via XSer) - auto-serializes dicts, lists, numpy, datetime
• Optional header metadata - Store version info, timestamps, and structured metadata
• CSV-compatible commented signatures - Write # comment signatures for pandas/Excel compatibility
• Atomic writes with platform-independent fsync
• Chunked reading for large files

📝 Code | 🧪 Tests | 📖 Documentation

---

File Operations

search.py

Flexible file search utilities with pattern matching and filtering.

Key Features:

• FileSearcher class for advanced file searching
• Pattern matching with regex support
• File type filtering and exclusion patterns
• Recursive and non-recursive search modes

📝 Code | 🧪 Tests | 📖 Documentation

---

Testing

debugging.py

Testing utilities for random data generation.

Key Functions:

• generate_random_sequence(dtype, n, percent_null, seed) - Generate deterministic test data
• Random generators for all common data types (TEXT, UUID, INTEGER, FLOAT, DATE, JSON, XML, etc.)
• debug_print(*args) - Print debug output with visual separators

📝 Code | 🧪 Tests | 📖 Documentation

---

Running Tests

All tests use pytest and follow the test_*.py naming convention.

Run All Tests

cd UNIT_TESTS
python run_all_tests.py

Run with Verbose Output

python run_all_tests.py -v

Run with Coverage

python run_all_tests.py --coverage

Run Specific Tests

# Run tests matching a pattern
python run_all_tests.py -k test_generics

# Run a specific test file
pytest test_functions.py -v

# Run a specific test class
pytest test_functions.py::TestGetFunc -v

# Run a specific test method
pytest test_functions.py::TestGetFunc::test_get_builtin_function -v

Test Statistics

• Total Tests: 223+
• Coverage: Comprehensive coverage of public APIs
• Frameworks: pytest (supports both pytest and unittest styles)
• Status: ✅ All tests passing

📖 View Test Documentation | 📊 View Test Summary

---

Requirements

Core Dependencies

numpy>=2.3.2          # Numerical computing
pandas>=2.2.3         # Data manipulation

Serialization

cbor2>=5.7.0          # CBOR encoding
PyYAML>=6.0.2         # YAML support

Security

cryptography>=45.0.7  # Encryption and signing

Testing

pytest>=8.4.2         # Test framework
pytest-cov>=4.1.0     # Coverage plugin

📖 View Full Requirements

---

Project Structure

CoreUtils-Python/
├── src/                          # Source modules
│   ├── core_types.py            # Type classification system
│   ├── debugging.py             # Testing and debugging utilities
│   ├── dictionaries.py          # Dictionary operations
│   ├── encrypt.py               # Encryption utilities
│   ├── encrypted_signature.py  # Combined encryption + signing
│   ├── enhanced_logging.py     # Advanced logging
│   ├── functions.py            # Function utilities
│   ├── generics.py             # Generic utilities
│   ├── git.py                  # Git metadata
│   ├── iterables.py            # Memory profiling
│   ├── lists.py                # List operations
│   ├── numerics.py             # Numerical utilities
│   ├── parrallelization.py     # Parallel processing
│   ├── search.py               # Search utilities
│   ├── serialization.py        # Extended serialization
│   ├── signature.py            # File signing
│   └── strings.py              # String manipulation
│
├── UNIT_TESTS/                  # Test suite
│   ├── test_*.py               # Test modules (223+ tests)
│   ├── run_all_tests.py        # Test runner
│   ├── README.md               # Test documentation
│   └── TEST_SUMMARY.md         # Test results summary
│
├── requirements.txt             # Project dependencies
└── README.md                    # This file

---

Contributing

Contributions are welcome! Please follow these guidelines:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Write tests for new functionality
4. Ensure all tests pass (python run_all_tests.py)
5. Follow existing code style (NumPy-style docstrings)
6. Commit changes (git commit -m 'Add amazing feature')
7. Push to branch (git push origin feature/amazing-feature)
8. Open a Pull Request

Code Style

• NumPy-style docstrings for all functions and classes
• Type hints where appropriate
• Comprehensive test coverage
• Clear, descriptive variable names

Releasing a New Version

To publish a new release to PyPI and GitHub:

1. Ensure all tests pass on the main branch
2. Create a git tag with the v prefix following semantic versioning:

   git tag v0.1.0        # stable release
   git tag v0.1.0a1      # alpha release
   git tag v0.1.0b1      # beta release
   git tag v0.1.0rc1     # release candidate
   

3. Push the tag to trigger the release workflow:

   git push origin v0.1.0
   

4. The release workflow will automatically:
   • Run tests against Python 3.12, 3.13, and 3.14
   • Build the package
   • Publish to PyPI
   • Create a GitHub release with release notes

---

License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Author

@Ruppert20

---

AI Authorship Disclaimer

This package was developed with the assistance of LLM-based coding tools (Claude Code by Anthropic). AI tools were used for the following activities:

• Code authorship - Implementation of utilities, functions, and classes
• Test development - Creation of comprehensive unit tests
• Documentation - Generation of NumPy-style docstrings and README content
• Code review - Identification of bugs, edge cases, and improvements

Users should evaluate the code for their specific use cases and report any issues through the GitHub issue tracker.

---

Acknowledgments

• Built with modern Python 3.12+
• Integrates with pandas, NumPy, Polars, and PyArrow
• Inspired by the need for clean, reusable utility functions
• Comprehensive testing ensures reliability
• Developed with assistance from Claude Code (Anthropic)

---

Quick Links

• 📖 Full Documentation
• 🧪 Test Suite
• 📊 Test Results
• 📋 Requirements
• 🐛 Issue Tracker

---

Made with ❤️ for the Python community