emp-utl 0.1.0

Utility modules for reusable infrastructure components in the Enterprise Management Program (EMP)

🧰 EMP – Utility Library (emp-utl)

The library emp-utl is the official reusable utility package of the Enterprise Management Program (EMP).

It provides standardized, production-ready infrastructure components for EMP microservices and serverless functions — bundled into a single installable Python package.

The library focuses on:

• 📝 Structured logging
• ⚙️ Centralized configuration loading
• 🗄️ MySQL schema & data orchestration

📦 Installation

pip3 install emp-utl

---

📝 Structured Logger

The logger provides a unified, environment-aware logging abstraction for EMP services.

It automatically adapts its behavior based on the ENVIRONMENT variable:

• dev → JSON file logging (local)
• int / stg / prd → Kafka-based centralized logging

Key Characteristics:

• 📦 Structured JSON format
• 🌍 Environment-aware behavior
• 📡 Kafka publishing in higher environments
• 🧾 Logback-compatible structure (ELK-ready)

Usage:

from emp_utl import setup_logger

logger = setup_logger(service="Customer", module="Serverless")
logger.info("Service initialized successfully")

Logs are automatically formatted and routed according to the deployment environment.

---

⚙️ Configuration Loader

The configuration loader enables secure and dynamic configuration resolution across environments.

Resolution Strategy

1. Primary Source – Spring Cloud Config Server (CNF-S)

   1. Authenticated access via EMP_CONFIG_USERNAME and EMP_ENCRYPT_KEY
   2. Nested configuration key resolution supported

2. Fallback – Local OS Environment

   1. Automatically triggered if CNF-S is unavailable
   2. Uses os.getenv for required keys

This ensures service startup even if the configuration server is unreachable.

Usage:

from emp_utl import load_cnf

keys = ["DB_HOST", "DB_PORT", "DB_USER", "DB_PASSWORD"]

env_vars = load_cnf(
    service = "utility-service",
    keys = keys,
    logger = logger
)

Returned values are provided as a dictionary.

---

🗄️ MySQL Loader

The MySQL loader orchestrates schema setup and DataFrame-based data loading.

It performs:

• MySQL connection initialization
• SQL script execution (./sql directory)
• Foreign key handling
• Bulk insertion of pandas DataFrames
• Post-insert null normalization

Designed for:

• Local development
• Serverless execution
• Automated schema bootstrapping

Usage:

from emp_utl import load_mysql

load_mysql(
    env_vars = env_vars,
    schema = env_vars["EMP_SCHEMA_UTILITY"],
    df_dict = df_dict,
    logger = logger
)

📌 df_dict must map table names to pandas DataFrames. SQL schema files must exist in ./sql.

---

✅ Quality Standards

• Unit test coverage ≥ 90%
• Explicit public API
• Environment-safe behavior
• Production-ready structured logging
• Designed for CI/CD publishing via GitHub Actions

🎯 Purpose

emp-utl standardizes infrastructure behavior across the EMP microservice ecosystem.

It ensures:

• Consistency
• Observability
• Secure configuration handling
• Reusable database orchestration

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[0.1.0] - 2026-03-06

Structured Logging

• Environment-aware structured logging via setup_logger
• JSON-formatted logs compatible with ELK Stack ingestion
• Local file logging for dev environment
• Kafka-based centralized logging for int, stg, and prd environments
• Custom Kafka logging handler with delivery reporting

Configuration Loader

• Dynamic configuration loading via load_cnf
• Primary configuration source: Spring Cloud Config Server (CNF-S)
• Secure authentication using EMP_CONFIG_USERNAME and EMP_ENCRYPT_KEY
• Automatic fallback to local OS environment variables if CNF-S is unavailable
• Nested configuration key resolution support

MySQL Utilities

• MySQL orchestration via load_mysql
• Automatic schema setup through SQL script execution
• Support for custom SQL DELIMITER parsing
• Foreign key management utilities (disable_fk_check, enable_fk_check)
• Bulk DataFrame insertion support
• Automatic placeholder normalization for NULL values
• Column-type-aware NULL restoration logic

Testing & Quality

• Complete unit test coverage (100%)
• Comprehensive error handling and logging
• CI-ready architecture for GitHub Actions publishing