jragbeer-common 0.2.68

Package to create JRAGBEER Common Functions

jragbeer_common

Overview

A Python utility library containing common functions and tools used across various projects. This library provides reusable components for:

• Data processing and engineering
• Azure blob storage operations
• Dask distributed computing
• Ubuntu system operations

Features

• Data Engineering Utilities

  • DataFrame manipulation and cleaning
  • SQL database operations
  • Email notifications
  • Date/time processing
  • Logging configuration

• Azure Integration

  • Blob storage upload/download
  • Parquet file handling
  • Container management
  • Batch operations

• Dask Distributed Computing

  • Cluster deployment and management
  • Worker allocation
  • Task scheduling
  • Remote execution

• Ubuntu System Operations

  • Remote command execution
  • Process management
  • System monitoring
  • File operations

Installation

1. Install uv (recommended):

pip install uv

2. Create and activate a virtual environment:

uv venv
source .venv/bin/activate  # Linux/Mac
# or
.venv\Scripts\activate  # Windows

3. Install the package:

uv pip install jragbeer-common

Development Setup

1. Clone the repository:

git clone https://github.com/jragbeer/jragbeer_common.git
cd jragbeer_common

2. Create a virtual environment and install dependencies:

uv venv
source .venv/bin/activate
uv pip install -r requirements.txt

3. Install development dependencies:

uv pip install -e ".[dev]"

4. Install pre-commit hooks:

pre-commit install

Usage

from jragbeer_common import (
    jragbeer_common_data_eng,
    jragbeer_common_azure,
    jragbeer_common_dask,
    jragbeer_common_ubuntu
)

# Data Engineering
jragbeer_common_data_eng.parse_date_features(df)

# Azure Operations
jragbeer_common_azure.adls_upload_file("path/to/file", "blob_name")

# Dask Operations
jragbeer_common_dask.deploy_dask_home_setup()

# Ubuntu Operations
jragbeer_common_ubuntu.execute_cmd_ubuntu_sudo("command")

Environment Variables

The following environment variables are required:

# Azure Storage
adls_connection_string="your_connection_string"
adls_container_name="your_container"

# Database
local_db_username="username"
local_db_password="password"
local_db_address="address"
local_db_port="port"

# Cluster Configuration
cluster_server_1_address="address"
cluster_server_1_username="username"
cluster_server_1_password="password"

Building and Distribution

1. Build the package:

uv build

2. Install locally for testing:

uv pip install dist/jragbeer_common-0.2.0-py3-none-any.whl

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests and linting:

uv sync --extra dev
pre-commit run --all-files
uv run pytest

CI runs the same full suite on each push and pull request (see .github/workflows/test.yml), including tests marked integration (Docker Postgres, etc.). You need a working Docker daemon for a green run locally and in GitHub Actions (the runner provides Docker). Markers (unit, integration, contract) still classify tests in pyproject.toml; nothing is excluded by default.

Integration tests (Docker Postgres)

Database tests that open PostgreSQL live in test/db/test_jragbeer_common_db.py and are marked @pytest.mark.integration. They use the session-scoped container from test/fixtures/postgresql.py plus the docker Python client and psycopg2-binary. They run as part of the default uv run pytest (and in CI).

Optional environment variables (defaults in parentheses):

• TEST_POSTGRESQL_PORT — host port mapped to the container (54329)
• TEST_POSTGRESQL_USER, TEST_POSTGRESQL_PASSWORD, TEST_POSTGRESQL_DATABASE — bootstrap credentials (devuser / devpassword / postgres)
• TEST_POSTGRESQL_VERSION — image tag after postgres: (17.4)

Readiness is checked with TCP connections via psycopg2 from the host (same as SQLAlchemy), not docker exec psql, so you do not need the docker CLI or host postgresql-client for the wait loop.

Future work: tests that hit real services should live under the appropriate test/<subpackage>/ package, be marked @pytest.mark.integration, and document required environment variables in the test module docstring.

License

Copyright 2026 Julien Ragbeer

Licensed under the Apache License, Version 2.0