hel-elmuro 0.3.0

Helpers for Cosmos DB and Azure Storage operations

hel-elmuro

Python helpers for the ElMuro transportation analytics platform.

Features

• Persist pandas DataFrames into Azure Cosmos DB containers.
• Perform common Azure Blob Storage operations (count, copy, delete, move).

Installation

pip install hel-elmuro

Quick Start

from hel_elmuro import CosmosManager, FileManager

cosmos = CosmosManager(
    endpoint="https://<account>.documents.azure.com:443/",
    key="<cosmos-key>",
    database_name="elmuro",
    container_name="drivers",
)
inserted = cosmos.save_dataframe(df, partition_key_column="driver_id")

files = FileManager(
    connection_string="<storage-connection-string>",
    container_name="dap0gescab0con",
)
files.move("bronze/file.json", "silver/file.json")

File Organization

• Is not defined by the workspace convention.
• The organization is defined by the build process defined in the /pyproject.toml file; by default the build artifacts source code is collected from src/hel_elmuro.
• All library modules should live under src/hel_elmuro so editable installs and wheels include the package correctly.

Development

• All examples assume Visual Studio Code and a power shell terminal environment.
• Install the package in editable mode for iterative work: pip3 install -e "hel_elmuro_pkg[dev]".
• Development dependencies (pytest, black, flake8) are defined in pyproject.toml under the dev extra.

Testing

• Based on the pytest python module. defined in dev depedencies in the .toml file of the proyect. Test is in the tests folder and it will execute every function declared in that files. Organization of the files is something reletaed with the workspace conventions to organize per file the different unit testing of specific objects. (Modules/class in this cases).

Run the suite from the project repository root so editable installs resolve correctly:

python3 -m pytest

When you need to run tests from the package directory with explicit logging parameters:

cd hel_elmuro_pkg
python3 -m pytest --log-cli-level=INFO -s

Release (PyPI)

These steps publish the hel-elmuro package only; other projects in the workspace have their own workflows.

1. Bump the project.version value in hel_elmuro_pkg/pyproject.toml to the desired semantic version (MAJOR.MINOR.PATCH), and update any information there.
2. Ensure the workspace has build and twine installed (pip3 install --upgrade build twine).
3. Remember to solve pendent action in the git to create the tag properly and get the info from the git properly.
4. From the repository root execute the hel-elmuro release helper at hel_elmuro_pkg/scripts/publish_release.py (tests are executed again as part of this step):

   python3 hel_elmuro_pkg/scripts/publish_release.py
   

   • The script reads the package name and version from pyproject.toml.
   • It verifies the working tree is clean, checks the declared version is newer than the latest git tag and PyPI release, and runs python -m pytest before proceeding.
   • Packaging uses python -m build to create both the source archive (dist/<name>-<version>.tar.gz) and wheel (dist/<name>-<version>-py3-none-any.whl).
   • Publishing uses python -m twine upload dist/*, which prompts for the PyPI API token and pushes the artifacts.
   • A git tag named hel-elmuro-v<version> is created and pushed so releases stay scoped to the package.
   • Build metadata (hel_elmuro_pkg/dist/ and hel_elmuro_pkg/src/hel_elmuro.egg-info/) is regenerated every release and stays ignored by Git via .gitignore.