shared-libs-py 1.0.2

Shared Python packages

Data engineering shared python libs

This project is used by the data engineering team to share common libraries between multiple projects

Table of Contents

1. Authors
2. Project Structure
3. Getting Started
4. Usage
5. Development Setup
6. Database Migrations
7. Documentation

1. Authors

Principal maintainers: data.engineering

Team: Data Engineering

2. Project Structure

shared-libs-py/

    ├── .tests/         <- local notebooks, data, etc.
    |
    ├── .vscode/        <- vscode config files
    |
    ├── devops/
    |   ├── ci
    |   |   └── build.yaml
    |   |   └── pr.yaml
    |   ├── cd
    |   |   └── deployment.yaml
    |
    ├── src/
    |   └──shared/
    │       ├── azure/           <- several azure libaries
    │       ├── db/              <- various config files related to project
    │       └── utils/           <- utility modules
    │
    ├── tests/
    │   ├── unit/
    │
    ├── Dockerfile          <- basic Dockerfile template
    ├── .dockerignore
    │
    ├── .gitignore
    ├── Makefile            <- environement creation and basic checks
    ├── pyproject.toml      <- basic standards config
    ├── README.md           <- project README template
    └── uv.lock             <- uv dependencies locker

3. Getting Started

Prerequisites

• Make
• Python 3.11
• Docker & Docker Compose
• Liquibase (for database migrations)

Installation

Clone this repository

git clone <repository-url>
cd shared-libs-py

Set create env and install libs

make env

Docker Setup

Start the services using Docker Compose:

docker-compose up -d

4. Usage

Below you will see the instructions to install the package under two different methods depending on your project's package manager.

4.1 Using UV

1. Go to: https://dev.azure.com/oneocean/
2. Click on "User Settings" -> "Personal access tokens".
3. Use "+ New Token" button to generate new token. It will only require read access to "Packaging"
4. Create 2 environment variables:

export UV_INDEX_ONEOCEAN_FEED_USERNAME=pylib
export UV_INDEX_ONEOCEAN_FEED_PASSWORD=[YOUR_TOKEN_HERE]

5. Add the package to pyproject.toml by using the following command:

uv add shared-libs-py --index oneocean-feed="https://pkgs.dev.azure.com/oneocean/Data/_packaging/OneOcean/pypi/simple/"

4.1 Using pip

1. Go to: https://dev.azure.com/oneocean/
2. Click on "User Settings" -> "Personal access tokens".
3. Use "+ New Token" button to generate new token. It will only require read access to "Packaging"
4. Create a pip.config file in the following path ~/.pip/pip.conf.
5. Add the index into the pip.conf by using the following configuration in it.

[global]
index-url = https://pypi.org/simple
extra-index-url = https://pylib:[YOUR_TOKEN_HERE]@pkgs.dev.azure.com/oneocean/Data/_packaging/OneOcean/pypi/simple/

5. Development Setup

PR checks

This project uses the following linters and formatters:

• Black - Code formatting (docs)
• isort - Import sorting (docs)
• Ruff - Fast Python linter (docs)
• Pylint - Static code analysis (docs)
• Bandit - Security-focused linter (docs)

To run all checks:

make checks

To automatically fix issues where possible:

make fix

CI

• Pr: Once a PR is created, checks like black, isort, ruff, etc. are automatically executed. The PR cannot be completed until at least 1 reviewer approves it.
• Build: Create a docker image and push it to acr. Cache is used to speed up the process. Once the image is created, dependency tracker is executed in order to scan for vulnerabilites and security issues.

CD

• Deployment: When the build is completed, the task will publish the python package into the company's atifacts feed.

6. Documentation