3phi-framework 0.2.5

Framework for 3phi project

3phi Framework

Utility classes for DB access, S3 interactions, and data processing via a DataExtractor.
Distributed privately via GitLab’s Package Registry (private PyPI).

Install name: 3phi-framework
Import package: threephi_framework

---

Features (MVP)

• DBConnector – simple connector + query helpers.
• S3Connector – upload/download and safe path helpers.
• DataExtractor – API to interact with the Data in S3.

This is an early version. API may evolve until v1.0. See “Versioning” below.

---

Requirements

• Python 3.9+
• Access to the GitLab Package Registry for this project or group.

---

Build a new version

Write & commit your code and then tag it:

git tag -a vx.x.x -m "vx.x.x"
git push --tags

Installation (GitLab private PyPI)

Create a Deploy Token with scope read_package_registry in this project.

pip install \
  --index-url "https://<DEPLOY_USER>:<DEPLOY_TOKEN>@gitlab.com/api/v4/projects/<PROJECT_ID>/packages/pypi/simple" \
  3phi-framework

Other ways to install

Replace placeholders like <PROJECT_ID>, , , and use the gitlab domain (gitlab.3pi-dev.io).

1. Quick one-liner (private project, for code completion etc.)

Use a Deploy Token (Settings → Repository → Deploy tokens) with read_package_registry, or a PAT with read_api.

# safer: export creds so they don't land in shell history
export GL_USER="gitlab+deploy-token-<id>"   # or your GitLab username if using a PAT
export GL_TOKEN="<token>"

pip install \
  --index-url "https://$GL_USER:$[e-mail]/api/v4/projects/<PROJECT_ID>/packages/pypi/simple" \
  --extra-index-url "https://pypi.org/simple" \
  3phi-framework==0.1.2

Package name to install: 3phi-framework Module you import: threephi_framework --extra-index-url https://pypi.org/simple lets pip resolve any third-party deps from PyPI.

2. Put it in requirements.txt (useful for docker image builds)

At the top of requirements.txt:

--index-url https://<USER>:<TOKEN>@gitlab.3pi-dev.io/api/v4/projects/<PROJECT_ID>/packages/pypi/simple
--extra-index-url https://pypi.org/simple
3phi-framework==0.1.2

Then install as you normally would.

3. Persist credentials in pip.conf (developer machines)

Linux/macOS: ~/.pip/pip.conf Windows: %APPDATA%\pip\pip.ini

[global]
index-url = https://<USER>:<TOKEN>@gitlab.3pi-dev.io/api/v4/projects/<PROJECT_ID>/packages/pypi/simple
extra-index-url = https://pypi.org/simple

Then you can just:

pip install 3phi-framework==0.1.2

---

Quickstart

The framework is set up for local development as well as for being used in a deployment. To set up your environment for local development, follow these steps:

Set up virtual environment

execute_data_app.sh expects a virtual environment to be set up under [.venv]. See the python docs on how to set it up.

Seed data

Obtain seed data for the database and the object storage and copy it to:

• 3_db_seed.sql: This should be a sql dump/snapshot of a working 3phi Database, PSQL will automatically seed the DB when it is created using docker compose.
• 3phi: This should be a copy of a bucket from a working object storage. It will be mounted in the minio object storage as a bucket.

Spin up DB and Object Storage

Navigate to docker and run docker compose up -d. This will bring up a local DB and a MinIO Object Storage seeded with the data you provided.

Run a data app locally

Use the utility script execute_data_app.sh and pass the data app name as an argument, e.g.:

./exectue_data_app.sh sm_classifier

In case the script is not executable, make it executable:

chmod +x execute_data_app.sh

The script will install the dependencies in requirements.txt in your virtual environment, set up environment variables as they are listed in .env and execute the data app as a python module.

Contibuting to 3phi Framework

The framework is currently not open-source, but the aim is to open-source it eventually.

Commit Message Guidelines

Commit Messages should follow the recommendations of Conventional Commits, supplemented by the Angular Convention. Every commit must contain type and description, while scope, body and footer are optional:

<type>(<scope>): <description>
<BLANK LINE>
<body>
<BLANK LINE>
<footer>

Type

type is mandatory, and should be one of the following:

• feat: A new feature
• fix: A bug fix
• build: Changes that affect the build system or external dependencies
• ci: Changes to our CI configuration files and scripts
• docs: Documentation only changes
• perf: A code change that improves performance
• refactor: A code change that neither fixes a bug nor adds a feature
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
• test: Adding missing tests or correcting existing tests

Scope

scopeis optional, but if included it should be a noun that gives context to the part of the codebase that is affected by the commit. Please do not use issue identifiers as scope.

Description

description is mandatory, and provides a short and concise description of the change. Be mindful of the following:

• use the imperative, present tense: "change" not "changed" nor "changes"
• don't capitalize the first letter
• no dot (.) at the end

Body

body is optional, but if included, it should use the imperative, present tense: "change" not "changed" nor "changes" (just as in description). The body should include the motivation for the change and contrast this with previous behavior.

Footer

footer is optional, but if included, it should use -in place of whitespace to distinguish it from a bodyor multiline body. It is the place to reference git issues that this commit closes.

Breaking Change

A commit that make incompatible API changes, should be denoted Breaking Change, either by a ! right after scope or by adding Breaking Change or Breaking-Change to footer.

Linter and formatter

Ruff is used for linting and formatting in both local development (pre-commit) and CI, with a shared configuration in pyproject.toml.

Ruff in CI pipeline

• Runs on pushes/merge requests in GitLab.
• Checks only (no auto-fix). If any check fails, the pipeline fails and the change cannot be merged.

Ruff in pre-commit hook

• Runs locally on commit (and optionally on push).
• Applies fixes and then fails if it changed files so you can review and commit the updates.
• First-time setup:

  pre-commit install
  # optional: also run on push
  pre-commit install --hook-type pre-push
  

• Manual sweep of the whole repo:

  pre-commit run --all-files
  

Documentation

This Project uses Sphinx for API docs. From the docs run:

make html

to generate the docs. Then you can open the docs index in your browser.

We leverage autodoc to generate the documentation from docstrings in code. To configure which classes are included in the API docs, edit and extend index.rst.