dlunch 3.3.0

The ultimate web app for a well organized lunch.

Data Lunch

The ultimate web app for a well organized lunch.

1. Table of contents

• 1. Table of contents
• 2. Development environment setup
  • 2.1. Miniconda
  • 2.2. Setup the development environment
  • 2.3. Environment variables
    • 2.3.1. General
    • 2.3.2. Docker and Google Cloud Platform
    • 2.3.3. TLS/SSL Certificate
    • 2.3.4. Encryption and Authorization
  • 2.4. Manually install the development environment
  • 2.5. Manually install data-lunch CLI
  • 2.6. Running the docker-compose system
  • 2.7. Running a single container
  • 2.8. Running locally
• 3. Additional installations before contributing
  • 3.1. Pre-commit hooks
  • 3.2. Commitizen
• 4. Release strategy from development to main branch
• 5. Google Cloud Platform utilities

2. Development environment setup

The following steps will guide you through the installation procedure.

2.1. Miniconda

Conda is required for creating the development environment (it is suggested to install Miniconda).

2.2. Setup the development environment

Use the setup script (setup_dev_env.sh) to install all the required development tools.

Use source to properly launch the script.

source setup_dev_env.sh

[!IMPORTANT] The setup script will take care of setting up the development environment for you. The script installs:

• 3 environments (data-lunch for development, ci-cd for pre-commit and other utilities, gc-sdk for interacting with Google Cloud Platform)
• pre-commit hooks
• data-lunch command line

2.3. Environment variables

The following environment variables are required for running the web app, the makefile or utility scripts.

2.3.1. General

Variable	Type	Required	Description
PANEL_APP	str	✔️	app name, data-lunch-app by default (used by makefile)
PANEL_ENV	str	✔️	environment, e.g. development, quality, production, affects app configuration (Hydra) and build processes (makefile)
PANEL_ARGS	str	❌	additional arguments passed to Hydra (e.g. panel/gui=major_release) in makefile and docker-compose commands
PORT	int	✔️	port used by the web app (or the container), default to 5000; affects app configuration and build process (it is used by makefile, Hydra and Docker)

2.3.2. Docker and Google Cloud Platform

[!NOTE] The following variables are mainly used during the building process or by external scripts.

Variable	Type	Required	Description
DOCKER_USERNAME	str	❌	your Docker Hub username, used by makefile and stats panel to extract container name
IMAGE_VERSION	str	❌	Docker image version, typically stable or latest (used by makefile and docker-compose commands)
GCLOUD_PROJECT	str	❌	Google Cloud Platform project_id, used by makefile for GCP's CLI authentication and for uploading the database to gcp storage, if active in web app configuration files (see panel.scheduled_tasks)
GCLOUD_BUCKET	str	❌	Google Cloud Platform bucket, used for uploading database to gcp storage, if active in web app configuration files (see panel.scheduled_tasks)
MAIL_USER	str	❌	email client user, used for sending emails containing the instance IP, [e-mail] (used only for Google Cloud Platform)
MAIL_APP_PASSWORD	str	❌	email client password used for sending emails containing the instance IP (used only for Google Cloud Platform)
MAIL_RECIPIENTS	str	❌	email recipients as string, separated by , (used for sending emails containing the instance IP when hosted on Google Cloud Platform)
DUCKDNS_URL	str	❌	URL used in compose_init.sh to update dynamic address (see Duck DNS's instructions for details, used when hosted on Google Cloud Platform)

2.3.3. TLS/SSL Certificate

[!TIP] Use the command make ssl-gen-certificate to generate local SSL certificates.

Variable	Type	Required	Description
CERT_EMAIL	str	❌	email for registering SSL certificates, shared with the authority Let's Encrypt (via certbot); used by docker-compose commands
DOMAIN	str	❌	domain name, e.g. mywebapp.com, used by docker-compose commands (certbot), in email generation (scripts folder) and to auto-generate SSL certificates

2.3.4. Encryption and Authorization

[!NOTE] All variables used by Postgresql are not required if db=sqlite (default value).
All variables used by OAuth are not required if server=no_auth (default value).

[!IMPORTANT] DATA_LUNCH_COOKIE_SECRET and DATA_LUNCH_OAUTH_ENC_KEY are required even if server=no_auth is set.

Variable	Type	Required	Description
DATA_LUNCH_COOKIE_SECRET	str	✔️	Secret used for securing the authentication cookie (use make generate-secrets to generate a valid secret)
DATA_LUNCH_OAUTH_ENC_KEY	str	✔️	Encription key used by the OAuth algorithm for encryption (use make generate-secrets to generate a valid secret)
DATA_LUNCH_OAUTH_KEY	str	❌	OAuth key used for configuring the OAuth provider (GitHub, Azure)
DATA_LUNCH_OAUTH_SECRET	str	❌	OAuth secret used for configuring the OAuth provider (GitHub, Azure)
DATA_LUNCH_OAUTH_REDIRECT_URI	str	❌	OAuth redirect uri used for configuring the OAuth provider (GitHub, Azure), do not set to use default value
DATA_LUNCH_OAUTH_TENANT_ID	str	❌	OAuth tenant id used for configuring the OAuth provider (Azure), do not set to use default value
DATA_LUNCH_DB_USER	str	❌	Postgresql user, do not set to use default value
DATA_LUNCH_DB_PASSWORD	str	❌	Postgresql password
DATA_LUNCH_DB_HOST	str	❌	Postgresql host, do not set to use default value
DATA_LUNCH_DB_PORT	str	❌	Postgresql port, do not set to use default value
DATA_LUNCH_DB_DATABASE	str	❌	Postgresql database, do not set to use default value
DATA_LUNCH_DB_SCHEMA	str	❌	Postgresql schema, do not set to use default value

2.4. Manually install the development environment

[!WARNING] This step is not required if the setup script (setup_dev_env.sh) is used.

Use the terminal for navigating to the repository base directory.
Use the following command in your terminal to create an environment named data-lunch manually.
Otherwise use the setup script to activate the guided installing procedure.

conda env create -f environment.yml

Activate the new Conda environment with the following command.

conda activate data-lunch

2.5. Manually install data-lunch CLI

[!WARNING] This step is not required if the setup script (setup_dev_env.sh) is used.

The CLI is distributed with setuptools instead of using Unix shebangs.
It is a very simple utility to initialize and delete the app database. There are different use cases:

• Create/delete the sqlite database used by the app
• Initialize/drop tables inside the sqlite database

Use the following command for generating the CLI executable from the setup.py file, it will install your package locally.

pip install .

If you want to make some changes to the source code it is suggested to use the following option.

pip install --editable .

It will just link the package to the original location, basically meaning any changes to the original package would reflect directly in your environment.

Now you can activate the Conda environment and access the CLI commands directly from the terminal (without using annoying shebangs or prepending python to run your CLI calls).

Test that everything is working correctly with the following commands.

data-lunch --version
data-lunch --help

2.6. Running the docker-compose system

Since this app will be deployed with an hosting service a Dockerfile to build a container image is available.
The docker compose file (see docker-compose.yaml) deploys the web app container along with a load balancer (the nginx container) to improve the system scalability.

Look inside the makefile to see the docker and docker-compose options.

To build and run the dockerized system you have to install Docker.
Call the following make command to start the build process.

make docker-up-init docker-up-build

up-init initialize the ssl certificate based on the selected environment (as set in the environment variable PANEL_ENV, i.e. development or production).
Call only make (without arguments) to trigger the same command.
A missing or incomplete ssl certificate folder will result in an nginx container failure on start-up.

Images are built and the two containers are started.

You can then access your web app at http://localhost:PORT (where PORT will match the value set through the environment variable).

[!NOTE] You can also use make docker-up to spin up the containers if you do not need to re-build any image or initialize ssl certificate folders.

[!IMPORTANT] An additional container named db is started if db=postgresql is set

2.7. Running a single container

It is possible to launch a single server by calling the following command.

make docker-build

make docker-run

You can then access your web app at http://localhost:5000 (if the deafult PORT is selected).

2.8. Running locally

Launch a local server with default options by calling the following command.

python -m dlunch

Use Hydra arguments to alter the app behaviour.

python -m dlunch server=basic_auth

See Hydra's documentation for additional details.

3. Additional installations before contributing

[!WARNING] This step is not required if the setup script (setup_dev_env.sh) is used.

Before contributing please create the pre-commit and commitizen environments.

cd requirements
conda env create -f pre-commit.yml
conda env create -f commitizen.yml

3.1. Pre-commit hooks

This step is not required if the setup script is used.

Then install the precommit hooks.

conda activate pre-commit
pre-commit install
pre-commit autoupdate

Optionally run hooks on all files.

pre-commit run --all-files

3.2. Commitizen

[!WARNING] This step is not required if the setup script (setup_dev_env.sh) is used.

The Commitizen hook checks that rules for conventional commits are respected in commits messages. Use the following command to enjoy Commitizen's interactive prompt.

conda activate commitizen
cz commit

cz c is a shorther alias for cz commit.

4. Release strategy from development to main branch

[!CAUTION] This step is required only if the CI-CD pipeline on GitHub does not work.

In order to take advantage of Commitizen bump command follow this guideline.

First check that you are on the correct branch.

git checkout main

Then start the merge process forcing it to stop before commit (--no-commit) and without using the fast forward strategy (--no-ff).

git merge development --no-commit --no-ff

Check that results match your expectations then commit (you can leave the default message).

git commit

Now Commitizen bump command will add an additional commit with updated versions to every file listed inside pyproject.toml.

cz bump --no-verify

You can now merge results of the release process back to the development branch.

git checkout development
git merge main --no-ff

Use "update files from last release" or the default text as commit message.

5. Google Cloud Platform utilities

[!WARNING] This step is not required if the setup script (setup_dev_env.sh) is used.

The makefile has two rules for conviniently setting up and removing authentication credentials for Google Cloud Platform command line interface: gcp-config and gcp-revoke.