ctao-qualpipe-webapp 0.1.2

CTAO DPPS QualPipe Web Application

QualPipe-Webapp

The QualPipe-webapp project uses FastAPI as backend and frontend, plus D3.js for dynamic frontend visualizations, separated into clean services using Docker Compose.

🚀 Quick Start

With Docker (Recommended for users)

make up                    # Start all services
# Open http://localhost/
make down                  # Stop all services

With Pixi (Recommended for developers)

pixi shell                 # Activate environment (optional)
pixi run dev-setup         # Install dependencies + generate models
pixi run generate-codegen  # Regenerate models (if needed, with schema validation files)

---

📂 Project Structure

/qualpipe-webapp
│
├── docker-compose.dev.yml           # Docker compose for developer
├── docker-compose.yml               # Docker compose for user
├── Makefile                         # Makefile to build Backend and Frontend
│
├── docs/                            # Document folder
│
├── src/
│   └── qualpipe_webapp/             # FastAPI backend
│       ├── backend/                 # FastAPI backend
│       │   ├── main.py              # Main FastAPI app
│       │   ├── data/                # JSON data sources
│       │   ├── requirements.txt     # Backend dependencies
│       │   ├── requirements-dev.txt # Backend dependencies for developer
│       │   └── Dockerfile           # Backend container
│       │
│       ├── frontend/                # FastAPI frontend
│       │   ├── /templates/          # Template pages
│       │   ├── /static/             # Static files (css, js, lib)
│       │   ├── main.py              # Main FastAPI app
│       │   ├── requirements.txt     # Frontend dependencies
│       │   ├── requirements-dev.txt # Frontend dependencies for developer
│       │   └── Dockerfile           # Frontend container
│       │
│       ├── nginx/
│       │   ├── nginx.conf
│       │   └── ssl/                 # (optional later)
│
└── .gitignore

Instructions

Docker Deployment

To run it in the background, from the project folder qualpipe-webapp simply execute:

make up

Then open your browser:

• Frontend: http://localhost/

• API: http://localhost/api/data

To stop everything, execute:

make down

Pixi Development

For development with automatic model generation:

# Activate pixi environment (optional)
pixi shell

# Setup environment and generate models
pixi run dev-setup

# Manually regenerate models (if criteria change)
pixi run generate-codegen

# Or use the installed console script
qualpipe-generate-models --help

If you also want to automatically see the logs you can then run it via docker compose

To run it the first time execute: (later on you can remove the --build)

docker-compose up --build

To stop it you can soft-kill with CTRL-C and shut down all the services executing make down.

If instead you do any modification to the config while the containers are running or in case Docker caches wrong stuff, you can restart it running:

make restart

or

docker-compose down -v --remove-orphans
docker-compose up --build

🛠️ Makefile Commands

Description of all the Makefile functionalities:

Command	What it does
make build	Build the Docker images
make up	Build and start services (backend + frontend) in background
make down	Stop all services
make logs	View combined logs (both frontend and backend)
make logs-backend	View only backend logs
make logs-frontend	View only frontend logs
make restart	Restart the containers
make prune	Clean up unused Docker containers/images
------------------	-----------------------------------------------------------
make build-dev	Build the Docker images
make up-dev	Build and start services (backend + frontend) in background

🔧 Code Generation with Pixi

This project uses Pixi for package management and includes automated code generation for Pydantic models based on qualpipe criteria, and automatic schema generation.

Prerequisites

Install Pixi on your system:

# On macOS
curl -fsSL https://pixi.sh/install.sh | bash

# On Linux
curl -fsSL https://pixi.sh/install.sh | bash

# Or using conda/mamba
conda install -c conda-forge pixi

Available Pixi Commands

Command	Description
pixi run dev-setup	Complete development setup (install + generate models)
pixi run install	Install the package with dependencies
pixi run install-all	Install with all optional dependencies
pixi run generate-codegen	Generate Pydantic models from qualpipe criteria
pixi run generate-frontend-schema	Generate yaml file for javascript frontend validation
pixi shell	Activate the pixi environment

Code Generation Workflow

The project automatically generates Pydantic models from qualpipe criteria classes:

# Generate models manually
pixi run generate-codegen

# Or use the console script directly (after installation)
qualpipe-generate-models --module qualpipe.core.criterion \
    --out-generated src/qualpipe_webapp/backend/generated \
    --out-schemas src/qualpipe_webapp/frontend/static

Generated Files

The code generation creates:

• Python Models: src/qualpipe_webapp/backend/generated/qualpipe_criterion_model.py

  • Pydantic models for each criterion type
  • Validation logic for telescope parameters
  • Type-safe configuration classes

• JSON/YAML Schemas: src/qualpipe_webapp/frontend/static/

  • criteria_schema.json - JSON schema for frontend criteria report validation
  • criteria_schema.yaml - YAML schema for configuration

Frontend schema generation

Such files are then implemented into metadata_schema.yaml with:

pixi run generate-frontend-schema

which is used for complete frontend validations, with all the correct references reads from the configuration file config.js. The metadata_schema.yaml file is auto-generated from a template, so

[!IMPORTANT] Do not edit metadata_schema.yaml directly — edit template_metadata_schema.yaml instead.

Development Setup with Pixi

1. Clone and setup environment:

   git clone <repository-url>
   cd qualpipe-webapp
   pixi run dev-setup  # Install dependencies then generate models and schema files
   

2. Activate the environment:

   pixi shell
   

3. Run tests:

   pixi run test
   

4. Regenerate models and schema files (if qualpipe criteria change):

   pixi run generate-codegen
   pixi run generate-frontend-schema
   

Integration with CI/CD

The generated models are automatically created during:

• Local development: pixi run dev-setup runs code generation
• CI/CD pipelines: Code generation runs before tests
• Package installation: Post-install hooks generate models

[!IMPORTANT] Important Notes ⚠️ Generated files are git-ignored - They're created automatically and should not be committed.

✅ Tests depend on generated models - Always run code generation before testing.

🔄 Automatic regeneration - Models update automatically when qualpipe criteria change.

🛠️ Troubleshooting

Import errors with generated models?

pixi run generate-codegen  # Regenerate models

Tests failing?

# Ensure models and schemas are generated before running tests
pixi run generate-codegen
pixi run generate-frontend-schema
pixi run test

Pixi environment issues?

pixi clean              # Clean cache
pixi run dev-setup      # Complete reinstall + code generation

Developers

For developers, you can build containers that automatically file changes, avoiding the need to restart the containers every time. To do so, simply execute:

make build-dev
make up-dev

This will automatically output also every log produced.

🧪 Running Tests

1. Python Tests with Pytest

Install dependencies:

pip install -r src/qualpipe_webapp/backend/requirements-dev.txt
pip install -r src/qualpipe_webapp/frontend/requirements-dev.txt

Run tests:

pytest -v src/qualpipe_webapp/backend/tests/
pytest -v src/qualpipe_webapp/frontend/tests/

Alternative method

The same result can be achieved in a more compact way using pixi:

pixi run test

---

2. JavaScript Unit Tests with Mocha

Install NodeJS from the official website.

Install NodeJS dependencies:

npm install

Run all JS unit tests:

npx mocha src/qualpipe_webapp/frontend/unit_tests_js/ --reporter spec

Run a specific test file:

npx mocha src/qualpipe_webapp/frontend/unit_tests_js/base.test.js --reporter spec

The report will be automatically saved in the file test-results.xml

[!NOTE] Add new unit tests in the folder: src/qualpipe_webapp/frontend/unit_tests_js/

If you also want to see the coverage of your unit-tests simply execute:

npm run coverage

---

3. End-to-End Tests with Playwright

In order to run end-to-end interface test with the real frontend DOM you need first to install NodeJS and its dependencies (see first step of previous section).

Install Playwright browsers:

To install the Playwright library and the browsers (Chromium, Firefox, WebKit) necessary to run the tests, execute in the terminal:

npx playwright install

Start the FastAPI webapp (in another terminal):

make up-dev

or

make up

Run all Playwright E2E tests:

npx playwright test

Run a specific Playwright test:

To run e.g. the home.test.js test execute:

npx playwright test src/qualpipe_webapp/frontend/tests_e2e_js/home.test.js

In this way Playwright will open the browser, load the page, and verify that the DOM successfully pass the selected test.

Generate and view HTML test report:

npx playwright test --reporter html
npx playwright show-report

It will serve the HTML report at http://localhost:9323. Press Ctrl+C to quit.

[!NOTE] Add new end-to-end tests in the folder: src/qualpipe_webapp/frontend/tests_e2e_js/

🛠️ Troubleshooting

[!IMPORTANT] Playwright test are failing for a specific browser? You can execute Playwright tests on a specific browser (chromium, firefox, webkit)

npx playwright test --project=chromium

⚠️ Note: WebKit browser generally works only with the latest Mac OS update.

[!IMPORTANT] Some tests about validation are failing? Ensure you have generated models and schema (see the Tests failing? section for detailed instructions.)

[!IMPORTANT] All test failing with ERR_CONNECTION_REFUSED? Verify that the WebApp is running. Open a browser and navigate to http://localhost/home . If you can't display the page, the WebApp is not running. See instruction above on how to run it.

---

👩‍💻 Contributing

If you want to contribute in developing the code, be aware that we are using pre-commit, code-spell and ruff tools for automatic adherence to the code style. To enforce running these tools whenever you make a commit, setup the pre-commit hook executing:

pre-commit install

The pre-commit hook will then execute the tools with the same settings as when a merge request is checked on GitLab, and if any problems are reported the commit will be rejected. You then have to fix the reported issues before tying to commit again.

📄 License

This project is licensed under the BSD 3-Clause License. See the LICENSE file for details.

---

Useful Links

• pre-commit documentation
• Mocha documentation
• Playwright documentation
• Pytest documentation