The General Aviation Aircraft Family Benchmark (GAAFpy)

A Python package of the GAA family benchmark problem developed by T.W. Simpson, W. Chen. J.K. Allen, F. Mistree, B.S. D'Souza, R. Shah, P.M. Reed, and D. Hadka.

This repository started as a Python translation of the RealWorldBenchmarks implementation in the MOEA framework by Zatarain Salazar, J., Hadka, D., Reed, P., Seada, H., & Deb, K (see reference 5).

Description

The General Aviation Aircraft Family Benchmark concerns the design of a family of three aircraft, modeled after the Beechcraft Bonanzo B36TC. Two-, four- and six-seater variants are designed. The benchmark is used to evaluate different product family design methods. Here, product family design trades the invididual performance of each variant within a family with the family level performance. This family level performance is typically expressed in terms of R&D cost/time, manufacturing costs, OEM profit, etc. This family level performance may be improved by introducing commonality across variants, such that design elements or systems are common across the variants.

Each variant in the GAA benchmark is parameterised using 9 design variables, being:

Our implementation of the GAA benchmark problem uses quadratic response surface models of the performance metrics present in the original problem definition (see reference 1) to reduce the computational cost. These performance metrics are the take-off noise, aircraft empty weight, direct operating cost, acquisition cost, maximum lift-to-drag ratio, maximum cruise speed, ride roughness coefficient, aircraft fuel weight, and maximum range. Some of these metrics have associated constraints defined, which are:

Additionally, several metrics have goals defined as:

Combined, these make the GAA problem challenging to solve, and an interesting candidate for testing new methods and algorithms.

Repository Structure

.
├── .gitignore
├── docs
│   ├── images
│   │   └── GAAModel.png
├── example
│   ├── example_family.py
│   ├── example_variant.py
│   └── README.md
├── LICENSE
├── pyproject.toml
├── README.md
├── src/GAAFpy
│   ├── __init__.py
│   ├── family.py
│   ├── rsm_coefficients.json
│   ├── utils.py
│   └── variant.py
└── verification_validation
    ├── __init__.py
    ├── gaa_verifier_validator.py
    ├── MOEA-GAA-output.csv
    ├── README.md
    └── validation_report.txt

The './example/' folder contains two example usages of the GAAFpy package. One for the optimisation of a single variant (either 2-, 4-, or 6-seater), and one for the optimisation of the complete family. The './verification_validation/' folder contains the code and input data used to check the implementation of the GAAFpy module against that of the MOEA framework. The validation report contained in this folder is summarised below.

Installation

Requirements

The GAA benchmark package is tested to work using the following Python version and dependencies:

1. Python >= 3.10
2. Numpy
3. (optional) Pymoo >= 0.6

Installation Guide

The recommended method is through PyPi by running the command:

pip install gaafpy

To install the optional Pymoo multi-objective optimisation package used in the example, use:

pip install -U pymoo

Example usage

A simple example, integrating the GAAFpy family implementation into a Pymoo NSGA-II optimisation problem, using vectorised evaluation:

# Import 3rd party libraries
import numpy as np
from pymoo.core.problem import Problem
from pymoo.algorithms.moo.nsga2 import NSGA2
from pymoo.termination import get_termination
from pymoo.optimize import minimize

# Import the relevant GAAFpy classes
from GAAFpy.family import GAABenchmark
from GAAFpy.utils import VARIABLE_BOUNDS


# Define the problem class for the GAA benchmark, inheriting from Pymoo's Problem class
class GAABenchmarkProblem(Problem):
    """
    GAA benchmark Pymoo problem definition with 27 design variables,
    10 objectives and 18 constraints. Evaluated using vectorised evaluation,
    where the _evaluate() method retrieves a set of solutions.
    """

    def __init__(self) -> None:
        """Initialise problem definition"""

        super().__init__(n_var=27,
                         n_obj=10,
                         n_ieq_constr=18,
                         xl=np.array(VARIABLE_BOUNDS[0]),
                         xu=np.array(VARIABLE_BOUNDS[1]),
                         )

    def _evaluate(self, x, out, *args, **kwargs) -> None:
        """Evaluate the problem"""

        # Create benchmark object and evaluate the design vector(s)
        benchmark = GAABenchmark(x)
        objectives, constraints, combined_CV = benchmark.evaluate()

        # Objectives need to be formatted to handle the
        # minimisation & maximisation objectives
        objectives_sign_convention = np.array([
            1,   # Minimise noise
            1,   # Minimise empty weight
            1,   # Minimise DOC
            1,   # Minimise roughness
            1,   # Minimise fuel weight
            1,   # Minimise purchase cost
            -1,  # Maximise range
            -1,  # maximise L/D
            -1,  # Maximise Vcruise_max
            1    # Minimise PFPF
        ])
        objectives = objectives * objectives_sign_convention

        # Write the objectives and constraints to the output dictionary
        out["F"] = objectives
        out["G"] = constraints


# Construct the problem and algorithm, using mostly default operators for NSGA-II
problem = GAABenchmarkProblem()
algorithm = NSGA2(pop_size=1000, eliminate_duplicates=True)

# Define termination after 100 generations
termination = get_termination('n_gen', 100)

# Run the optimisation and print the best solutions found
res = minimize(
    problem,
    algorithm,
    termination,
    seed=42,
    save_history=True,
    verbose=True
)

print("Best solution found:")
print(f"X = {res.X}")
print(f"F = {res.F}")
print(f"G = {res.G}")

Validation

Validation of the implemented GAAFpy package was done using the outputs from the provided Example Java file in the MOEA RealWorldBenchmarks repository. This example outputs 100 solutions to the GAA problem obtained using the NSGAII algorithm after 10,000 generations. The resulting validation report shows a maximum relative difference of 3.33e-06, a mean relative difference of 1.79e-07, and a std dev of differences of 3.69e-07. For each of the objectives, the validation performance is summarised in the below table:

The 100 solutions obtained from the MOEA framework show zero constraint violations across the 18 constraints present in the problem. This behaviour is replicated by this Python implementation.

Community Guidelines

This package is currently being maintained by me @TSVermeulen. If you find any bugs, want to contribute or have any questions, you can open an issue on GitHub.

Authors

The code scripts provided in this repository have been developed by Thomas Vermeulen 0009-0000-0182-0244, Technische Universiteit Delft

License

The Benchmark problem itself is copyright by the respective authors. Please cite them as appropriate if using the benchmark problem. The GAAFpy package itself is licensed under an MIT license.

References

1. T. W. Simpson, W. Chen, J. K. Allen, and F. Mistree (1996). "Conceptual design of a family of products through the use of the robust concept exploration method." In 6th AIAA/USAF/NASA/ ISSMO Symposium on Multidiciplinary Analysis and Optimization, vol. 2, pp. 1535-1545. (Link)

2. T. W. Simpson, B. S. D'Souza (2004). "Assessing variable levels of platform commonality within a product family using a multiobjective genetic algorithm." Concurrent Engineering: Research and Applications, vol. 12, no. 2, pp. 119-130. (Link)

3. R. Shah, P. M. Reed, and T. W. Simpson (2011). "Many-objective evolutionary optimization and visual analytics for product family design." Multiobjective Evolutionary Optimisation for Product Design and Manufacturing, Springer, London, pp. 137-159. (Link)

4. D. Hadka, P. M. Reed, and T. W. Simpson (2012). "Diagnostic Assessment of the Borg MOEA on Many-Objective Product Family Design Problems." WCCI 2012 World Congress on Computational Intelligence, Congress on Evolutionary Computation, Brisbane, Australia, pp. 986-995. (Link)

5. Zatarain Salazar, J., Hadka, D., Reed, P., Seada, H., & Deb, K. (2024). Diagnostic benchmarking of many-objective evolutionary algorithms for real-world problems. Engineering Optimization, 1–22. (Link)