debugging-benchmark

Quickstart

Generating Passing and Failing Inputs:

from debugging_benchmark.calculator.calculator import CalculatorBenchmarkRepository
from debugging_framework.tools import GrammarBasedEvaluationFuzzer

calc = CalculatorBenchmarkRepository().build()
param = calc.to_dict()
fuzzer = GrammarBasedEvaluationFuzzer(**param)
fuzzer.run()
gen_inps = fuzzer.get_generated_inputs()

Evaluation:

from debugging_benchmark.calculator.calculator import CalculatorBenchmarkRepository
from debugging_framework.evaluator import Evaluation
from debugging_framework.tools import InputsFromHellEvaluationFuzzer

tools = [InputsFromHellEvaluationFuzzer]

subjects = SieveOfEratosthenesStudentAssignmentBenchmarkRepository().build()

result = Evaluation(
        tools=tools, 
        subjects=subjects[0:1],
        repetitions=1, 
        timeout=3600
        ).run()

Deeper Look into the Class Structure

Check out the Class Diagram for a first overview. Further down in this section we take a look at some key functions of interest.

Class Diagram

BenchmarkRepository and BenchmarkProgram can be found in debugging_framework/benchmark.py

StudentAssignmentBenchmarkProgram,StudentAssignmentRepository and GCDStudentAssignmentBenchmarkRepository can be found in debugging_benchmark/student_assignments.py

The faulty programs can be found at debugging_benchmark/student_assignments/problem_1_GCD and the correct implementation at debugging_benchmark/student_assignments/reference1.py

build()

Returns a List of BenchmarkPrograms. Calls internally _construct_test_program(). This function is our interface.

_construct_test_program()

Returns a BenchmarkProgram. Calls internally construct_oracle() to construct a oracle for our program.

construct_oracle()

Where the magic happens. Returns a Functions which loads the faulty and correct implementation, executes both with the input and checks if they are the same or not. If they are the same return OracleResult.PASSING, if not return OracleResult.FAILING

to_dict()

PLACEHOLDER

Install, Development, Testing

Install

If all external dependencies are available, a simple pip install PLACEHOLDER suffices. We recommend installing PLACEHOLDER inside a virtual environment (virtualenv):

python3.10 -m venv venv
source venv/bin/activate

pip install --upgrade pip
pip install PLACEHOLDER

Development and Testing

For development and testing, we recommend using PLACEHOLDER inside a virtual environment (virtualenv). By doing the following steps in a standard shell (bash), one can run the PLACEHOLDER tests:

git clone https://github.com/martineberlein/debugging-benchmark
cd debugging-benchmark

python3.10 -m venv venv
source venv/bin/activate

pip install --upgrade pip

# Run tests
pip install -e .[dev]
python3 -m pytest