Ctestgen

Russian readme version

Python lib for generating C-programms and running them by your target-program. Designed to assist in automated testing of programs working with C-programms.

Installation

pip3 install ctestgen

Example usage

Ctestgen has two main use cases: C-programs generating and running target program with C-programs as input.

C-programs generating

To generating programs, you need to inherit abstract class ctestgen.generator.TestGenerator and reimplement method _generate_programs(), that returns c-programs abstract syntax trees. Abstract syntax tree describing in OOP style, more here. Method run() calls _generate_programs() and then stores programs in output dir, which is defined when creating a class object: ExampleTestGenerator('example_generator_output').

Describe the class, that generating programs with sum function, that takes from 2 to 5 arguments. Full code here. Generated code here.

from ctestgen.generator import TestGenerator
class ExampleTestGenerator(TestGenerator):
    def _generate_programs(self):
        generated_programs = list()
        for i in range(2, 6):
            sum_arguments = [Int('num_' + str(arg_idx)) for arg_idx in range(i)]
            sum_function = Function('sum_' + str(i) + '_nums', Int, sum_arguments)
            sum_result = Int('sum')
            sum_body = CodeBlock(
                Assignment(VarDeclaration(sum_result), Add(sum_arguments)),
                Return(sum_result)
            )
            sum_function.set_body(sum_body)
            example_program = Program('sum_' + str(i))
            example_program.add_function(sum_function)
            generated_programs.append(example_program)
        return generated_programs

example_generator = ExampleTestGenerator('example_generator_output')
example_generator.run()

This generates, as example sum_3.c.

int sum_3_nums(int num_0, int num_1, int num_2) {
  int sum = num_0 + num_1 + num_2;
  return sum;
}

Generator API documentation here. It contains wide OOP tools to describe desired C-programs.

Tests running

To running programs, you need to inherit abstract class ctestgen.runner.TestRunner and reimplement method _on_test(), that called on every program in defined directory. TestRunner designed to run target program with every file, that contained in the target directory and its subdirectories and have target file extensions, which defines in class object creation, by default it`s .c.

TestRunner method run() describes testing pipeline, and class methods designed to override them in subclasses.

Testing pipeline:

• recursively collect test files in defined directory
• call _on_run_start(tests)
• for every dir, that contained test files, call _on_testdir(test_dir, test_filenames)
• for every file in that dir, call _on_test(test_dir, test_filename, env), that returns object of TestRunResults class, that contains run result (SUCCESS or FAIL) and output, that may be stored in corresponding files success.txt or fail.txt, which defines by argument dump_results_to_files in class object creation.
• call _on_run_finish(tests)

TestRunner collects metrics of successes and fails for every testset, and global metrics, that stores in file metrics.txt.

In most cases it`s enough to work with abstract class ctestgen.runner.BasicTestRunner, which takes target program arguments by argument run_args in class object creation. In _on_test(test_dir, test_filename, env) it calls target program as subprocess, and pass results to method _process_program_response(self, test_dir, test_filename, program_response). So it`s enough to override this method in subclass.

Describe the class, that checking word error in target program output, test fails if it was found. Full code here.

from ctestgen.runner import BasicTestRunner
class ExampleTestRunner(BasicTestRunner):
    def _process_program_response(self, test_dir, test_filename, program_response):
        if 'error' in program_response[0] or 'error' in program_response[1]:
            return TestRunResult(TestRunResult.ResultType.FAIL, program_response[0] + program_response[1])
        return TestRunResult(TestRunResult.ResultType.SUCCESS, program_response[0] + program_response[1])

example_runner = ExampleTestRunner(['tsar'],
                                   output_base_dir='runner_output',
                                   test_filename_extensions=['.c'],
                                   test_base_dir='example_generator_output',
                                   runner_name='example_runner')
example_runner.run()

License

This project is licensed under the MIT License - see the LICENSE file for details.