Module tester's Gtest GUI is a full-featured graphical user-interface to C++ test applications using the GoogleTest framework.
Project description
Description
Module-tester’s GtestGui is a full-featured graphical user-interface to C++ test applications using the GoogleTest framework.
The tool will work with any application with Gtest command line interface, however it is designed especially for C++ developers using test-driven design based on module testing and integration testing. These differ from unit-testing by longer execution times and usually not fully predictable results, which in turn require multiple repetitions of each test case. To support this well, GtestGui offers firstly easily accessible ways for test case filtering and concurrent scheduling across multiple CPUs; Secondly, there are multiple features for tracking progress and managing results via sorting and filtering, which are fully usable already while a test campaign still is in progress.
GtestGui typically is started with the path of an executable on the command line which is built using the gtest library. Using the “Run” button in the GUI, this executable can then be started and its progress be monitored live in the result log frame of the main window. Additional controls allow specifying options such as test case filter string and repetition count, which are forwarded to the executable via the respective “--gtest_*” command line arguments.
While tests are running, test verdicts can be monitored live in the result log. Traces of passed or failed tests can already be analyzed simply by double-clicking on an entry in the result log, which will open the trace of that particular test case run. For this purpose GtestGui comes bundled with Trowser, which is a graphical browser for large line-oriented text files. Trowser in principle is just a text browser with syntax highlighting and search, but its search capabilities are designed especially to facilitate analysis of complex debug logs, essentially by allowing to build a condensed (filtered) view of the trace file in the search result window via incremental searches and manual manipulations. By default, GtestGui is configured to user trowser.py, but there is also a more modern-looking Qt5 variant. GtestGui can also be configured to use any other GUI application for opening trace snippets.
Test control
The application main window consists of a menu bar, a frame containing test controls, a frame containing the test result log, and a text frame for trace preview. This chapter describes the top-most frame with the test controls.
Test case filter
The entry field at the top of the test control frame allows specifying a test case filter, so that only a matching sub-set of test cases is run. The filter can be entered manually using the same syntax as the “--gtest-filter” command line option: The format of a filter expression is a “:”-separated list of test case names of wildcard patterns (positive patterns), optionally followed by a “-” and another “:”-separated pattern list (negative patterns). A test matches the filter if and only if it matches any of the positive patterns, but none of the negative ones. Wildcard characters are “*” (matching any sub-string) and “?” (matching any single character). As a special case, when no positive pattern is specified, all test cases are considered matching.
Alternatively, the test case filter can be modified via the drop-down menu below the entry field (which can be opened by the Cursor-Down key or a click on the drop-down button next to the entry field). The menu has entries for selecting and deselecting entries test suites as well as individual test cases. When modifying the filter this way, GtestGui will update the entry field with the shortest filter expression it can find using trailing wild card and negative patterns.
Yet another alternative for modifying test case filters is the test case list dialog, either via its context menu or the “Return” and “Delete” key bindings. Finally note any modification to the test case filter can be undone using “Control-Z” key binding in the entry field, or redone using “Control-Y” key binding.
Test control options
- Repetitions:
If a value larger than 1 is entered here, it is passed via the “--gtest_repeat=NNN” option on the executable’s command line. This causes each test case to be repeated the given number of times.
- CPUs:
This option is described in Test campaign control buttons.
- Ignore filter:
When more than one CPU is configured, this option can be used for scheduling different sets of test cases on different CPUs: The first set of CPUs runs only test cases matching the test case filter. The second set of CPUs runs all test cases. The size of the second set is determined by the given number.
This feature is useful when running a long test campaign after modifying a test case as it allows effectively increasing the repetition count of the modified test case. It is also useful when running test cases that aim to find race conditions, as the additional concurrent execution of all test cases serves to generate a background load that increases randomness of thread scheduling.
- Fail limit:
When set to a non-zero value, the test campaign is stopped after the given number of failed test cases was reached. Note for the limit the total of failures is counted across all test cases and all CPUs.
This option is not using the respective Gtest option, as that option would not work as expected when using multiple CPUs (as it would work independently for tests on each CPU). Instead, the handling is implemented in result handling in GtestGui. As there is a delay resulting from buffering in the pipeline between test application and GtestGui, more test cases may have failed in the mean time, so that the actual number of failures after the actual end of all test processes may be higher than the limit.
- Clean traces of passed tests:
When enabled, trace output of passed test case runs is not written to the trace output file. If all test cases of a test campaign passed, the complete output file is removed automatically when tests are stopped. This feature is intended for long test campaigns to reduce consumption of disk space.
- Clean core files:
When enabled, core files with pattern “core.PID” are deleted automatically after a test case crashed. (See chapter Result log for more details on core files.) Note GtestGui can only clean core files from processes it controls directly. It is not able to clean core dumps created by death tests that are child processes of the main test process.
- Shuffle execution order:
When enabled, “--gtest_shuffle” option is set via the executable’s command line. This option randomizes the order of test execution.
- Run disabled tests:
When enabled, “--gtest_also_run_disabled_tests” option is set via the executable’s command line. The option enables execution of test suites and individual test cases whose name starts with “DISABLED”.
The option also affects test case filter and test case selection menus within GtestGui: When the option is not set, entering filter pattern “*DISABLED*” would raise a warning that it matches no test cases (even if there are some with that name). The drop-down menu below the entry field would no show such names.
- Break on failure:
When enabled, “--gtest_break_on_failure” option is set via the executable’s command line. This will cause SIGTRAP to be sent to the test process upon the first failure. As no debugger is attached, this will cause the process to crash.
When core dumps are enabled in the kernel, the core will be save by GtestGui and can be analyzed via the Extract stack trace from core dump command in the result log’s context menu. When core dumps are not enabled, this option is probably not very useful.
- Break on exception:
When enabled, “--gtest_catch_exceptions=0” option is set via the executable’s command line. This will disable catching of exceptions by the Gtest framework, which means the exception causes the process to terminate.
- Valgrind and Valgrind - 2nd option set:
The two valgrind options serve to run each execution of the test executable under valgrind using the configured command line. Notably, valgrind checks are performed across the complete lifetime of the test process, thus spanning all test cases or test repetitions. Therefore, if for example a memory leak is reported at the end, it cannot be determined which test case caused it (or it may even be caused by interaction of the test sequence.) Therefore valgrind errors are reported with a special entry in the result log. Some kind of errors such as invalid memory accesses can be mapped to test cases based on the position of the error report in the output stream. Note however that the position may not exactly reflect the timing of occurrence due to possible buffering in output streams within the test executable.) See Result log and Configuration for more details.
Result log
The result log frame is located in the middle of the main window. When started for the first time, the log is usually empty. However, results can also be imported via the command line, for example from a file that contains output from a test executable that was redirected into a file. The result log may also show results from a previous run of GtestGui, if auto-import is enabled in Configuration.
The result log contains one line for each [ OK ], [ SKIPPED ] and [ FAILED ] line in the test application’s gtest-generated output. The test executable’s output stream is redirected to a pipe that is read continuously by GtestGui for this purpose. GtestGui also stores this output to a file, so that the trace output between [ RUN ] and verdict text line can be opened in a trace browser.
In addition to the standard verdicts generated by the gtest library, GtestGui supports verdict [ CRASHED ], which is appended to the trace file when an executable terminates with a non-zero exit code within a test case.
When running tests under valgrind, a special result log entry “Valgrind error” is added if valgrind’s exit code signals detection of an error. This case is special, as it’s not known which test case caused the error, if more than one was running. Double-clicking this entry will therefore open the complete trace file.
Each entry in the result log contains the following information:
Local time at which the result was captured.
Verdict: Passed, failed, skipped, crashed, or special case valgrind of startup errors.
Test case name.
“Seed” value parsed from trace output, if a regular expression was configured for that purpose in Configuration.
Test duration as reported by gtest (in milliseconds).
In case of failure, source code file and line where of the first “Failure” was reported in trace output.
Timestamp or name of executable that generated the test output, in case the executable has changed since running the test.
When selecting an entry by clicking on it, the corresponding trace output is shown in the trace preview frame below the result log. In case of a test case failure, the view is centered on the first line containing “Failure” and the text is marked by light red background.
When clicking on an entry with the right mouse button, a context menu opens that allow opening the trace file, adding or removing the selected test case from the filter option, scheduling a test for repetition, excluding a result from the log display, or removing results.
Additional commands are offered in the “Result log” drop-down of the main window menu. The commands allow sorting and filtering the result log by various criteria.
On UNIX platform, the context menu additionally supports extracting a thread overview and stack-trace (backtrace) of each thread from a core dump in case of a crash during execution of a test case. To allow this, /proc/sys/kernel/core_pattern needs to be configured as “core.%p”, or alternatively as “core”, when additionally /proc/sys/kernel/core_uses_pid is set to “1”. If GtestGui thus finds a file named “core.PID” with PID matching that of the test executable process after a process crashed, it will automatically preserve that core file for analysis by renaming it and moving it into the directory where trace text output files are stored. It will also preserve the executable file version by keeping a hard link to the same executable.
Configuration
User-interface options
A few simple options for the user interface are available directly in the Configure menu:
- Select font for result log:
Selects the font used in the result log list in the main window, as well as in the test case list and job list dialogs. By default, the font is determined by Python’s Tkinter and depends on the platform.
- Select font for trace preview:
Selects the font used in the trace preview frame at the bottom of the main window. By default, a fixed font is used; Font family and size are determined by Python’s Tkinter and depend on the platform.
- Show test controls:
This option can be unchecked for temporarily hiding the test control frame in the main window. This allows for more space for the result log during result analysis. This option is not stored in persistent configuration and will always be enabled upon start of the application. Note while the controls are not shown, some operations are still possible via key bindings as well as the Control menu.
- Show tool-tip popups:
The option can be unchecked to disable display of “tool-tip” popup windows when hovering with the mouse on labels or check-buttons in the main window and dialogs which have such built-in help. Changes of the option are stored in the configuration file, so that it is persistent. This may be useful once you are sufficiently familiar with the tool.
Test management options
- Trace browser
This entry field selects the external application used for displaying trace files and trace snippets, when double-clicking on an entry in the result log. By default, trace browser trowser.py is used. You can either specify just the application file name, or its full path if it is not found via PATH configured in environment. The path of the trace file to be displayed will be appended to the given command line.
Currently the application path name or parameters cannot contain space characters, as these are assumed be be separators.
Note for the Windows platform: When using the default of trowser.py, you may need to insert the Python interpreter in front of “trowser.py”, depending on your Python installation. In that case you need to add the full path to where trowser.py is installed.
Enable option Browser supports reading from STDIN if the selected trace browser supports reading text from “standard input” via a pipeline. In this case filename “-” is passed on the command line instead of a file name. The default browser trowser.py supports this. When not enabled, GtestGui has to create temporary files for passing trace snippets to the browser application.
- Pattern for seed
If a regular expression pattern is specified here, it will be applied to the trace of each test case. The string returned by the first match group (i.e. the first set of capturing parenthesis) will be shown in the corresponding result log as “seed”. (This is intended for allowing repeat of a test sequence exactly even for test cases using randomness, by starting their PRNG with the same seed. This is not yet supported however, due to lack of an interface for passing a list of seed values via the GTest command line interface.)
- Directory for trace files
Specifies the directory where to store temporary files for trace output and core dump files collected from the executable under test. If empty, the current working directory at the time of starting GtestGui is used. Note sub-directories will be created in the given directory for each executable file version. If you want to use the “copy executable” option, the specified directory needs to be in the same filesystem as the executables. If you want to keep core dumps, the directory needs to be in the same filesystem as the working directory (because they will be moved, not copied due to size.)
- Automatically remove trace files of passed tests upon exit
When enabled, output from passed test cases is automatically removed from created trace files upon exiting the application. Trace files and sub-directories only containing passed test results are thus removed entirely. Note imported trace files are never modified or removed automatically, so you may need to remove these manually once after enabling this option (e.g. via result log context menu).
- Automatically import trace files upon start
When enabled, all trace files found in sub-directories under the configured trace directory are read after starting GtestGui. Test case results found in the files are shown in the result log window.
- Create copy of executable under test
When enabled, a copy of the current executable under test is made within the configured trace directory. (Technically, the copy is achieved by creating a so-called “hard link”, so that no additional disk space is needed.) This is recommended so that recompiling the executable does not affect the current test run (i.e. compilation may either fail with error “file busy” when tests are running, or tests may crash). This option is required for allowing to extract stack traces from core dump files taken from an older executable version. Note this option may not work when using trace directories in locations such as /tmp on UNIX-like systems, as these usually are configured to disallow executable files for security reasons.
- Valgrind command line
UNIX only: Command lines to use for running test executables when one of the “Valgrind” options in the main window is enabled. The executable name and gtest options will be appended to the given command line.
There are two separate entry fields, corresponding to the two check-buttons in the main window. This is intended for allowing to configure a configuration variant that runs faster and one that is slower but performs deeper analysis. By default, the command will perform check for memory leaks (notably at end of all test cases, not for individual test cases in a run of multiple tests) and for use of uninitialized memory. The second command line has additional options for tracking the origin of uninitialized memory.
Currently the path or parameters cannot contain space characters, as these are assumed be be separators.
- Valgrind supports –exit-code
When this option is set, parameter “–error-exitcode=125” will be appended to the given valgrind command lines. This is required for detecting automatically that valgrind found errors during test execution. Only when enabled, result logs will report valgrind errors.
The debugger used for extracting stack traces from core files (POSIX only) is currently not configurable; It is hard-coded to “gdb”, which should be somewhere in the PATH configured in environment.
Caveats
This chapter lists notable limitations that are not easy to overcome due to design choices.
- Concurrent scheduling
Concurrent scheduling requested via option CPUs is based on the “sharding” feature provided by Gtest framework. Unfortunately, Gtest only supports static case test partitioning, which means for example when using two CPUs, the set of test cases is split in two parts, of which the first is executed in one test process and the second in the second process.
One problem arises when the number of test cases is smaller than the number of CPUs. This typically occurs, when trying to run a single test case many times. Sharding would then only use a single CPU, as it does not consider repetitions in its “sharding” algorithm. GtestGui works around that by calculating if it is more efficient to partition test cases by repetition than by sharding, or if a combination of both is even better. In the mentioned example, it would not use sharding, but instead run half the number of repetitions in one process, and the second half in the second. For more complex configurations such as 10 repetitions of 9 test cases on 8 CPUs, a combination of both methods will be used.
A second problem that GtestGui cannot work around occurs when test cases have non-uniform execution time. As the “sharding” algorithm uses static partitioning solely based on the number of test cases per process, differences in execution times are not considered. For example, when two tests are scheduled for 100 times and test case A takes 1 second, but test case B only 1 millisecond, Gtest will still schedule all runs of A in the first process and all runs of B in the second process. Thus, the second process will sit idle for 99.9% of the total test execution time.
- Test case crashes
In a well-behaved test application, failures are normally reported via Gtest macros or exceptions. Thus, the failure is recorded and the next test case is executed. Sometimes however, a test case may have a bug that leads to a crash of the complete process. In this case all following test cases or test case repetitions are no longer executed.
Currently GtestGui will not attempt to restart tests after a crash, because it expects that the same test case will crash again and thus keep blocking following tests. It is not possible to disable the instable test, as this would interfere with partitioning by Gtest “sharding”, i.e. the set of test cases run by that test case would be altered. The only way around is for the use to manually disable the instable test case and then restart the test campaign.
- Overload of GUI by concurrent unit-testing
When using multiple CPUs for running very short-running test cases, such as typical unit-tests, the GUI application may be overloaded and thus appear unresponsive and hung. This is a result of Python’s implementation of multi-threading, which does not allow using more than one effective CPU due to exclusive locks in interpreter execution. Therefore, when parsing of test output streams takes 100% of a CPU, no time is left for updating the GUI even though separate threads are used.
For more minor constraints and ideas for enhancements see file TODO.txt, which is part of the package.
Files
- $HOME/.config/gtest_gui/gtest_gui.rc
This file stores parameters that are configured via the Configuration dialog. Additional it contains persistent state such as the list of previously loaded executable files and the size and position of resizable dialog windows.
- trace.NNNN/trace.NNNN
Output from test applications is stored to sub-directories files called “trace.” with a number appended. The number is an arbitrary identifier for the test executable whose output they contain. The directory contains a separate output file for each spawned test process. Files in this directory are removed when removing results in the Result log of the GUI. By default, traces containing only passed test case results are also cleaned upon exiting the GUI.
By default, the sub-directories are created in the current working directory where GtestGui is started. Another base directory may be specified in Configuration.
If multiple instances of the GUI are started, they will use the same directory for storage. For single-user systems this works well enough, as conflicts may occur only when pressing “Run” button concurrently in different instances. For multi-user setup it is not recommended to share the directory. (Seeing other user’s test results would be confusing anyway.)
- TEMP/gtest_gui_tmpXXX
A temporary directory at the default place used by the operating system for such purpose will be created, for for example for unpacking trace snippets for display, or exporting trace files to archives. A different directory is used by each instance of GtestGui. The directory is removed automatically when the GUI is closed. Note the latter may fail on some platforms if a trace browser application still has opened an exported trace file at the time of quitting the GUI.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
