cfgcoverage

A Python package to analyze and verify structural coverage criteria for Python programs using control-flow graphs (CFGs). Currently: all-nodes, all-edges (and essential edges), edge pairs and prime paths.

V 0.1 - This is a very early, highly experimental version.

V 0.1.1 - Fixes module execution. Include new criteria: essential edges, edge pairs and prime paths. Still a very early, highly experimental version.

V 0.1.2 - Introduces --report argument.

V 0.1.3 - Fixes Prime Paths requirements calculation. Introduces --csv option and --f argument to set infeasible requirements.

V 0.1.4 - Fixes bugs ("for" cfg structure). Introduces --debug for csv command.

V 0.1.5 - Improves CFG visualization.

V 0.1.6 - Fixes bugs. Introduces argument --nopp.

Installation

pip install cfgcoverage

🚀 How to Use

Analyzes the file foo.py, builds its control-flow graphs, and stores the internal analysis data in foo.py.cfg.

python -m cfgcoverage foo.py

Same as before but skips Prime Path requirement computation to avoid combinatorial explosion.

python -m cfgcoverage --nopp foo.py

Executes foo.py with instrumentation and records covered blocks and edges.

python -m cfgcoverage --run foo.py

Executes foo.py with arguments ['bar', '1972'] and records the covered structural requirements.

python -m cfgcoverage --run --args "bar 1972" foo.py

Runs foo.py using the test cases defined in test_foo.py.
The --unittest argument can be used multiple times in a command line.

Both --run and --unittest execute cumulatively relative to previous runs. To reset the coverage count, the cfgcoverage command must be run without arguments, as shown previously.

python -m cfgcoverage --unittest test_foo.py foo.py
python -m cfgcoverage --unittest tests/test_foo.py src/foo.py
python -m cfgcoverage --unittest test_a.py --unittest test_b.py foo.py

You can also use one or more --path <folder path> arguments to include additional folders in sys.path when executing unit tests. In this way, imports will also be searched in these folders.

python -m cfgcoverage --path src --unittest tests/test_foo.py src/foo.py
python -m cfgcoverage --path src --path tests --unittest tests/test_foo.py src/foo.py

Creates an HTML report in html/foo.py/index.html using the data stored in foo.py.cfg.

python -m cfgcoverage --html foo.py

Generate a text report using the data stored in foo.py.cfg

python -m cfgcoverage --report foo.py

Generate a report with only the total coverage numbers

python -m cfgcoverage --report --total foo.py

Remove one or some of the requirements from all test criteria. For example, remove(5,2) means that the edge will be removed from the all-edges criterion and that all paths containing that arc will also be removed from the other criteria.

If an edge is removed along with the --recalc parameter, then the requirements of the other criteria are recalculated, including the essential edges criterion. And, in that case, the execution data is discarded.

The first number indicates the function from which the requirement is to be removed. The remaining numbers represent the edge or path that should be removed. The --f argument can be used to read infeasible requirements from a text file. This file must contain one infeasible requirement per line and must not contain empty lines, including at the end of the file.

python -m cfgcoverage --infeasible "1 5 2" "2 7 9 5 1" foo.py
python -m cfgcoverage --infeasible "1 5 2" --recalc foo.py
python -m cfgcoverage --infeasible --f infeasible_foo.txt foo.py

Creates CSV files that show, for each test requirement (spreadsheet rows), whether or not they are covered by the test cases (spreadsheet columns) contained in the unittest files mentioned in the command line. Note that coverage is not updated in the ".cfg" file; it is only recorded in the CSV files. More than one test file can be used in the same command. Option --debug shows the traces for each test case in the test files.

python -m cfgcoverage --csv --unittest test_foo.py foo.py
python -m cfgcoverage --csv --unittest test_a.py --unittest test_b.py foo.py
python -m cfgcoverage --csv --unittest test_foo.py --debug foo.py

🔍 Example

python3 -m cfgcoverage example.py

Example output:

Function: foo -- 1
Function: foo.inner -- 2
Run time: 0.00052

python3 -m cfgcoverage --run example.py

Example output:

Run time: 0.00071

python3 -m cfgcoverage --unittest test_example.py example.py

Example output:

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Run time: 0.00110

python3 -m cfgcoverage --html example.py

Example output:

File generated at: html/example.py/index.html

📊 Coverage Criteria

The current HTML report includes the following criteria for each analyzed function:

• Blocks
• Edges
• Essential edges
• Edge Pairs
• Prime paths

📝 Notes

1. The analysis stores its internal data in a file named <source_file>.cfg.
2. HTML reports depend on previously generated .cfg data.
3. The tool analyzes functions and nested functions found in the source file.
4. For each execution, the coverage data from previous executions is accumulated, not replaced.
5. The HTML report is generated per source file and organized under the html/ folder.
6. To use visualization features, install Graphviz on your system and make sure the dot executable is available on your PATH.

👤 Author

Marcio Delamaro

📄 License

MIT

🤝 Contributions

Feel free to contact the author.