differential-coverage 1.1.0

Compute differential coverage, a better way of comparing testing tools.

Differential Coverage

A better way of comparing testing tools.

Why?

Looking only at absolute (code) coverage when comparing testing tools loses a lot of information: Do all approaches cover the same blocks? Or just the same total count? How different is their coverage?

Here, differential coverage can help: By comparing what parts of the target is covered by each approach, we get much better insight into what is actually happening.

What?

In principle, differential coverage measures how much of approach $a_2$'s coverage can also be covered by approach $a_1$. This can be done in multiple ways:

• relcov does exactly this. It is an asymetrical measure between coverage of two approaches. If the approach relies on randomness (such as most fuzzers), it can aggregate over multiple trials of these.
• relscore aggregates all this data to an order of all approaches. This measure, compared to simple coverage numbers, takes into account and values higher approaches that can cover code that no other approach covers. However, compared to relcov, this hides a lot of information in this aggregation.

For more precise definitions, including formulas, look at DEFINITIONS.md.

How?

Campaign data lives in one directory per approach, with one file per trial. Two input formats are supported:

afl-showmap

Files contain <edge_id>: <count> rows; only whether count ≥ 1 matters. Edge IDs are opaque strings from the fuzzer bitmap (e.g. 12345).

coverage_data
|-- approach_1
|   |-- trial_1.out
|   |-- trial_2.out
|   `-- trial_3.out
|-- approach_2
|   |-- trial_1.out
|   `-- trial_2.out
|   `-- trial_3.out
`-- seeds
    `-- seeds.out

llvm-cov export JSON

Same layout, with JSON trial files from llvm-cov export (LLVM's -format=text produces JSON):

coverage_data
|-- approach_1
|   |-- trial_1.json
|   `-- trial_2.json
`-- seeds
    `-- seeds.json

Generate each trial file with:

llvm-cov export \
  -instr-profile=default.profdata \
  -object=./my_binary \
  -format=text \
  > trial_1.json

Run once per trial (merge profdata first if a trial replays multiple inputs). Each export is parsed at branch granularity when available, otherwise code-region blocks, otherwise regions. Source paths in edge IDs are reduced to basenames (e.g. calc.c:18:9-18:17:true).

cov-analysis

For fuzzer campaigns, cov-analysis can generate trial JSON for you: it replays AFL++, libFuzzer, libafl, or honggfuzz output through an instrumented binary and writes an llvm-cov export to <fuzzer-out>/cov/coverage.json.

cov-analysis build make -j$(nproc)
cov-analysis -d /path/to/fuzzer-out/ -e "./cov @@"

Copy one coverage.json per trial into your campaign directory (same layout as above). The format matches llvm-cov export -format=text.

Edge IDs and mixing formats

All approaches in one campaign must use the same input format and the same edge ID scheme. afl-showmap edge IDs (numeric bitmap indices) and llvm-cov edge IDs (source locations) are not comparable — do not mix them in one campaign.

llvm-cov edge IDs use source basenames only, but you still need the same binary/build when generating exports for all trials in a campaign.

Format is detected from file content (auto, default). Override with --input-format if needed.

Installation

Install differential-coverage directly from pypi:

pip install differential-coverage

# if you need latex output support
pip install differential-coverage[latex]

Command Line Interface

Generally, the command line interface follows the following structure:

differential-coverage {relcov,relscore} [--input-format {auto,afl-showmap,llvm-cov}] <campaign-dir>

Examples:

differential-coverage relscore ./coverage_data
differential-coverage --input-format llvm-cov relcov ./coverage_data
differential-coverage --output csv relscore ./coverage_data

There are more options available, e.g. for output as csv or (colored) LaTeX table:

differential-coverage --help

API

from differential_coverage import (
    DifferentialCoverage,
    ApproachData,
    CollectionReducer,
    ValueReducer,
)

# last part (EdgeId) can be any comparable type, e.g. str, int
dc: DifferentialCoverage[str, str, Any]

# read from campaign directory, see above for structure
dc = DifferentialCoverage.from_campaign_dir("<your-input-dir>")
# or from a memory structure
dc = DifferentialCoverage(
    {
        "approach_1": {"trial_1": {1, 2}, "trial_2": {1, 3}},
        "approach_2": {"trial_1": {1, 3}, "trial_2": {1, 3}},
        "approach_3": {"trial_1": {1, 2, 3}, "trial_2": {1, 2, 3}},
        "seeds": {"seeds": {1}},
    }
)

a1_name: str
a1_data: ApproachData[str, Any]
a2_name: str
a2_data: ApproachData[str, Any]

for a1_name, a1_data in dc.approaches.items():
    for a2_name, a2_data in dc.approaches.items():
        relcov: float = a1_data.relcov(
            a2_data,
            value_reducer=ValueReducer.MEDIAN,  # how to reduce relcov values from multiple trials from a1
            collection_reducer=CollectionReducer.UNION,  # how to reduce the edges from multiple trials from a2
        )
        print(f"relcov {a1_name} vs {a2_name}: {relcov}")

relscores: dict[str, float] = dc.relscores()
for approach_name, relscore in relscores.items():
    print(f"relscore {approach_name}: {relscore}")

Hints

When using differential coverage to compare different testing approaches, you may want to do the following:

• Use this approach early on in your development process already, it has helped me uncover problems with my evaluation setup. And you want to discover those before having to redo your entire evaluation.
• For approaches that rely on radomness, such as most fuzzers, run multiple trials! This is generally necessary, not just for differential coverage[^sok].
• For approaches that rely on some sort of input corpus, such as some fuzzers, record coverage of the target when passed just the input corpus, and add that to your data. This allows analyzing how much extra coverage a fuzzer reaches, and how the corpus itself compares to other fuzzers.
• Related: If you compare an approach with different seed corpora, add coverage maps from all input corpora to your table, to compare them within each other, and against the approaches' results based on them.

Development

Install dev dependencies and set up the pre-commit hook (runs a couple of checks before committing):

pip install -e ".[dev]"
pre-commit install

[^sok] SoK: Prudent Evaluation Practices for Fuzzing, https://ieeexplore.ieee.org/document/10646824