Skip to main content

knots-test-complexity

A Rust-based test quality analyzer for C projects that validates unit tests have sufficient complexity to thoroughly exercise source code.

Motivation

Traditional code coverage metrics (line, branch, function) can be misleading. A test can achieve 100% branch coverage with simple assertions while missing critical edge cases like:

  • Overflow scenarios: uint16_t timer wrapping at 65535
  • Boundary conditions: Off-by-one errors at array bounds
  • State transitions: Complex state machines with temporal dependencies
  • Error paths: Multiple error conditions not all tested

This tool enforces that tests have sufficient cyclomatic complexity to exercise all logical paths and boundary value testing to catch edge cases.

Philosophy

"A test with lower complexity than its source code is likely not testing all scenarios."

Example: The Overflow Bug

// Source: Cyclomatic Complexity = 2
uint16_t timer_ms = 0;

void periodic_1ms() {
    timer_ms++;  // Overflows at 65535!
}

bool is_timeout(uint16_t start_ms, uint16_t duration_ms) {
    return (timer_ms - start_ms) >= duration_ms;
}

Traditional Coverage (100% line, 100% branch):

void test_timeout() {
    timer_ms = 0;
    TEST_ASSERT_TRUE(is_timeout(0, 100));   // Happy path
    TEST_ASSERT_FALSE(is_timeout(0, 1));    // Boundary
}
// PASSES coverage but MISSES overflow bug!

Tool Enforcement - Would detect:

  • Test complexity (2) barely meets source complexity (2)
  • Missing boundary tests: 0, 65535, wrap-around scenarios
  • Missing state variation: timer_ms at different values

Better Tests (Higher Complexity):

void test_timeout_boundaries() {
    // Boundary: timer at 0
    timer_ms = 0;
    TEST_ASSERT_TRUE(is_timeout(0, 100));

    // Boundary: timer near max
    timer_ms = 65530;
    TEST_ASSERT_TRUE(is_timeout(65520, 100));

    // CRITICAL: Overflow scenario
    timer_ms = 5;  // Wrapped from 65535
    TEST_ASSERT_TRUE(is_timeout(65530, 100));  // Catches overflow!

    // Multiple start/duration combinations
    for (int i = 0; i < 5; i++) {
        test_scenario(scenarios[i]);
    }
}
// Higher complexity test catches the bug!

Features

Core Metrics

  1. Test-to-Source Complexity Ratio

    • Aggregate cyclomatic complexity of all test functions
    • Compare to aggregate complexity of source functions
    • Default threshold: 70% (configurable)
  2. Boundary Value Detection

    • Detects integer types: uint8_t, uint16_t, uint32_t, int8_t, etc.
    • Identifies range checks: if (x > MAX), if (x < MIN)
    • Counts required boundary tests
    • Validates tests cover: MIN, MIN-1, MAX, MAX+1
  3. State Variable Tracking (Future Enhancement)

    • Identifies static, volatile, and global variables
    • Requires multiple test scenarios per state variable
    • Validates state transitions are tested

Output Modes

  • Warning Mode (default): Reports violations but doesn't fail pre-commit
  • Error Mode: Fails pre-commit on violations
  • Verbose Mode: Shows detailed per-function complexity breakdown

Building

cargo build --release --workspace

The binary will be at target/release/knots-test-complexity

Installation

From Source

cd knots
cargo build --release --workspace
# Binary is at target/release/knots-test-complexity

Add to your PATH or copy to a location in your PATH.

Pre-Commit Integration

Add to your project's .pre-commit-config.yaml:

repos:
  - repo: https://github.com/brandon-arrendondo/knots
    rev: v0.3.0  # Use specific version tag
    hooks:
      - id: test-complexity
        args:
          - --threshold=0.70
          - --boundary-threshold=0.80
          - --level=warn
          - --framework=ceedling
          - --test-dir=Test

Configuration Options:

  • --threshold=0.70: Minimum test-to-source complexity ratio (default: 0.70 = 70%)
  • --level=warn: Enforcement level (warn or error, default: warn)
  • --no-check-boundaries: Disable boundary value detection (enabled by default)
  • --verbose: Show detailed per-file analysis

Example: Strict Enforcement

args:
  - --threshold=0.80
  - --level=error
  - --verbose

Example: Warning Only (No Boundaries)

args:
  - --threshold=0.70
  - --level=warn
  - --no-check-boundaries

Usage

Command Line

Analyze a test file and its corresponding source:

knots-test-complexity Test/test_battery_service.c Core/Src/modules/battery_service/battery_service.c

With verbose output:

knots-test-complexity -v Test/test_battery_service.c Core/Src/modules/battery_service/battery_service.c

With custom thresholds:

knots-test-complexity \
  --threshold=0.70 \
  --boundary-threshold=0.80 \
  --level=error \
  Test/test_timer.c Core/Src/timer.c

Output Example

Analyzing Test Quality: test_battery_service.c
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Source File: battery_service.c
  Functions: 15
  Total Cyclomatic Complexity: 87
  Boundary Values Detected: 12
    - uint8_t variables: 4 (boundaries: 0, 255)
    - Range checks: 8 (if (x > MAX), etc.)

Test File: test_battery_service.c
  Functions: 58
  Total Cyclomatic Complexity: 74
  Boundary Tests Found: 15

Complexity Analysis:
  Test/Source Ratio: 85% ✓ (threshold: 70%)
  Test Complexity: 74
  Source Complexity: 87
  Ratio: 74/87 = 0.85

Boundary Analysis:
  Required Boundary Tests: 12
  Found Boundary Tests: 15 ✓
  Coverage: 125%

Result: ✓ PASS

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Failure Example

Analyzing Test Quality: test_lin_comm_service.c
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Source File: lin_comm_service.c
  Functions: 8
  Total Cyclomatic Complexity: 54
  Boundary Values Detected: 8

Test File: test_lin_comm_service.c
  Functions: 12
  Total Cyclomatic Complexity: 28
  Boundary Tests Found: 3

Complexity Analysis:
  Test/Source Ratio: 52% ✗ (threshold: 70%)
  Test Complexity: 28
  Source Complexity: 54
  Ratio: 28/54 = 0.52

Boundary Analysis:
  Required Boundary Tests: 8
  Found Boundary Tests: 3 ✗
  Missing Boundaries:
    - rxByteCounter: 0, 3, 11, 12 (RX_HEADER_SIZE boundaries)
    - rxFrameId: 0, 0x3F (FRAME_MASK boundary)

Recommendations:
  1. Add tests for edge cases and error paths
  2. Test boundary conditions: 0, max values, overflow
  3. Add state transition tests (4 static variables detected)
  4. Consider parametrized tests or loops in test code

Result: ✗ FAIL (--level=error)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Algorithm Details

1. Complexity Ratio Calculation

For each test file:
  1. Parse test file with tree-sitter-c
  2. Calculate cyclomatic complexity for all test functions
     - Count: if, while, for, switch, &&, ||, ?:
     - Include test helper functions
  3. Sum total test complexity

  4. Find corresponding source file
  5. Calculate cyclomatic complexity for all source functions
  6. Sum total source complexity

  7. Calculate ratio = test_complexity / source_complexity
  8. Compare ratio >= threshold (default 70%)

  9. Report: PASS or FAIL with recommendations

2. Boundary Value Detection

For source file:
  1. Find all integer type declarations
     - uint8_t → boundaries: 0, 255
     - uint16_t → boundaries: 0, 65535
     - int8_t → boundaries: -128, 127

  2. Find all range checks
     - if (x > MAX) → test MAX, MAX+1
     - if (x < MIN) → test MIN-1, MIN
     - if (x >= threshold) → test threshold-1, threshold

  3. Count required boundary tests

For test file:
  1. Find all numeric literals in assertions
  2. Match literals to source boundaries
  3. Count covered boundaries

  4. Report: boundary_coverage = found / required
  5. Warn if coverage < 100%

3. Test Helper Function Handling

Test helpers ARE included in complexity calculation:

// Helper complexity counts!
void simulate_frame(uint8_t id) {
    setup_mocks();
    if (id == SPECIAL) {  // +1 complexity
        special_handling();
    }
    verify_results();
}

// Test also counts
void test_multiple_frames() {
    for (int i = 0; i < 10; i++) {  // +1 complexity
        simulate_frame(i);  // Helper complexity included
    }
}
// Total test complexity = 2 (loop + helper's if)

This encourages well-structured tests with reusable helpers.

Integration with knots

knots-test-complexity complements knots:

Tool Purpose Applied To Metric
knots Source code quality Production code (.c, .h) McCabe & Cognitive complexity per function
knots-test-complexity Test quality Test code (test_*.c) Aggregate complexity ratio & boundary coverage

Example Combined Workflow:

- repo: local
  hooks:
    # Check source code complexity (per-function limits)
    - id: knots
      name: Code Complexity Check
      entry: hooks/pre-commit-wrapper.sh
      language: script
      files: \.(c|h)$
      exclude: ^Test/
      args: [--mccabe-threshold=15, --cognitive-threshold=15]

    # Check test quality (aggregate complexity ratio)
    - id: test-complexity
      name: Test Quality Check
      entry: hooks/test-complexity-wrapper.sh
      language: script
      files: ^Test/test_.*\.c$
      args: [--threshold=70, --level=warn]

Future: Unified Tool

A future enhancement could merge both tools:

# Unified complexity tool
complexity-check --source <file.c> --test <test_file.c>
  --source-mccabe-max=15
  --source-cognitive-max=15
  --test-ratio-min=0.70
  --check-boundaries

Dependencies

[dependencies]
tree-sitter = "0.22"
tree-sitter-c = "0.21"
anyhow = "1.0"
clap = { version = "4.5", features = ["derive"] }
regex = "1.10"

Project Structure

knots-test-complexity/
├── Cargo.toml                          # Rust project manifest
├── README.md                           # This file
├── src/
│   ├── main.rs                        # CLI entry point
│   ├── analyzer.rs                    # Test quality analyzer
│   ├── boundary.rs                    # Boundary value detector
│   └── reporter.rs                    # Output formatting
└── examples/
    ├── test_timer_good.c              # Example: sufficient complexity
    ├── test_timer_bad.c               # Example: insufficient complexity
    └── README.md                      # Example documentation

Testing

Run the test suite:

cargo test

Run with verbose output:

cargo test -- --nocapture

License

MIT License. See LICENSE file.

See Also

  • knots: Source code complexity analyzer
  • pmccabe: Industry-standard McCabe complexity tool (validation reference)
  • Cognitive Complexity: SonarSource specification
  • Mutation Testing: Alternative approach for test quality (future consideration)

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

knots_test_complexity-1.15.1.tar.gz (251.3 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

knots_test_complexity-1.15.1-py3-none-win_amd64.whl (4.1 MB view details)

Uploaded Python 3Windows x86-64

knots_test_complexity-1.15.1-py3-none-musllinux_1_2_x86_64.whl (4.6 MB view details)

Uploaded Python 3musllinux: musl 1.2+ x86-64

knots_test_complexity-1.15.1-py3-none-musllinux_1_2_aarch64.whl (4.4 MB view details)

Uploaded Python 3musllinux: musl 1.2+ ARM64

knots_test_complexity-1.15.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

knots_test_complexity-1.15.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.5 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ARM64

knots_test_complexity-1.15.1-py3-none-macosx_11_0_arm64.whl (4.5 MB view details)

Uploaded Python 3macOS 11.0+ ARM64

knots_test_complexity-1.15.1-py3-none-macosx_10_12_x86_64.whl (4.3 MB view details)

Uploaded Python 3macOS 10.12+ x86-64

File details

Details for the file knots_test_complexity-1.15.1.tar.gz.

File metadata

  • Download URL: knots_test_complexity-1.15.1.tar.gz
  • Upload date:
  • Size: 251.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for knots_test_complexity-1.15.1.tar.gz
Algorithm Hash digest
SHA256 1c0e364ccdcb66323471db51288d97a9fc562cb486e2f6d2b3cd10081325600f
MD5 317dd3f36988ee787ceef39ba3b2c08b
BLAKE2b-256 bfa3dcbd9136ec700ea5c26b6b84c4a0f246d2c6df655be89460f78f7925c768

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.15.1.tar.gz:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.15.1-py3-none-win_amd64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.15.1-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 c65f0be00667d151463b5d109e30acd450cb59f4aff4f52145937b5e42e62b78
MD5 89b8239c88de2a2bbdaa14e6456186aa
BLAKE2b-256 60cf67b918131f83c20fd817bcc93833a773b3002ea7b757ab0519086843d24c

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.15.1-py3-none-win_amd64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.15.1-py3-none-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.15.1-py3-none-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 0b901596837e92246d1fc17fbfdc48587bec263436d913dca925be0c5fc8e414
MD5 56e6acadd046415a8b6320e59ad2935b
BLAKE2b-256 b409039193ef815490cade0b9d1363eb3e760a146db7fb9e5ec86dcc43be970b

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.15.1-py3-none-musllinux_1_2_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.15.1-py3-none-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.15.1-py3-none-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 be258c50f786b7f7da45d4d020fc32dec4db900b7630a21b83291b9e813d5083
MD5 bf6e9eecaba0027f6acd0826a43cd132
BLAKE2b-256 9df32794c402394b8d530dcd15e0314f8911e3c27021c7afa1d63a6686b9278c

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.15.1-py3-none-musllinux_1_2_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.15.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.15.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 167bd4a5f2c69947afe04c7798f17b48b526dfaf9d939bc6c01a7cadd0859fa5
MD5 2daffe9a237d13c8bb8c975342ee3fc3
BLAKE2b-256 bf30c5d20d778fdf0de9098cc7cb26742d99e9eecff4fe7daf58f4299771f361

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.15.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.15.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.15.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 82202fbb84af1cfa50f7477fb3df577d4e2f6c50a920a930428a36376162a1ac
MD5 1744f1202e30239d63ac3481f50863d7
BLAKE2b-256 a0bcd47ea9f2f85393c556dbca3089089632eb1cbf2cd42114c8bb8e3015cbb1

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.15.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.15.1-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.15.1-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 be1d5ac387b0e8cf19c68a59aff5f233bb248ada79b72e9fce1418e2f5e27322
MD5 d8331001aabf8fbcf9ff079d8ab4446a
BLAKE2b-256 0657143ec0aad4483be0a9a6b94a263268981b5aa69086474a564e4c48b3ee4b

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.15.1-py3-none-macosx_11_0_arm64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.15.1-py3-none-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.15.1-py3-none-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 2fcb0c49f9276dc36828b62f8a4c0141823d296063a26a9321994ee27765de11
MD5 d4bd4e1d9d187e2ea679b9f2e6a4e993
BLAKE2b-256 c4bf094c9cc36d3df65d46124ff97cb26c9dc711c843e46524e341062d5bda07

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.15.1-py3-none-macosx_10_12_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.16.0

8 files

This release

1.15.1 This release

8 files

1.15.0

8 files

1.14.0

8 files

1.13.1

8 files

1.13.0

8 files

1.12.0

8 files

1.11.0

8 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page