Skip to main content

knots-test-complexity

A Rust-based test quality analyzer for C projects that validates unit tests have sufficient complexity to thoroughly exercise source code.

Motivation

Traditional code coverage metrics (line, branch, function) can be misleading. A test can achieve 100% branch coverage with simple assertions while missing critical edge cases like:

  • Overflow scenarios: uint16_t timer wrapping at 65535
  • Boundary conditions: Off-by-one errors at array bounds
  • State transitions: Complex state machines with temporal dependencies
  • Error paths: Multiple error conditions not all tested

This tool enforces that tests have sufficient cyclomatic complexity to exercise all logical paths and boundary value testing to catch edge cases.

Philosophy

"A test with lower complexity than its source code is likely not testing all scenarios."

Example: The Overflow Bug

// Source: Cyclomatic Complexity = 2
uint16_t timer_ms = 0;

void periodic_1ms() {
    timer_ms++;  // Overflows at 65535!
}

bool is_timeout(uint16_t start_ms, uint16_t duration_ms) {
    return (timer_ms - start_ms) >= duration_ms;
}

Traditional Coverage (100% line, 100% branch):

void test_timeout() {
    timer_ms = 0;
    TEST_ASSERT_TRUE(is_timeout(0, 100));   // Happy path
    TEST_ASSERT_FALSE(is_timeout(0, 1));    // Boundary
}
// PASSES coverage but MISSES overflow bug!

Tool Enforcement - Would detect:

  • Test complexity (2) barely meets source complexity (2)
  • Missing boundary tests: 0, 65535, wrap-around scenarios
  • Missing state variation: timer_ms at different values

Better Tests (Higher Complexity):

void test_timeout_boundaries() {
    // Boundary: timer at 0
    timer_ms = 0;
    TEST_ASSERT_TRUE(is_timeout(0, 100));

    // Boundary: timer near max
    timer_ms = 65530;
    TEST_ASSERT_TRUE(is_timeout(65520, 100));

    // CRITICAL: Overflow scenario
    timer_ms = 5;  // Wrapped from 65535
    TEST_ASSERT_TRUE(is_timeout(65530, 100));  // Catches overflow!

    // Multiple start/duration combinations
    for (int i = 0; i < 5; i++) {
        test_scenario(scenarios[i]);
    }
}
// Higher complexity test catches the bug!

Features

Core Metrics

  1. Test-to-Source Complexity Ratio

    • Aggregate cyclomatic complexity of all test functions
    • Compare to aggregate complexity of source functions
    • Default threshold: 70% (configurable)
  2. Boundary Value Detection

    • Detects integer types: uint8_t, uint16_t, uint32_t, int8_t, etc.
    • Identifies range checks: if (x > MAX), if (x < MIN)
    • Counts required boundary tests
    • Validates tests cover: MIN, MIN-1, MAX, MAX+1
  3. State Variable Tracking (Future Enhancement)

    • Identifies static, volatile, and global variables
    • Requires multiple test scenarios per state variable
    • Validates state transitions are tested

Output Modes

  • Warning Mode (default): Reports violations but doesn't fail pre-commit
  • Error Mode: Fails pre-commit on violations
  • Verbose Mode: Shows detailed per-function complexity breakdown

Building

cargo build --release --workspace

The binary will be at target/release/knots-test-complexity

Installation

From Source

cd knots
cargo build --release --workspace
# Binary is at target/release/knots-test-complexity

Add to your PATH or copy to a location in your PATH.

Pre-Commit Integration

Add to your project's .pre-commit-config.yaml:

repos:
  - repo: https://github.com/brandon-arrendondo/knots
    rev: v0.3.0  # Use specific version tag
    hooks:
      - id: test-complexity
        args:
          - --threshold=0.70
          - --boundary-threshold=0.80
          - --level=warn
          - --framework=ceedling
          - --test-dir=Test

Configuration Options:

  • --threshold=0.70: Minimum test-to-source complexity ratio (default: 0.70 = 70%)
  • --level=warn: Enforcement level (warn or error, default: warn)
  • --no-check-boundaries: Disable boundary value detection (enabled by default)
  • --verbose: Show detailed per-file analysis

Example: Strict Enforcement

args:
  - --threshold=0.80
  - --level=error
  - --verbose

Example: Warning Only (No Boundaries)

args:
  - --threshold=0.70
  - --level=warn
  - --no-check-boundaries

Usage

Command Line

Analyze a test file and its corresponding source:

knots-test-complexity Test/test_battery_service.c Core/Src/modules/battery_service/battery_service.c

With verbose output:

knots-test-complexity -v Test/test_battery_service.c Core/Src/modules/battery_service/battery_service.c

With custom thresholds:

knots-test-complexity \
  --threshold=0.70 \
  --boundary-threshold=0.80 \
  --level=error \
  Test/test_timer.c Core/Src/timer.c

Output Example

Analyzing Test Quality: test_battery_service.c
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Source File: battery_service.c
  Functions: 15
  Total Cyclomatic Complexity: 87
  Boundary Values Detected: 12
    - uint8_t variables: 4 (boundaries: 0, 255)
    - Range checks: 8 (if (x > MAX), etc.)

Test File: test_battery_service.c
  Functions: 58
  Total Cyclomatic Complexity: 74
  Boundary Tests Found: 15

Complexity Analysis:
  Test/Source Ratio: 85% ✓ (threshold: 70%)
  Test Complexity: 74
  Source Complexity: 87
  Ratio: 74/87 = 0.85

Boundary Analysis:
  Required Boundary Tests: 12
  Found Boundary Tests: 15 ✓
  Coverage: 125%

Result: ✓ PASS

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Failure Example

Analyzing Test Quality: test_lin_comm_service.c
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Source File: lin_comm_service.c
  Functions: 8
  Total Cyclomatic Complexity: 54
  Boundary Values Detected: 8

Test File: test_lin_comm_service.c
  Functions: 12
  Total Cyclomatic Complexity: 28
  Boundary Tests Found: 3

Complexity Analysis:
  Test/Source Ratio: 52% ✗ (threshold: 70%)
  Test Complexity: 28
  Source Complexity: 54
  Ratio: 28/54 = 0.52

Boundary Analysis:
  Required Boundary Tests: 8
  Found Boundary Tests: 3 ✗
  Missing Boundaries:
    - rxByteCounter: 0, 3, 11, 12 (RX_HEADER_SIZE boundaries)
    - rxFrameId: 0, 0x3F (FRAME_MASK boundary)

Recommendations:
  1. Add tests for edge cases and error paths
  2. Test boundary conditions: 0, max values, overflow
  3. Add state transition tests (4 static variables detected)
  4. Consider parametrized tests or loops in test code

Result: ✗ FAIL (--level=error)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Algorithm Details

1. Complexity Ratio Calculation

For each test file:
  1. Parse test file with tree-sitter-c
  2. Calculate cyclomatic complexity for all test functions
     - Count: if, while, for, switch, &&, ||, ?:
     - Include test helper functions
  3. Sum total test complexity

  4. Find corresponding source file
  5. Calculate cyclomatic complexity for all source functions
  6. Sum total source complexity

  7. Calculate ratio = test_complexity / source_complexity
  8. Compare ratio >= threshold (default 70%)

  9. Report: PASS or FAIL with recommendations

2. Boundary Value Detection

For source file:
  1. Find all integer type declarations
     - uint8_t → boundaries: 0, 255
     - uint16_t → boundaries: 0, 65535
     - int8_t → boundaries: -128, 127

  2. Find all range checks
     - if (x > MAX) → test MAX, MAX+1
     - if (x < MIN) → test MIN-1, MIN
     - if (x >= threshold) → test threshold-1, threshold

  3. Count required boundary tests

For test file:
  1. Find all numeric literals in assertions
  2. Match literals to source boundaries
  3. Count covered boundaries

  4. Report: boundary_coverage = found / required
  5. Warn if coverage < 100%

3. Test Helper Function Handling

Test helpers ARE included in complexity calculation:

// Helper complexity counts!
void simulate_frame(uint8_t id) {
    setup_mocks();
    if (id == SPECIAL) {  // +1 complexity
        special_handling();
    }
    verify_results();
}

// Test also counts
void test_multiple_frames() {
    for (int i = 0; i < 10; i++) {  // +1 complexity
        simulate_frame(i);  // Helper complexity included
    }
}
// Total test complexity = 2 (loop + helper's if)

This encourages well-structured tests with reusable helpers.

Integration with knots

knots-test-complexity complements knots:

Tool Purpose Applied To Metric
knots Source code quality Production code (.c, .h) McCabe & Cognitive complexity per function
knots-test-complexity Test quality Test code (test_*.c) Aggregate complexity ratio & boundary coverage

Example Combined Workflow:

- repo: local
  hooks:
    # Check source code complexity (per-function limits)
    - id: knots
      name: Code Complexity Check
      entry: hooks/pre-commit-wrapper.sh
      language: script
      files: \.(c|h)$
      exclude: ^Test/
      args: [--mccabe-threshold=15, --cognitive-threshold=15]

    # Check test quality (aggregate complexity ratio)
    - id: test-complexity
      name: Test Quality Check
      entry: hooks/test-complexity-wrapper.sh
      language: script
      files: ^Test/test_.*\.c$
      args: [--threshold=70, --level=warn]

Future: Unified Tool

A future enhancement could merge both tools:

# Unified complexity tool
complexity-check --source <file.c> --test <test_file.c>
  --source-mccabe-max=15
  --source-cognitive-max=15
  --test-ratio-min=0.70
  --check-boundaries

Dependencies

[dependencies]
tree-sitter = "0.22"
tree-sitter-c = "0.21"
anyhow = "1.0"
clap = { version = "4.5", features = ["derive"] }
regex = "1.10"

Project Structure

knots-test-complexity/
├── Cargo.toml                          # Rust project manifest
├── README.md                           # This file
├── src/
│   ├── main.rs                        # CLI entry point
│   ├── analyzer.rs                    # Test quality analyzer
│   ├── boundary.rs                    # Boundary value detector
│   └── reporter.rs                    # Output formatting
└── examples/
    ├── test_timer_good.c              # Example: sufficient complexity
    ├── test_timer_bad.c               # Example: insufficient complexity
    └── README.md                      # Example documentation

Testing

Run the test suite:

cargo test

Run with verbose output:

cargo test -- --nocapture

License

MIT License. See LICENSE file.

See Also

  • knots: Source code complexity analyzer
  • pmccabe: Industry-standard McCabe complexity tool (validation reference)
  • Cognitive Complexity: SonarSource specification
  • Mutation Testing: Alternative approach for test quality (future consideration)

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

knots_test_complexity-1.16.0.tar.gz (258.5 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

knots_test_complexity-1.16.0-py3-none-win_amd64.whl (4.1 MB view details)

Uploaded Python 3Windows x86-64

knots_test_complexity-1.16.0-py3-none-musllinux_1_2_x86_64.whl (4.5 MB view details)

Uploaded Python 3musllinux: musl 1.2+ x86-64

knots_test_complexity-1.16.0-py3-none-musllinux_1_2_aarch64.whl (4.4 MB view details)

Uploaded Python 3musllinux: musl 1.2+ ARM64

knots_test_complexity-1.16.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

knots_test_complexity-1.16.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.5 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ARM64

knots_test_complexity-1.16.0-py3-none-macosx_11_0_arm64.whl (4.5 MB view details)

Uploaded Python 3macOS 11.0+ ARM64

knots_test_complexity-1.16.0-py3-none-macosx_10_12_x86_64.whl (4.3 MB view details)

Uploaded Python 3macOS 10.12+ x86-64

File details

Details for the file knots_test_complexity-1.16.0.tar.gz.

File metadata

  • Download URL: knots_test_complexity-1.16.0.tar.gz
  • Upload date:
  • Size: 258.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for knots_test_complexity-1.16.0.tar.gz
Algorithm Hash digest
SHA256 4ab43ded405412a09c486343a12a4bea0ceeb2da45b2d1ee116465ba5628c591
MD5 f42651d3f5e88eece2c0485ccaa200eb
BLAKE2b-256 676630173df008f20a0b57941bbd5c927de40f263fe134a26449e7f0adcc55fc

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.16.0.tar.gz:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.16.0-py3-none-win_amd64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.16.0-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 b08e4548a8dc3a1c6b7ab68a4acc740d9537fe2f7a1a97658519b2ec1f04f30b
MD5 1c33e04772117cd994942b28320005b5
BLAKE2b-256 cb8d262d5a5c23575eb1bb26ff4dacb2bd8218719d3cde59f02fd772abb8b4ac

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.16.0-py3-none-win_amd64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.16.0-py3-none-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.16.0-py3-none-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 067cd5183a169251a3d5071acb3cbe83dc0f425e7fc6586bfe5b0b69a8885b78
MD5 c2aa080b69549b72a0e1420b07b0e1d8
BLAKE2b-256 589666b9fab46cf01862a0ffd6c8e78759e4e051db060e937aeb6dd09769a218

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.16.0-py3-none-musllinux_1_2_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.16.0-py3-none-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.16.0-py3-none-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 b50e216942b278f4d8e104abbe1af93a87ae412f2d47bae314b7a7fe0214184f
MD5 ff9aed1aeaa04526b3cb84c9824cd2c0
BLAKE2b-256 8245a63ecbd1968ad7b49c79db0ed1d2a6c18eed6cc28d8f49470882b4388ab3

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.16.0-py3-none-musllinux_1_2_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.16.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.16.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 33a42d83a0c453470eca5d4cdc3b99f2e448603a095cd0df86314a2bd361f098
MD5 3a796bb18f6f73418b71e0e5eb60fde7
BLAKE2b-256 1de2c961db7d4c40c7ca8bad7ad68d81ec101f84a1f22361d774cf5c5d6cdcec

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.16.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.16.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.16.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 3087b78e0b8068b4ad6b7ea9e3d48aba1e32615127519bf70ce84c4d0a828b48
MD5 fe8fdbd8f360aa1ba4eb405c69637fb4
BLAKE2b-256 a24e4d5d5628bd79c897b2923f32f8e09450484e157aba53d485894ffaceff4b

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.16.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.16.0-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.16.0-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 520923c27548858df07d745bf4f78885db569d3ab67cde3d826d1d844dc38b95
MD5 c305864482dccd6ddc260fc2ef51aa74
BLAKE2b-256 e797790c3d1468160c5893d19ae338ecc39f0481e55b1235e02eeedd14655834

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.16.0-py3-none-macosx_11_0_arm64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.16.0-py3-none-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.16.0-py3-none-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 0b619f6a636579a9ddef661ac7168fe750f57f7c7d46c5a55c1e11376d867ed8
MD5 9586545924ca53dcb13e23367b939eb5
BLAKE2b-256 f052de8e157c185c5676a948aa08c8b6d449c856eea6acac72ce6253fb1377af

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.16.0-py3-none-macosx_10_12_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

1.16.0 This release

8 files

1.15.1

8 files

1.15.0

8 files

1.14.0

8 files

1.13.1

8 files

1.13.0

8 files

1.12.0

8 files

1.11.0

8 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page