knots-test-complexity

A Rust-based test quality analyzer for C projects that validates unit tests have sufficient complexity to thoroughly exercise source code.

Motivation

Traditional code coverage metrics (line, branch, function) can be misleading. A test can achieve 100% branch coverage with simple assertions while missing critical edge cases like:

  • Overflow scenarios: uint16_t timer wrapping at 65535
  • Boundary conditions: Off-by-one errors at array bounds
  • State transitions: Complex state machines with temporal dependencies
  • Error paths: Multiple error conditions not all tested

This tool enforces that tests have sufficient cyclomatic complexity to exercise all logical paths and boundary value testing to catch edge cases.

Philosophy

"A test with lower complexity than its source code is likely not testing all scenarios."

Example: The Overflow Bug

// Source: Cyclomatic Complexity = 2
uint16_t timer_ms = 0;

void periodic_1ms() {
    timer_ms++;  // Overflows at 65535!
}

bool is_timeout(uint16_t start_ms, uint16_t duration_ms) {
    return (timer_ms - start_ms) >= duration_ms;
}

Traditional Coverage (100% line, 100% branch):

void test_timeout() {
    timer_ms = 0;
    TEST_ASSERT_TRUE(is_timeout(0, 100));   // Happy path
    TEST_ASSERT_FALSE(is_timeout(0, 1));    // Boundary
}
// PASSES coverage but MISSES overflow bug!

Tool Enforcement - Would detect:

  • Test complexity (2) barely meets source complexity (2)
  • Missing boundary tests: 0, 65535, wrap-around scenarios
  • Missing state variation: timer_ms at different values

Better Tests (Higher Complexity):

void test_timeout_boundaries() {
    // Boundary: timer at 0
    timer_ms = 0;
    TEST_ASSERT_TRUE(is_timeout(0, 100));

    // Boundary: timer near max
    timer_ms = 65530;
    TEST_ASSERT_TRUE(is_timeout(65520, 100));

    // CRITICAL: Overflow scenario
    timer_ms = 5;  // Wrapped from 65535
    TEST_ASSERT_TRUE(is_timeout(65530, 100));  // Catches overflow!

    // Multiple start/duration combinations
    for (int i = 0; i < 5; i++) {
        test_scenario(scenarios[i]);
    }
}
// Higher complexity test catches the bug!

Features

Core Metrics

  1. Test-to-Source Complexity Ratio

    • Aggregate cyclomatic complexity of all test functions
    • Compare to aggregate complexity of source functions
    • Default threshold: 70% (configurable)

  2. Boundary Value Detection

    • Detects integer types: uint8_t, uint16_t, uint32_t, int8_t, etc.
    • Identifies range checks: if (x > MAX), if (x < MIN)
    • Counts required boundary tests
    • Validates tests cover: MIN, MIN-1, MAX, MAX+1

  3. State Variable Tracking (Future Enhancement)

    • Identifies static, volatile, and global variables
    • Requires multiple test scenarios per state variable
    • Validates state transitions are tested

Output Modes

  • Warning Mode (default): Reports violations but doesn't fail pre-commit
  • Error Mode: Fails pre-commit on violations
  • Verbose Mode: Shows detailed per-function complexity breakdown

Building

cargo build --release --workspace

The binary will be at target/release/knots-test-complexity

Installation

From Source

cd knots
cargo build --release --workspace
# Binary is at target/release/knots-test-complexity

Add to your PATH or copy to a location in your PATH.

Pre-Commit Integration

Add to your project's .pre-commit-config.yaml:

repos:
  - repo: https://github.com/brandon-arrendondo/knots
    rev: v0.3.0  # Use specific version tag
    hooks:
      - id: test-complexity
        args:
          - --threshold=0.70
          - --boundary-threshold=0.80
          - --level=warn
          - --framework=ceedling
          - --test-dir=Test

Configuration Options:

  • --threshold=0.70: Minimum test-to-source complexity ratio (default: 0.70 = 70%)
  • --level=warn: Enforcement level (warn or error, default: warn)
  • --no-check-boundaries: Disable boundary value detection (enabled by default)
  • --verbose: Show detailed per-file analysis

Example: Strict Enforcement

args:
  - --threshold=0.80
  - --level=error
  - --verbose

Example: Warning Only (No Boundaries)

args:
  - --threshold=0.70
  - --level=warn
  - --no-check-boundaries

Usage

Command Line

Analyze a test file and its corresponding source:

knots-test-complexity Test/test_battery_service.c Core/Src/modules/battery_service/battery_service.c

With verbose output:

knots-test-complexity -v Test/test_battery_service.c Core/Src/modules/battery_service/battery_service.c

With custom thresholds:

knots-test-complexity \
  --threshold=0.70 \
  --boundary-threshold=0.80 \
  --level=error \
  Test/test_timer.c Core/Src/timer.c

Output Example

Analyzing Test Quality: test_battery_service.c
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Source File: battery_service.c
  Functions: 15
  Total Cyclomatic Complexity: 87
  Boundary Values Detected: 12
    - uint8_t variables: 4 (boundaries: 0, 255)
    - Range checks: 8 (if (x > MAX), etc.)

Test File: test_battery_service.c
  Functions: 58
  Total Cyclomatic Complexity: 74
  Boundary Tests Found: 15

Complexity Analysis:
  Test/Source Ratio: 85% ✓ (threshold: 70%)
  Test Complexity: 74
  Source Complexity: 87
  Ratio: 74/87 = 0.85

Boundary Analysis:
  Required Boundary Tests: 12
  Found Boundary Tests: 15 ✓
  Coverage: 125%

Result: ✓ PASS

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Failure Example

Analyzing Test Quality: test_lin_comm_service.c
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Source File: lin_comm_service.c
  Functions: 8
  Total Cyclomatic Complexity: 54
  Boundary Values Detected: 8

Test File: test_lin_comm_service.c
  Functions: 12
  Total Cyclomatic Complexity: 28
  Boundary Tests Found: 3

Complexity Analysis:
  Test/Source Ratio: 52% ✗ (threshold: 70%)
  Test Complexity: 28
  Source Complexity: 54
  Ratio: 28/54 = 0.52

Boundary Analysis:
  Required Boundary Tests: 8
  Found Boundary Tests: 3 ✗
  Missing Boundaries:
    - rxByteCounter: 0, 3, 11, 12 (RX_HEADER_SIZE boundaries)
    - rxFrameId: 0, 0x3F (FRAME_MASK boundary)

Recommendations:
  1. Add tests for edge cases and error paths
  2. Test boundary conditions: 0, max values, overflow
  3. Add state transition tests (4 static variables detected)
  4. Consider parametrized tests or loops in test code

Result: ✗ FAIL (--level=error)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Algorithm Details

1. Complexity Ratio Calculation

For each test file:
  1. Parse test file with tree-sitter-c
  2. Calculate cyclomatic complexity for all test functions
     - Count: if, while, for, switch, &&, ||, ?:
     - Include test helper functions
  3. Sum total test complexity

  4. Find corresponding source file
  5. Calculate cyclomatic complexity for all source functions
  6. Sum total source complexity

  7. Calculate ratio = test_complexity / source_complexity
  8. Compare ratio >= threshold (default 70%)

  9. Report: PASS or FAIL with recommendations

2. Boundary Value Detection

For source file:
  1. Find all integer type declarations
     - uint8_t → boundaries: 0, 255
     - uint16_t → boundaries: 0, 65535
     - int8_t → boundaries: -128, 127

  2. Find all range checks
     - if (x > MAX) → test MAX, MAX+1
     - if (x < MIN) → test MIN-1, MIN
     - if (x >= threshold) → test threshold-1, threshold

  3. Count required boundary tests

For test file:
  1. Find all numeric literals in assertions
  2. Match literals to source boundaries
  3. Count covered boundaries

  4. Report: boundary_coverage = found / required
  5. Warn if coverage < 100%

3. Test Helper Function Handling

Test helpers ARE included in complexity calculation:

// Helper complexity counts!
void simulate_frame(uint8_t id) {
    setup_mocks();
    if (id == SPECIAL) {  // +1 complexity
        special_handling();
    }
    verify_results();
}

// Test also counts
void test_multiple_frames() {
    for (int i = 0; i < 10; i++) {  // +1 complexity
        simulate_frame(i);  // Helper complexity included
    }
}
// Total test complexity = 2 (loop + helper's if)

This encourages well-structured tests with reusable helpers.

Integration with knots

knots-test-complexity complements knots:

Tool Purpose Applied To Metric
knots Source code quality Production code (.c, .h) McCabe & Cognitive complexity per function
knots-test-complexity Test quality Test code (test_*.c) Aggregate complexity ratio & boundary coverage

Example Combined Workflow:

- repo: local
  hooks:
    # Check source code complexity (per-function limits)
    - id: knots
      name: Code Complexity Check
      entry: hooks/pre-commit-wrapper.sh
      language: script
      files: \.(c|h)$
      exclude: ^Test/
      args: [--mccabe-threshold=15, --cognitive-threshold=15]

    # Check test quality (aggregate complexity ratio)
    - id: test-complexity
      name: Test Quality Check
      entry: hooks/test-complexity-wrapper.sh
      language: script
      files: ^Test/test_.*\.c$
      args: [--threshold=70, --level=warn]

Future: Unified Tool

A future enhancement could merge both tools:

# Unified complexity tool
complexity-check --source <file.c> --test <test_file.c>
  --source-mccabe-max=15
  --source-cognitive-max=15
  --test-ratio-min=0.70
  --check-boundaries

Dependencies

[dependencies]
tree-sitter = "0.22"
tree-sitter-c = "0.21"
anyhow = "1.0"
clap = { version = "4.5", features = ["derive"] }
regex = "1.10"

Project Structure

knots-test-complexity/
├── Cargo.toml                          # Rust project manifest
├── README.md                           # This file
├── src/
│   ├── main.rs                        # CLI entry point
│   ├── analyzer.rs                    # Test quality analyzer
│   ├── boundary.rs                    # Boundary value detector
│   └── reporter.rs                    # Output formatting
└── examples/
    ├── test_timer_good.c              # Example: sufficient complexity
    ├── test_timer_bad.c               # Example: insufficient complexity
    └── README.md                      # Example documentation

Testing

Run the test suite:

cargo test

Run with verbose output:

cargo test -- --nocapture

License

MIT License. See LICENSE file.

See Also

  • knots: Source code complexity analyzer
  • pmccabe: Industry-standard McCabe complexity tool (validation reference)
  • Cognitive Complexity: SonarSource specification
  • Mutation Testing: Alternative approach for test quality (future consideration)

Project links

Data verified by PyPI on Jul 8, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.16.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for szarta from gravatar.com szarta

Credits

Author: Brandon Arrendondo

GitHub Statistics

Data verified by PyPI on Jul 8, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python >=3.8

Tags

testing complexity static-analysis coverage

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

knots_test_complexity-1.14.0.tar.gz (243.1 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

knots_test_complexity-1.14.0-py3-none-win_amd64.whl (4.1 MB view details)

Uploaded Python 3Windows x86-64

knots_test_complexity-1.14.0-py3-none-musllinux_1_2_x86_64.whl (4.6 MB view details)

Uploaded Python 3musllinux: musl 1.2+ x86-64

knots_test_complexity-1.14.0-py3-none-musllinux_1_2_aarch64.whl (4.4 MB view details)

Uploaded Python 3musllinux: musl 1.2+ ARM64

knots_test_complexity-1.14.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

knots_test_complexity-1.14.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.5 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ARM64

knots_test_complexity-1.14.0-py3-none-macosx_11_0_arm64.whl (4.5 MB view details)

Uploaded Python 3macOS 11.0+ ARM64

knots_test_complexity-1.14.0-py3-none-macosx_10_12_x86_64.whl (4.4 MB view details)

Uploaded Python 3macOS 10.12+ x86-64

File details

Details for the file knots_test_complexity-1.14.0.tar.gz.

File metadata

  • Download URL: knots_test_complexity-1.14.0.tar.gz
  • Upload date:
  • Size: 243.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for knots_test_complexity-1.14.0.tar.gz
Algorithm Hash digest
SHA256 206a3c9187f61854d011577979cb6818b5f36640df288f7e3f72d9cfbdbf45ba
MD5 89e5a157567cc998104cc39d84cc10a2
BLAKE2b-256 b70bb307c39122f4f5abf0a2d1bac8817f06e233be3239efa564fe2b3d32464f

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.14.0.tar.gz:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.14.0-py3-none-win_amd64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.14.0-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 dcb7a0ad9c306d03bbb479cc8f14c96df4e4034cd946bd81ae86d5a5c1925d6b
MD5 ed2f263944b81d957c160176d177c7d7
BLAKE2b-256 b24b9802bae753d3eda74621a07741efaec14de4342e774d2b08bfd95704a08f

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.14.0-py3-none-win_amd64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.14.0-py3-none-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.14.0-py3-none-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 008386792db9164765b9c13dde421c57172c767b53a31fdbf6db5eec8c45a84a
MD5 e32f474aa2713c4499bcdb3d414cbdda
BLAKE2b-256 bbdfc42fa7526b1918871c807e8c1ae263b1d4ddbb57ef103696b56073eb24c0

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.14.0-py3-none-musllinux_1_2_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.14.0-py3-none-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.14.0-py3-none-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 8723d03c9ba8e02850f8c55bda5c16d8be6d1e313d41e804d8302f9281b2040b
MD5 7281f85ae63fb92d094a7b1f275e64a1
BLAKE2b-256 751c5e3ed34bb235bd1b08119cc6a781532c1b3fc90a6d0187e820c92e92537e

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.14.0-py3-none-musllinux_1_2_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.14.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.14.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4cb0166df4a0e40799cc43f8b770642355309bdd67d5f4d0d2cd6dd2e255a0a1
MD5 06adc1af8b8ec974ff722efd3a1343a0
BLAKE2b-256 ac533f8d557a214096a1153815af719476b266c951d4536ce772cffbba0d391c

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.14.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.14.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.14.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 ba31a4a93f3aac6542068261f3f11b81f6b24dcd6b33601c06aac723630bf02f
MD5 42b18136edabe071c0077b530d0f7af0
BLAKE2b-256 f890b0ae9f72c11903a5bf48b3c3e3c36504036869f8844eabe08fe4c45e0255

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.14.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.14.0-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.14.0-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 ff58a575e532f17c0b49b43542ab11b06e2b7360cd6e4b006db6e8496e38ce7f
MD5 c5bcf77382d3de2afefeba2c1fe488c4
BLAKE2b-256 82d27f5a6f5729c1ba0d10d6551ba94193362dfee698e1626364da79107e1db8

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.14.0-py3-none-macosx_11_0_arm64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file knots_test_complexity-1.14.0-py3-none-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for knots_test_complexity-1.14.0-py3-none-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 a938c3272a0c913060836dc978aa18d6276b17c16169cba07f2e36a692cf9c52
MD5 288cd9c868f72bd13861fb41edd05d8c
BLAKE2b-256 3f9a62117d32999bda3adcf3ae479f835d424ef497b2a7715d674450e48b6e60

See more details on using hashes here.

Provenance

The following attestation bundles were made for knots_test_complexity-1.14.0-py3-none-macosx_10_12_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/knots

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.16.0

8 files

1.15.1

8 files

1.15.0

8 files

This release

1.14.0 This release

8 files

1.13.1

8 files

1.13.0

8 files

1.12.0

8 files

1.11.0

8 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page