robotframework-falsegreen

One problem, one tool: the false positive. robotframework-falsegreen finds Robot Framework tests that pass green without protecting anything - tests that let broken behavior through because no keyword verifies anything, the failure is swallowed, the check is always true, or the test is skipped.

Deterministic static scan over the official Robot Framework parser (robot.api.get_model) - no execution. Sibling of falsegreen (Python/pytest) and falsegreen-js (JS/TS). The semantic, intent-based pass lives in falsegreen-skill.

The falsegreen family (install the one for your stack):

Why

A green Robot suite is not proof of correctness. A test case can run keywords and never call a verification keyword; a Run Keyword And Ignore Error can absorb the failure; a Should Be True ${TRUE} can never fail. This tool flags the patterns a parser can prove, before they reach review.

Install

pip install robotframework-falsegreen

Usage

rffalsegreen                  # scan cwd
rffalsegreen tests/           # scan a path
rffalsegreen --format json    # machine-readable output (--json is an alias)
rffalsegreen --format sarif   # SARIF 2.1.0 for GitHub code scanning
rffalsegreen --format junit   # JUnit XML for a CI test report
rffalsegreen --output report.sarif  # write to a file
rffalsegreen --output .falsegreen/  # write report.<ext> into a directory
rffalsegreen --config-audit   # audit the Robot run config (project-layer PL codes)
rffalsegreen --disable C16    # turn off specific codes

--format selects the output shape: text (default), json, sarif, or junit. SARIF 2.1.0 (tool name robotframework-falsegreen) maps confidence to severity - HIGH to error, LOW to warning, off/info to note - and tags each result with its judgment family and pyramid level, so GitHub code scanning can group and filter findings. JUnit XML emits one testcase per finding: HIGH becomes a <failure>, anything lower becomes a <skipped>. --json stays as an alias for --format json and keeps its existing envelope (tool / version / judgments / findings).

Each finding is reported with its pyramid level (unit / integration / e2e, read from the suite's imported libraries) and a one-line fix hint, and the text summary breaks the findings down by level and lists the most common fixes. --output takes a file or a directory: an extension-less or trailing-slash path (e.g. .falsegreen/) receives report.<ext> for the chosen format (report.sarif, report.xml for JUnit). Reports are run artifacts; keep the output directory gitignored.

Inline suppression

To silence a single justified finding without disabling the code suite-wide, add a comment on the offending line. The token and bracket syntax match falsegreen (Python) and falsegreen-js:

*** Test Cases ***
Polls A Real Service
    Sleep    1s    # falsegreen: ignore[C16]      # silence only C16 on this line
    Should Be Equal    ${result}    ${expected}

# falsegreen: ignore (no brackets) silences every code on that line; ignore[C16,C20] silences only the listed codes. The suppression is scoped to its line, so a sibling test is unaffected, and only the exact falsegreen: token suppresses (a plain # ignore does not).

Baseline (adopt on a suite that already has findings)

To add the scanner to a suite with pre-existing findings without a wall of red, record a baseline and then fail only on new findings:

rffalsegreen --write-baseline    # record current findings to .falsegreen-baseline.json
rffalsegreen --baseline          # scan, suppressing everything in the baseline

Both flags take an optional path (default .falsegreen-baseline.json). The baseline fingerprints a finding by content - sha1(relative path, code, test/keyword name, detail) with no line number - so it survives edits that shift a test up or down the file. Commit the baseline so CI sees the same set. A new false-green that is not in the baseline still fails the run; shrink the baseline as you fix the recorded ones.

pre-commit

Run the scanner as a pre-commit hook so a false-green test is caught before it lands. Add to your .pre-commit-config.yaml:

repos:
  - repo: https://github.com/vinicq/robotframework-falsegreen
    rev: v0.3.0
    hooks:
      - id: rffalsegreen

The hook scopes to .robot/.resource files and passes the staged paths to the scanner, so it never re-scans results//output/. It honors the exit codes (0 clean, 10 low only, 20 high), so a high-confidence finding fails the commit. To run it on demand against only the staged files without committing, use pre-commit run rffalsegreen.

--config-audit is a separate mode: instead of scanning suites, it reads the Robot run config (robot.toml, pyproject.toml [tool.robot], and *.args argument files found recursively, skipping ignored directories like results//output/) and reports PL9 - a --skiponfailure / --noncritical option that turns a failing test into a non-fatal pass (legacy, removed in RF 4+). The per-file scan cannot see run config.

For the layer no static scan reaches (does a green test fail when the code is wrong?), Robot has no standard mutation tester, so that check is manual review; the semantic falsegreen-skill covers the intent-level cases.

Exit code: 0 clean, 10 low-confidence only, 20 high-confidence present. Wire exit 20 into CI to block the merge.

What it detects

The oracle in Robot is the verification keyword. The scanner recognizes them across libraries (the Should convention plus library-specific forms: SeleniumLibrary Element Should Be Visible, Browser's assertion engine Get Text sel == expected, RESTinstance schema keywords, DatabaseLibrary Row Count Should Be Equal, custom Verify*/Assert* keywords). A test with none of them verifies nothing.

Scans *** Test Cases ***, *** Tasks *** (RPA), and *** Keywords *** definitions in both .robot and .resource files. R2 catches the root cause of a missed C2b: a test calls Verify Login and looks protected, but that keyword never asserts anything.

Opt-in: maintainability group (default off)

Not false-green - the test still verifies - so off by default. Enable with --diagnostics. Three groups, mirroring falsegreen and falsegreen-js: false-positive (C*/R*, on), diagnostic (D*, opt-in), coupling (M*, opt-in).

rffalsegreen --diagnostics    # include D*/M* as warnings

Codes share ids with the sibling scanners where the concept matches (C2/C2b/C3/C5/C7/C9/C16/C20/C21/C32/C37/CC). R* are Robot-specific. A Browser Get keyword with no assertion operator is a plain getter, so a test whose only step is Get Text h1 surfaces as no-verification (C2b). The consolidated catalog's Robot ids map onto these: RF3 is C3 (here it also catches the form where the status is captured in a variable but never asserted), RF17 is R6, RF18 is R5, RF20 is C7.

Test levels (the pyramid)

rffalsegreen scans Robot suites at every level of the pyramid. Discovery is level-agnostic - it reads any .robot/.resource - but a few codes are read in light of the level, so a valid pattern at one level is not flagged at another.

• Unit: keyword logic with the boundaries doubled. The oracle is a Should keyword.
• Integration (API and database): API tests through RequestsLibrary and RESTinstance (the schema keywords count as the oracle), database tests through DatabaseLibrary (Row Count Should Be Equal, Check If Exists In Database). These hit a real endpoint or datastore on purpose, so the request or row IS the verification at that level.
• E2E: the Browser library and SeleniumLibrary/Appium. The page assertion (Page Should Contain, Get Text ... == ...) is the oracle; the presence of a rendered element is a real check at this level, not a weak one.

A real API or database hit inside a test that claims to be a unit test is itself the smell (environment coupling, mystery guest), not the level of the test. C23 flags the strongest form: a hard-coded IP-address endpoint.

Scope and honesty

Static scan: it owns what the keyword structure proves. It does not run the suite, so it cannot see runtime-only smells (Test Run War, order dependence across suites). Whether the expected value contradicts the intended behavior is semantic and belongs to falsegreen-skill. Precision over recall: C2b is low-confidence because a custom keyword may assert internally without Should in its name.

Measured against the Open Catalog of Test Smells (517 documented smells), only the false-green slice is in scope. What stays out, on purpose: brittleness / false-red (sensitive equality, fragile fixtures - the opposite axis), hygiene / maintainability (long tests, magic values - linter territory, Robocop), and slow, design, naming, duplication, runtime/culture. The boundary is deliberate: where a smell has a statically provable false-green form, that form is a code here - Sleep as synchronization is C16, a hard-coded IP URL is C23, conditional-only verification is C21, and a test with no verification keyword is C2b. See CREDITS.md for the full cross-walk.

Not implemented, on purpose

Some catalog codes are left out because the gain is not worth the false positives, or because another tool already owns them. Listing them is part of the scope:

• RF16 (Wait Until Keyword Succeeds) - a retry wrapper. Legitimate retry around genuinely asynchronous behavior is common and idiomatic, so flagging every use would be mostly noise. The false-positive rate is too high for a static rule; this is a judgment call left to review.
• Hygiene already covered by Robocop — RF6 (dead keyword, cross-file), RF8 (unused argument), RF13 (duplicate name), RF14 (missing documentation), RF15 (too long), RF19 (unused import). These are maintainability lint, not false-green, and Robocop detects them well. Run Robocop alongside this scanner; there is no reason to reimplement them.

C-codes with no idiomatic Robot form

Three Python/JS sibling codes have no clean Robot equivalent and are intentionally skipped:

• C8 (exact float equality) - Robot test data is untyped text, so a value cannot be proven to be a float from the parse tree (Should Be Equal As Numbers even takes a precision argument). Any rule here would guess, with a high false-positive rate.
• C18 (compare a stringified value to a literal) - Robot has no str()/repr() round-trip concept; everything is already a string. There is no structural signal to key on.
• C48 (dark patch: flip a test-mode flag, then assert) - Robot has a shape for it (Set Environment Variable TESTING true / Set Global Variable then a verification), but Robot test data is untyped text: the parse tree cannot prove a variable name is a test-mode flag, and a truthy value is just a string. Detecting it would need variable-lifecycle tracking with no clean signal, so the false-positive ceiling is too high. Skipped on purpose.

C9 (broad error assertion) and C20 (dead step after a terminator) do have idiomatic Robot forms (Run Keyword And Expect Error * and a verification after [Return]/Fail), so both are implemented above. Robot has no standard mutation tester, so the semantic layer (does a green test fail when the code is wrong?) stays manual review; the intent-level cases live in falsegreen-skill.

License

MIT, Vinicius Queiroz.

Contributors ✨

Thanks to the people who keep false-green tests out of real suites (emoji key):

Vinicius Queiroz 💻 📖 🤔 🚧 🚇 ⚠️ 🔬

Home Seller 💻 📖 ⚠️

New contributors are added automatically; the table also recognizes non-code work (docs, ideas, infrastructure, tests, research) via the all-contributors spec.