robotframework-falsegreen 0.2.0

Find Robot Framework tests that pass green without protecting anything: tests with no verification keyword, swallowed failures, always-true checks. Deterministic static scan, sibling of falsegreen (Python) and falsegreen-js (JS/TS).

robotframework-falsegreen

One problem, one tool: the false positive. robotframework-falsegreen finds Robot Framework tests that pass green without protecting anything - tests that let broken behavior through because no keyword verifies anything, the failure is swallowed, the check is always true, or the test is skipped.

Deterministic static scan over the official Robot Framework parser (robot.api.get_model) - no execution. Sibling of falsegreen (Python/pytest) and falsegreen-js (JS/TS). The semantic, intent-based pass lives in falsegreen-skill.

Why

A green Robot suite is not proof of correctness. A test case can run keywords and never call a verification keyword; a Run Keyword And Ignore Error can absorb the failure; a Should Be True ${TRUE} can never fail. This tool flags the patterns a parser can prove, before they reach review.

Install

pip install robotframework-falsegreen

Usage

rffalsegreen                  # scan cwd
rffalsegreen tests/           # scan a path
rffalsegreen --json           # machine-readable output
rffalsegreen --output report.json   # write to a file
rffalsegreen --output .falsegreen/  # write report.<ext> into a directory
rffalsegreen --config-audit   # audit the Robot run config (project-layer PL codes)
rffalsegreen --disable C16    # turn off specific codes

Each finding is reported with its pyramid level (unit / integration / e2e, read from the suite's imported libraries) and a one-line fix hint, and the summary breaks the findings down by level and lists the most common fixes. --output takes a file or a directory: an extension-less or trailing-slash path (e.g. .falsegreen/) receives report.<ext> for the chosen format. Reports are run artifacts; keep the output directory gitignored.

--config-audit is a separate mode: instead of scanning suites, it reads the Robot run config (robot.toml, pyproject.toml [tool.robot], *.args argument files) and reports PL9 - a --skiponfailure / --noncritical option that turns a failing test into a non-fatal pass (legacy, removed in RF 4+). The per-file scan cannot see run config.

For the layer no static scan reaches (does a green test fail when the code is wrong?), Robot has no standard mutation tester, so that check is manual review; the semantic falsegreen-skill covers the intent-level cases.

Exit code: 0 clean, 10 low-confidence only, 20 high-confidence present. Wire exit 20 into CI to block the merge.

What it detects

The oracle in Robot is the verification keyword. The scanner recognizes them across libraries (the Should convention plus library-specific forms: SeleniumLibrary Element Should Be Visible, Browser's assertion engine Get Text sel == expected, RESTinstance schema keywords, DatabaseLibrary Row Count Should Be Equal, custom Verify*/Assert* keywords). A test with none of them verifies nothing.

Code	Confidence	What it flags
C2	high	empty test case, task, or keyword (no keywords run)
C2b	low	runs keywords but no verification keyword (no oracle)
C3	high	Run Keyword And Ignore Error/Return Status swallows the failure, status never asserted
C5	high	always-true (Should Be True ${TRUE}, Should Be Equal with equal literals)
C6	low	weak check — Should Be True on a bare variable (truthiness only, not a comparison)
C7	high	self-compare (Should Be Equal ${x} ${x})
C16	low	Sleep used as synchronization (timing dependence)
C21	low	verification only runs conditionally (inside IF / Run Keyword If) — it may never execute
C23	low	hard-coded IP-address URL in test data (http://10.0.0.5:8080) — environment coupling
C32	low	skipped test (robot:skip / Skip)
R1	high	Pass Execution forces the test green regardless of any check
R2	low	user keyword named like a verifier (Verify/Assert/Should...) but its body verifies nothing — a hollow oracle
R3	high	*** Test Cases *** inside a .resource file — invalid; the cases never run
R4	high	No Operation is the only step — the test/task/keyword does nothing
R5	high	[Template] with no data rows — the templated test runs zero cases

Scans *** Test Cases ***, *** Tasks *** (RPA), and *** Keywords *** definitions in both .robot and .resource files. R2 catches the root cause of a missed C2b: a test calls Verify Login and looks protected, but that keyword never asserts anything.

Opt-in: maintainability group (default off)

Not false-green - the test still verifies - so off by default. Enable with --diagnostics. Three groups, mirroring falsegreen and falsegreen-js: false-positive (C*/R*, on), diagnostic (D*, opt-in), coupling (M*, opt-in).

Code	Group	What it flags
D2	diagnostic	control flow (IF/FOR/WHILE/TRY) at the test/task level (the guide advises against it)
M2	coupling	test/task with too many steps (guide suggests max ~10)

rffalsegreen --diagnostics    # include D*/M* as warnings

Codes share ids with the sibling scanners where the concept matches (C2/C2b/C3/C5/C7/C16/C21/C32). A Browser Get keyword with no assertion operator is a plain getter, so a test whose only step is Get Text h1 surfaces as no-verification (C2b).

Test levels (the pyramid)

rffalsegreen scans Robot suites at every level of the pyramid. Discovery is level-agnostic - it reads any .robot/.resource - but a few codes are read in light of the level, so a valid pattern at one level is not flagged at another.

• Unit: keyword logic with the boundaries doubled. The oracle is a Should keyword.
• Integration (API and database): API tests through RequestsLibrary and RESTinstance (the schema keywords count as the oracle), database tests through DatabaseLibrary (Row Count Should Be Equal, Check If Exists In Database). These hit a real endpoint or datastore on purpose, so the request or row IS the verification at that level.
• E2E: the Browser library and SeleniumLibrary/Appium. The page assertion (Page Should Contain, Get Text ... == ...) is the oracle; the presence of a rendered element is a real check at this level, not a weak one.

A real API or database hit inside a test that claims to be a unit test is itself the smell (environment coupling, mystery guest), not the level of the test. C23 flags the strongest form: a hard-coded IP-address endpoint.

Scope and honesty

Static scan: it owns what the keyword structure proves. It does not run the suite, so it cannot see runtime-only smells (Test Run War, order dependence across suites). Whether the expected value contradicts the intended behavior is semantic and belongs to falsegreen-skill. Precision over recall: C2b is low-confidence because a custom keyword may assert internally without Should in its name.

License

MIT, Vinicius Queiroz.

Contributors ✨

Thanks to the people who keep false-green tests out of real suites (emoji key):

Vinicius Queiroz 💻 📖 🤔 🚧 🚇 ⚠️ 🔬

Home Seller 💻

New contributors are added automatically; the table also recognizes non-code work (docs, ideas, infrastructure, tests, research) via the all-contributors spec.