robotframework-events

A Robot Framework listener that writes every suite, test, keyword, control structure and log message of a run as JSON Lines, while the run happens. Item ids are Robot Framework's own (s1-t2-k3), so events join with output.xml and link into log.html#<id>.

Requires Robot Framework 7.0 or newer and Python 3.10 or newer. Status: alpha.

pip install robotframework-events
robot --listener robotframework_events.Listener tests/                       # writes events.jsonl next to output.xml
robot --listener "robotframework_events.Listener;file=ev.jsonl;level=DEBUG" tests/

Listener arguments: file (NONE for no file), level (minimum message level, default INFO), messages, args, structures (each true/false, default true), max_text (default 4096) and flush (seconds between flushes of the file while the run goes on, default 0.1, so rfevents tail -f and editors that follow the file see the keyword that is running; 0 flushes after every test only).

Step screenshots

With screenshots, the listener takes a screenshot after the keywords of the libraries you name, for a Playwright-like playback of each test (robotframework-dashboard's Live page). Libraries are opt-in, keywords opt-out: in a named library every keyword gets one, except the read-only ones its preset excludes (Get ..., Wait ..., ... Should ... and similar); failing keywords always get one.

pip install "robotframework-events[screen]"     # Pillow: desktop libraries, and JPEG instead of PNG
robot --listener "robotframework_events.Listener;screenshots=actions;screenshot_libraries=Browser,SeleniumLibrary" tests/

screenshots is off (default), actions, all (every keyword, exclusions ignored) or failure, or the path of a JSON file with more settings:

{
  "version": 1,
  "mode": "actions",
  "libraries": ["Browser", "FlaUI"],
  "include": ["MyShop.Log In"],
  "exclude": ["Browser.Hover", "Scroll To *"],
  "custom": {
    "MyDesktopLibrary": {"match": ["mydesktop:MyDesktopLibrary"], "capture": {"type": "screen", "region": "window"}}
  }
}

include adds own keywords (from resource files); they show the page of the library used last. custom adds a library without a preset: match names its class (topmodule:Class, base classes count) or the name it is imported with (name:JavaFXLibrary), capture.type is browser, webdriver (with driver: driver or _current_application()), borrow (with from: the libraries whose page it works on), method (with method: the library's screenshot method, called with a file path) or screen (with region: screen or window, the window in front). The presets are in presets.json.

The images go to ${OUTPUT DIR}/rfevents-screenshots/, each announced with a step.screenshot event after its step.ended. They are taken through the library directly, so nothing is added to output.xml or log.html. Each one takes time, so they are off by default.

Libraries change fast, so a screenshot never gets in the way of a test: none is taken while a JavaScript dialog is open (WebDriver would dismiss it), Browser's Take Screenshot arguments are read from its signature, and a library whose API does not fit or whose screenshots keep timing out is left alone for the rest of the run. That is reported on the console and as a screenshot.unavailable event, which the Live page shows: please report it with the library's version.

Supported libraries

Every library from awesome-robotframework that can take a screenshot. "Tested" means a test run with and without screenshots gave the same results, with real screenshots, on Windows 10 (Android 14 emulator for the mobile libraries). Browser, SeleniumLibrary, QWeb, AppiumLibrary, RPA Framework (RPA.Browser.*, RPA.Windows), WhiteLibrary, Puppeteer Library and FlaUI are checked again every week, with the versions in the table and with their latest releases. Problems with a library: open a step screenshots issue.

Libraries built on one of these are recognised by their base class. Not supported: Mainframe 3270 (its screenshot is the terminal's text). "The screen" and "the window in front" need a display, and show what is visible there, other windows included.

Event format

One JSON object per line. Every event has v (format version), seq (1, 2, 3, ...), t (Unix time) and type: run.started, suite.started, test.started, step.started, message.logged, step.ended, step.screenshot, test.ended, suite.ended, status.changed, resultfile.created, run.ended.

{"v":1,"seq":6,"t":1791581563.542,"type":"test.started","id":"s1-t1","parent":"s1","name":"Passing","longname":"Sample.Passing","lineno":6}
{"v":1,"seq":61,"t":1791581563.601,"type":"test.ended","id":"s1-t2","status":"FAIL","message":"1 != 2","elapsed":0.004}

The JSON Schema events-v1.schema.json is the reference.

Command line

rfevents tail -f results/events.jsonl          # follow a run in the terminal, keyword by keyword
rfevents tail --steps 2 --level INFO events.jsonl
rfevents convert output.xml -o events.jsonl    # events for a run recorded without the listener
rfevents rebuild events.jsonl -o output.xml    # output.xml for a run that crashed before writing it

tail prints suites, tests, steps down to --steps levels (default 1), log messages from --level (default WARN) and a summary. convert reads output.xml or output.json; steps then have no source or line number, which result files do not store. rebuild is the other way round, for a run that was killed or crashed: the tests and steps that were still running fail with "Not finished", tests that never started are left out, and the top-level suite says the run did not finish.

With pabot, every process writes its own events file in its own output folder (pabot --listener robotframework_events.Listener tests/); run.started then has a pabot field with the process's index, pool, processes and whether it is the last one pabot starts.

Python API

from robotframework_events import RunState, add_callback, convert, read_events, rebuild

state = RunState.from_events(read_events("events.jsonl"))  # also .jsonl.gz
print(state.status, state.counts())
for event in read_events("events.jsonl", follow=True):  # while the run writes it
    ...

events = list(convert("output.xml"))  # dicts, as the listener writes them
rebuild(read_events("events.jsonl")).save("output.xml")  # a Robot Framework result
add_callback(fn)  # in-process consumers, e.g. robotcode's debugger

read_events gives typed events (robotframework_events.events). RunState holds the tree of suites, tests and steps with status, messages and times, and the running test and step.

Development

pip install -e .[dev]
ruff check . && ruff format --check . && mypy
pytest                 # format, options, schema, reader, CLI and conformance tests
pytest -m bench        # listener overhead
pytest -m e2e          # through robotcode's debugger (needs ../robotcode)

The conformance tests also run on Robot Framework's acceptance test data, found through RF_ATEST_DIR or at ../robotframework/atest/testdata, and are skipped when it is missing.

Licence

MIT