ovos-routing-judge

Who took an utterance, and was it the skill that should have?

One judge for every tool that replays golden utterances (test/end2end/golden_utterances_<lang>.jsonl) against OpenVoiceOS and asks whether each sentence reached the right skill. It is pure Python over the bus messages of one session: no messagebus connection, no UI, no store knowledge, no dependencies. So the same row gets the same verdict wherever it runs:

• ovos-klondike-mercantile's CI, on an in-process MiniCroft;
• ovos-tui-client's test runs (ovos-tui --run), on a live device over the messagebus.

A failure in CI can then be reproduced on a device and judged the same way (andlo/ovos-klondike-mercantile#48).

Use

from ovos_routing_judge import Claim, in_session, judge

claim = Claim.from_messages(recorded, known_ids=loaded_skill_ids,   # Message objects or dicts
                            session_id=row_session_id)
v = judge(claim, own_ids={"ovos-skill-date-time.openvoiceos"},
          expected="ovos-skill-date-time.openvoiceos:what_time_is_it",
          hung=timed_out, known_ids=loaded_skill_ids)
v.kind     # hit | wrong_intent | captured | other | unhandled | hang
v.taker    # who took it
v.via      # its best signal: intent | provider | ocp | skill | speak
v.detail   # one line on what happened

Claim.observe(message, known_ids) folds messages in one at a time, for a live run that wants to know as soon as something matched.

known_ids is a filter by default: only loaded skills can take a row, so a pipeline plugin's own intent with nothing behind it is unhandled. A caller whose skill list may be incomplete (a live device with skills in other containers) passes strict_known=False: known skills still win, anyone else is the fallback.

Verdicts

What counts as taking a sentence

Signals come in tiers; the taker is the first known skill in the best tier:

1. intent: <skill_id>:<intent> dispatched, ovos.intent.matched, or a pending <skill_id>.converse.* capturing it
2. provider: a pipeline plugin asked it for content (ovos.common_reading.fetch_content.<skill_id>)
3. ocp: OCP picked its result (ovos.common_play.play media, or the first track of play / search.populate without a player)
4. skill: handler start, a fallback that answered, common query's question:action
5. speak: it spoke in this session (speak or core 3's ovos.utterance.speak)

ovos.intent.unmatched means nobody, whatever else was seen.

With session_id, other sessions' messages are ignored, and a message without a session can show an intent, a provider or an OCP pick but never a tier 4 or 5 claim: unrelated skills' background activity (a scheduled event's handler, a fallback probe's reply) arrives without a session and is not theft (learned in ovos-test-harness's fleet suite).

Intent names are compared in one spelling: Name.intent (ovos-workshop 1.x) vs Name, CamelCase in older releases vs snake_case, padatious what.time.is.it vs padacioso what_time_is_it. A row with "intent_type": "ocp" must go through OCP.

The contract

tests/fixtures/cases.json holds recorded message sequences and the verdict each must get. When a core changes its signals, a case goes in there and every tool using the package gets the fix.

License

Apache-2.0