ovos-media-classifier 0.0.3a1

Pluggable media-type classification for OCP (Open Conversation Platform).

ovos-media-classifier

⚠️ Work in progress — pre-release software. Under active development, not yet deployed in OpenVoiceOS, and APIs may change without notice. Published in the open for transparency; do not depend on it in production yet.

A self-describing, pluggable media-intent classifier for voice assistants. Given a spoken request — "play some music", "watch an anime", "read me a chapter of Dune" — it answers, fast and offline, what kind of media is wanted so the OVOS Common Play (OCP) pipeline can route it to the right provider and player.

It is a router, not a resolver. It gates (is this a media request at all?), routes by media_type / playback_type (which MediaProviders to call, which to skip), applies content policy (adult → drop adult providers), and hands the providers a mediavocab.Signals as search context. It does not resolve a title to a stream — the providers do that — so Signals.title carries the raw query by design. The honest measure of a router is whether it routes real speech correctly: see the routing eval.

It is the single home for OCP's media-command NLP. It is multi-task — every request is classified along several orthogonal axes at once rather than into one label:

• domain — is this a media request at all (ocp_play / ocp_control / not_ocp)
• media_type — the concrete mediavocab.MediaType leaf (music, movie, podcast, …)
• playback_type — the modality (audio / video / paged / interactive)
• structure — the temporal shape (single / episodic / continuous / collection)
• explicitness — clean / adult
• tags — a multi-label, namespaced descriptive axis: genre:rock / mood:chill / era:1980s
• qualifiers — result-narrowing filters: black_and_white / silent / live / subtitled / …
• content-form genres — adult / anime / animation / asmr (drives the content filter)

The supervision comes from translatable .intent templates slot-filled with real entity metadata (IMDb, MusicBrainz, AniList, LibriVox, …), and the trained backend ships as a rules → learned-context → learned-context+NER ladder in self-describing ONNX bundles — each rung a drop-in upgrade over the last.

Quickstart

pip install ovos-media-classifier

from ovos_media_classifier import load_media_classifier

clf = load_media_classifier()                  # bundled .voc keyword classifier — zero deps
clf.classify("play some music", "en-us")       # -> (<MediaType.MUSIC: 'music'>, 0.6)

That is the whole minimum: install, load, classify. The default needs no model files and no ML dependencies — it runs fully offline.

The full multi-axis result, and a provider-ready mediavocab.Signals:

clf.classify_full("i want to watch an anime", "en-us").as_dict()
# {'media_type': 'episodic_series', 'playback_type': 'video', 'structure': 'episodic',
#  'domain': 'ocp_play', 'genres': ['anime'], 'confidence': 0.6, 'control_intent': None}

clf.classify_tags("play some 80s rock", "en-us")  # -> ['genre:rock', 'era:1980s']  (trained backend)

clf.to_signals("play some music", "en-us")        # -> mediavocab.Signals (hand to a MediaProvider)

(classify_full().as_dict() carries the single-label axes plus the content-form genres; the multi-label tags / qualifiers axes are read with their own classify_tags() / classify_qualifiers() methods. On the zero-dep keyword default tags is empty — it is the trained ONNX backend that fills it.)

classify_full is context-aware: the minimal call is classify_full(query, lang), and two optional arguments thread per-query context with no retraining — player_status (now-playing state, for control / "play something else" follow-ups) and ner_list (the user's live entities, for NER matching and the embedding router's runtime injection). See contextual classification.

Benchmark

The headline result is the lift across the ladder — the deterministic keyword rules, then a model on context (keyword) features only, then the same model once a NER store has surfaced the user's entities. Per-axis, on the held-out test split (34,700 synthetic utterances):

axis (metric)	rules	learned-context	learned-context+NER
domain (acc)	0.833	0.866	0.986
media_type (acc)	0.629	0.778	0.964
playback_type (acc)	0.702	0.895	0.988
structure (acc)	0.708	0.907	0.990
explicitness (acc)	0.988	0.989	0.997
content_form_genres (macro-F1)	0.706	0.738	0.975
qualifiers (macro-F1)	0.000	0.746	0.906
tags (macro-F1)	0.000	0.547	0.581

Content filter (driven by the content_form_genres axis), same ladder:

rung	adult recall	hentai recall	false-block	median ms	p95 ms	bundle
rules	0.481	0.510	0.000	0.32	0.50	—
learned-context	0.481	0.510	0.000	0.21	0.25	176 KiB
learned-context+NER	0.922	0.936	0.001	0.21	0.26	289 KiB

Sub-millisecond, in a 289 KiB bundle. Honesty notes: these are on the synthetic eval split — they measure the model's capability given populated features, not field accuracy on arbitrary speech (the keyword floor is ~0.29 on a neutral real-text split). The keyword backend is the zero-dependency default; the context+NER column requires a wired-in NER store to surface the ner_* features. And tags stays low by design (~0.58) — see limitations. Full table and method: docs/model.md · benchmarks/.

Axes vs. tags

The single-label heads above are axes — exactly one answer per query (a request is audio, is episodic). The open-vocabulary descriptive signals — genre, mood and era — are not axes: a query can carry several at once, and they all live in slot value text (the decade is in the year, the mood is in the activity phrase). So they are folded into one multi-label, namespaced tags head (genre: / mood: / era:) instead of three starved single-label heads. classify_content_genres() / classify_mood() / classify_era() read the matching slice. This framing is what keeps the axis count honest while still modelling the descriptive signal — classification-model.md.

Backends

load_media_classifier(config) returns one classifier. They all implement the same AbstractMediaClassifier contract, so callers never care which ran, and any load failure falls back to the keyword default.

Backend	What it is	Install
keyword (.voc)	zero-dependency phrase matching — high-precision, abstains when unsure; the offline default	core
NER	Aho-Corasick exact match over the user's entity lists (their real library)	[ner]
embedding-router (hybrid)	learned open-vocab router: keyword stays the floor, the router fills keyword's abstains using a gazetteer + the user's injected library	[onnx]
ONNX	the trained multi-task per-axis heads, loaded from a self-describing bundle	[onnx]
external	any classifier registered under opm.media.classifier	a plugin

pip install ovos-media-classifier[ner]     # entity-list matching (the user's library)
pip install ovos-media-classifier[onnx]    # trained ONNX + embedding-router backends

The keyword backend is the floor; a learned backend has to earn its place. The zero-dep keyword classifier is deliberately high-precision and abstains to GENERIC when it has no cue — a safe outcome, because an abstain still lets every provider search. The embedding-router hybrid keeps keyword as the first pass and only fills those abstains, resolving open-vocabulary titles by injecting the user's own library as entities at runtime (no retraining). That entity injection is what lowers mis-route below the keyword floor — see the routing eval.

Live routing uses a bounded entity set. Entity / gazetteer matching cost scales with the number of injected titles, so live classification runs on a bounded set — the user's library plus a capped popular gazetteer (default ~1000 titles/type, p95 a few ms). A 1M-entity set (e.g. full MusicBrainz) is for OFFLINE tagging only, never live classification. The optional online metadatarr.resolve layer (~seconds per title) is for offline tagging or a long-running agent — off by default, never in the live OCP pipeline.

See docs/backends.md. To write your own backend or train your own bundle (including adding a brand-new axis end-to-end), see docs/extending.md.

Content filtering

A detect-to-block moderation layer recognises sensitive requests so OVOS can refuse them. It reads the content_form_genres axis, so adult can be flagged independently of the media-type leaf (a single leaf mistake never unblocks it). adult is blocked by default (lift it with allow_adult_content); the adult / hentai data exists for detection only, never for provision.

from ovos_media_classifier import ContentFilter
ContentFilter().check(clf, "play some porn", "en-us")   # (True, 'blocked genre: adult')

See content filtering.

Command vs content classification

This package classifies a voice command (what does the user want?). That is a different problem from mediavocab.text.classify, which classifies a piece of catalog content (what kind of item is this?). They share the mediavocab.MediaType vocabulary but answer opposite questions — do not substitute one for the other. See taxonomy.md.

Documentation

Start at docs/index.md for the audience-routing table, or read the glossary first if the terms are new.

• New here → glossary · index · examples/
• API reference → stable API
• The model → classification model · the trained model · taxonomy · hierarchical experiment
• The data → dataset · data sources · dataset plots
• Tuning backends → backends · embedding-router · entity lists · contextual classification · open-vocab routing
• Moderation → content filtering
• Writing / training a classifier → extending · external plugins
• Measuring → routing eval (the source of truth) · benchmarks

Credits

Media-metadata datasets by TigreGotico on Hugging Face.

License

Apache-2.0.