MustRD

"MustRD: Validate your SPARQL queries and transformations with precision and confidence, using BDD and Given-When-Then principles."

Why?

SPARQL is a powerful query language for RDF data, but how can you ensure your queries and transformations are doing what you intend? Whether you're working on a pipeline or a standalone query, certainty is key.

While RDF and SPARQL offer great flexibility, we noticed a gap in tooling to validate their behavior. We missed the robust testing frameworks available in imperative programming languages that help ensure your code works as expected.

With MustRD, you can:

• Define data scenarios and verify that queries produce the expected results.
• Test edge cases to ensure your queries remain reliable.
• Isolate small SPARQL enrichment or transformation steps and confirm you're only inserting what you intend.

What?

MustRD is a Spec-By-Example ontology with a reference Python implementation, inspired by tools like Cucumber. It uses the Given-When-Then approach to define and validate SPARQL queries and transformations.

MustRD is designed to be triplestore/SPARQL engine agnostic, leveraging open standards to ensure compatibility across different platforms.

Write a spec once, run it against any supported engine:

Engine configuration lives in a triplestore .ttl — see GETSTARTED.adoc.

What it is NOT

MustRD is not an alternative to SHACL. While SHACL validates data structures, MustRD focuses on validating data transformations and query results.

How?

You define your specs in Turtle (.ttl) or TriG (.trig) files using the Given-When-Then approach:

• Given: Define the starting dataset.
• When: Specify the action (e.g., a SPARQL query).
• Then: Outline the expected results.

Depending on the type of SPARQL query (CONSTRUCT, SELECT, INSERT/DELETE), MustRD runs the query and compares the results against the expectations defined in the spec.

Expectations can also be defined as:

• INSERT queries.
• SELECT queries.
• Higher-order expectation languages, similar to those used in various platforms.

Example

Configuration File

You'll have a configuration .ttl file, which acts as a suite of tests. It tells MustRD where to look for test specifications and any triplestore configurations you might have:

:test_example a :MustrdTest;
              :hasSpecPath "test/specs/";
              :hasDataPath "test/data/";
              :hasPytestPath "example";
              :triplestoreSpecPath "test/triplestore_config/triplestores.ttl";
              :filterOnTripleStore triplestore:example_test .

Test Specification

In the directory specified by :hasSpecPath, you'll have one or more .mustrd.ttl files. These can be organized in a directory structure. MustRD collects them and reports results to your test runner.

:test_example :given [ a :FileDataset ;
                       :file "test/data/given.ttl" ] ;
              :when [ a :TextSparqlSource ;
                     :queryText "SELECT ?s ?p ?o WHERE { ?s ?p ?o }" ;
                     :queryType :SelectSparql ] ;
              :then [ a :OrderedTableDataset ;
                     :hasRow [ :variable "s" ; :boundValue "example:subject" ;
                               :variable "p" ; :boundValue "example:predicate" ;
                               :variable "o" ; :boundValue "example:object" ] ].

And you will have a 'test/data/given.ttl' which contains the given ttl.

example:subject example:predicate example:object .

Running Tests

Run the test using the MustRD Pytest plugin:

poetry run pytest --mustrd --config=test/mustrd_configuration.ttl --md=render/github_job_summary.md

This will validate your SPARQL queries against the defined dataset and expected results, ensuring your transformations behave as intended.

You can refer to SPARQL inline, in files, or in Anzo Graphmarts, Steps, or Layers. See GETSTARTED.adoc for more details.

Named graphs

A quad given (TriG, NQuads, TriX) keeps its named graphs, so a when can read one with a GRAPH clause. A when with no GRAPH clause still sees everything.

must:given [ a must:FileDataset ; must:file "data/two-graphs.trig" ] ;
must:when  [ a must:TextSparqlSource ;
             must:queryText "SELECT ?v WHERE { GRAPH ex:graph-a { ?s ex:value ?v } }" ;
             must:queryType must:SelectSparql ] ;

A then is compared as one flat union by default. You should not have to say which graph a triple is in just to assert that it exists, so a quad then passes as long as the triples are all there, wherever they sit.

When the graph a triple lands in is the thing under test — a pipeline writing each layer to its own graph — opt in:

must:then [ a must:FileDataset ;
            must:matchNamedGraphs true ;
            must:file "data/expected.trig" ] .

Now the layout is part of the assertion: the right triples in the wrong graphs is a failure, and the failure names the graphs that differ rather than handing you a merged diff to work out which layer moved.

Which query forms can produce named graphs is engine-specific, and it catches people out. An UPDATE always can — INSERT { GRAPH ?g { … } } is standard. A CONSTRUCT cannot, in standard SPARQL 1.1: the template is triple patterns only, so a GRAPH there is a syntax error, not an empty result. On RDFLib you get Expected ConstructQuery, found 'GRAPH'.

Several engines extend it anyway — Jena/ARQ since 3.0.1, Stardog via a graph template, and Anzo, whose CONSTRUCT clause takes "a graph and triple template". The Jena and Stardog forms are recorded in the W3C SPARQL CG inventory of extensions; standardising it is w3c/sparql-dev#31.

That extension is what makes a dry run possible: swap INSERT for CONSTRUCT, keep the template and WHERE as they are, and you get back the quads the update would have written without writing them — then must:matchNamedGraphs true asserts they land in the right graphs. mustrd's Stardog backend asks for TriG rather than Turtle so those graph names survive the response.

Anzo already ships that rewrite as a button: for an INSERT or DELETE, the Query Builder's Dry Run "runs a version of the query where INSERT or DELETE is replaced with CONSTRUCT", reporting "additions or removals per graph". What that gives you once, interactively, a spec gives you on every run with the expected graphs written down.

See GETSTARTED.adoc for the per-engine table.

When a spec fails

A failing SELECT names the binding that differs and what it differs by, on the first line — before the table. The commonest failure of all, right shape and wrong value, otherwise reports two identical shapes and leaves you to find the one cell that matters:

Expected 1 row(s) and 1 column(s), got 1 row(s) and 1 column(s) — differs in: month (datatype: expected xsd:string, actual xsd:gYearMonth)

|    | ('month_datatype', 'expected')          | ('month_datatype', 'actual')                |
|---:|:----------------------------------------|:--------------------------------------------|
|  0 | http://www.w3.org/2001/XMLSchema#string | http://www.w3.org/2001/XMLSchema#gYearMonth |

A wrong term rather than a wrong type reads the same way, using the prefixes your given declares:

Expected 2 row(s) and 3 column(s), got 2 row(s) and 3 column(s) — differs in: o (expected ex:object, actual ex:obj)

The ones that are hardest to see and easiest to make are two IRIs naming the same thing under a different scheme or host. Those are named rather than printed:

— differs in: s (scheme: expected http, actual https)
— differs in: s (host: expected company.dev, actual company.com)
— differs in: s (origin: expected http://company.dev, actual https://company.com)

The same ontology served from a dev host and a prod one, or over http on one side and https on the other, agrees everywhere the eye lands. Printing both IRIs makes it a spot-the-difference puzzle; eliding them makes it worse.

Only claimed when the origin really is the whole difference. If anything after the host differs too you get both IRIs, because naming the host would send you after the wrong thing.

Some details worth knowing, because they are what make it readable on real data:

• The pair is only named when the two shapes match. A row- or column-count mismatch already says what is wrong, and listing every column would be noise.

• A binding whose value differs is named once. Its datatype nearly always differs too; (datatype: …) is reserved for the case you cannot see otherwise — same text, different type.

• A long value is elided around the difference, not from the start — cutting the tail off two near-identical IRIs would show you the half they agree on and hide the half they don't:

  — differs in: s (expected <…o/the/thing/alpha>, actual <…o/the/thing/beta>)
  

• A wide result falls back to bare column names. Past a readable length the line has stopped being a summary, and the table below is the right tool.

What a run can produce

Every flag below works identically on pytest --mustrd and on mustrd report:

# the usual pair: something to read in the terminal, something to attach to CI
pytest --mustrd --config=config.ttl --cq --term-coverage --md=report.md --viewer=report.html

--viewer already implies the coverage and competency-question graphs; --cq and --term-coverage only affect the terminal and --md output.

Running on Windows

Windows is a supported, CI-tested platform — the matrix covers windows-latest on Python 3.11, 3.12 and 3.13. Both front ends work the same as on Linux:

pytest --mustrd --config=test\test_config_local.ttl --md=report.md
mustrd report --config test\test_config_local.ttl --viewer report.html

Two things differ in practice:

• Non-ASCII output is handled for you. The reports use ↳, ▸, ✅ and ❌, and a Windows console still defaults to cp1252, where printing those raises UnicodeEncodeError. The CLI reconfigures its own streams to UTF-8 on startup, so this is not something you need to work around. If you drive mustrd from your own script and see an encoding error, set PYTHONIOENCODING=utf-8 fixes it.
• Paths in a config are resolved relative to the config file, not the working directory, so backslashes and drive letters are fine and you can run from anywhere.

Running on locked-down Windows

Enterprise builds commonly block the bare .exe shims that pip installs into Scripts\. The package installs fine, but the mustrd command won't start. Run the module through the venv's interpreter instead:

python -m venv .venv
.venv/Scripts/python -m pip install mustrd

.venv/Scripts/python -m mustrd report --config config.ttl --viewer report.html
.venv/Scripts/python -m pytest --mustrd --config=config.ttl --md=report.md

python -m mustrd is exactly equivalent to the mustrd command — same entry point, same flags. It exists for this.

Integrating with Visual Studio Code (vscode)

We have a pytest plugin.

1. Choose a python interpreter (probably a venv)
2. pip install mustrd in it.
3. add to your settings.json

    "python.testing.pytestArgs": [
        "--mustrd", "--md=junit/github_job_summary.md", "--config=test/test_config_local.ttl"
    ],

4. VS Code should auto discover your tests and they'll show up in the flask icon 'tab'.

Each .mustrd.ttl is a node in the tree, in the folder it actually lives in, with one test under it per spec and triple store (<spec>@<store>). Running a single one runs exactly that one. :hasPytestPath no longer shapes the tree — the directories do — but it still filters, via --pytest-path.

Also worth installing: Mentor

If you are writing RDF in VS Code, get Mentor (faubulous.mentor, mentor-vscode.dev, GPL-3.0). It is the missing IDE for knowledge graphs, and it makes authoring mustrd specs markedly less painful: syntax highlighting and validation for Turtle, TriG, N-Triples, N-Quads, RDF/XML and SPARQL, browsable RDFS/OWL/SHACL/SKOS definition trees with structural reasoning, workspace-wide autocomplete with prefix.cc lookup, go-to-definition and cross-file references, prefix and IRI renaming, and a built-in triple store you can run SPARQL against.

A mustrd spec is just Turtle, and Mentor treats it as such — jump straight from a must:fileurl to the query it points at, or from a term in a spec to its definition in your ontology. Nothing to configure on our side. Credit where it's due: it is an excellent piece of work and not ours.

Competency questions & ontology coverage

A competency question (CQ) is a first-class cq:CompetencyQuestion node (its vocabulary is cq: = https://mustrd.org/competencyQuestion/) — it owns the question (cq:question, a sub-property of rdfs:label) and optionally links to the test(s) that answer it with cq:cqSpec. CQ nodes live in any .mustrd.ttl in the suite:

@prefix cq: <https://mustrd.org/competencyQuestion/> .

:rotterdamCountryCQ a cq:CompetencyQuestion ;
    cq:question "In which country is Rotterdam?" ;
    cq:cqSpec :test_example .        # optional — omit for a CQ with no test yet

Because the link is optional, you can record a CQ before writing its test; the report lists such CQs as gaps (Test column "—").

Two opt-in report flags build on this, and compose:

• --cq adds a Competency Questions table (one row per CQ node — its linked test(s) and status) and a per-CQ breakdown. Needs no ontology.
• --term-coverage adds ontology term coverage over all mustrd tests (see below). Needs an ontology.

Plain --md (neither flag) is unchanged: it still writes the standard test-results summary.

pytest --mustrd --config=config.ttl --md=report.md            # test-results summary
pytest --mustrd --config=config.ttl --cq --md=report.md       # + competency questions

Ontology term coverage

--term-coverage reports how much of your ontology your tests actually exercise — an overall percentage and a per-term table (to stdout, and the --md file if given). Add --cq too and it also shows how much is backed by a competency question (a stricter number) via a CQ Term Coverage column.

Tell MustRD which ontology to measure against with :hasOntologyPath in your config — a file or a directory (scanned recursively), repeatable:

:myTest a :MustrdTest ;
    :hasSpecPath     "specs/" ;
    :hasDataPath     "data/" ;
    :hasOntologyPath "ontology/" ;   # file or directory; repeat for several
    :filterOnTripleStore triplestore:RdfLib .

pytest --mustrd --config=config.ttl --term-coverage             # coverage to stdout
pytest --mustrd --config=config.ttl --term-coverage --cq --md=report.md

(If --term-coverage is set without :hasOntologyPath, MustRD fails early and tells you exactly what to add.)

Coverage is data-based: a declared term counts as covered when a passing test populates it in input data (as an instance type or asserted predicate) — whether or not a query also names it. A term named only in a query but never instantiated is query-only and does not count (the test passes without it); it's often a sign a TBox axiom belongs in the ontology, not the fixture — the report flags those under ⚠️ TBox axioms in test data. Terms that are only structurally referenced (the rdfs:domain/rdfs:range of a used property, a superclass of a used class, or a metadata property such as an owl:AnnotationProperty/owl:OntologyProperty) are reported separately as structural terms and excluded from the percentage. Every term is classified as fully exercised, data-only, query-only, structural, or unused — so untested terms surface immediately. When you pass --md, the parent directory is created automatically if it doesn't exist.

RDF output. --term-coverage-rdf=coverage.ttl writes the result as RDF (W3C DQV + PROV) for a knowledge graph: quality measurements computedOn the ontology IRI and its owl:versionIRI (value a decimal ratio), a per-term breakdown, and quality issues — all with stable IRIs, no blank nodes.

See docs/ontology-term-coverage.md for the full definition and docs/examples/geography-example/report/term-coverage-example.md for sample output.

The HTML report

--viewer=report.html writes one self-contained HTML file: no build step, no CDN, no server. It carries the run's RDF, a Turtle parser and VanJS (~5KB, MIT, vendored inline), and builds everything in the browser — a pass/fail/skip test tree with timings, the term coverage table (classes nested by rdfs:subClassOf, properties under their rdfs:domain), the competency questions, and the quality issues. Attach it to a CI run, email it, or open it from disk.

pytest --mustrd --config=config.ttl --viewer=report.html

--viewer implies the coverage and competency-question graphs, so you don't need --term-coverage/--cq as well (they only affect the terminal and --md output). Coverage appears whenever the config declares :hasOntologyPath.

Sources are embedded. By default each spec's Turtle and the SPARQL it ran are inlined into the page, syntax-highlighted, so the report is readable without the files it was generated from — a path in a graph only resolves from the directory the run happened in, which is no use in an emailed file or a CI artifact. Any file reference in the report opens the embedded copy in place. --no-viewer-sources turns this off for a smaller page.

The page is also a viewer for any mustrd graph: drop a .ttl or .jsonld from --term-coverage-rdf / --results-rdf onto it, or point it at one with ?ttl=path/to/run.ttl. Drop several to compare or merge runs.

Related flags: --results-rdf / --results-jsonld write the per-test results graph (every test, passed/failed/skipped, with timing) on its own; --term-coverage-jsonld writes the coverage graph as JSON-LD. --viewer-title sets the page title, and --viewer-src-base prefixes the page's source-file links when it is served from somewhere other than the working directory.

Live example report.

The mustrd CLI

The pytest plugin is a front end, not the engine. Spec execution (mustrd.runner) and reporting (mustrd.reporting) are plain libraries, and the mustrd command drives them without pytest — same config file, same specs, same reports:

mustrd run    --config config.ttl                      # run the specs, review results
mustrd report --config config.ttl --viewer report.html  # run + emit the reports

config.ttl is your own MustrdTest configuration — the same file pytest --mustrd --config= takes. To try it against the worked example in this repo:

mustrd report --config docs/examples/geography-example/mustrd-config.ttl \
              --viewer report.html

Paths inside a config are resolved relative to the config file, so it can be run from anywhere; the viewer's source links are relative to the working directory (see --viewer-src-base).

mustrd report takes the same reporting flags as the plugin (--md, --term-coverage, --cq, --term-coverage-rdf, --results-rdf, --viewer, …) plus --ontology to override :hasOntologyPath. Both exit non-zero if any spec does not pass.

When?

MustRD is a work in progress, built to meet the needs of our projects across multiple clients and vendor stacks. While we find it useful, it may not meet your needs out of the box.

We invite you to try it, raise issues, or contribute via pull requests. If you need custom features, contact us for consultancy rates, and we may prioritize your request.

Contributing: see CONTRIBUTING.md for setup, the two test suites, and the traps.

Releasing

Maintainers: releases are cut by pushing a version tag (git tag 0.7.5 && git push origin 0.7.5). See RELEASING.md for the full flow, including beta/candidate releases.

Support

Semantic Partners is a specialist consultancy in Semantic Technology. If you need more support, contact us at info@semanticpartners.com or mustrd@semanticpartners.com.