Mash

Mash attempts to solve the mess that is working with data spread between systems and people by making it easy to store your data alongside rich metadata. This brings with it organization, searchability, traceability and many other useful tings.

In practice mash allows you to:

• Search your data by any criteria from metadata
• See where your data originated and its history
• Validate that your data meets the criteria for a specified usage
• Connect to remote data sources while maintaining all of the above

Batteries-included

Mash comes pre-loaded with useful tools for working with RDF and grid models. The most helpful things for getting started are the profiles, standards and organizations found in the grid module. These can be used to search, validate and describe our data.

Standards-based

All models used in mash are based on widely-implemented standard vocabularies like dcat, prof and prov-o, and can be serialized into RDF following those standards.

Further documentation

Some deeper documentation can be found here:

• Introduction to Mash
• Datasets and distributions
• Profiles
• Developer guidelines

Examples of usage

(more incoming!)

Define new data using datasets and distributions

import mash

from mash import models
from mash.core.profiles import MetadataProfile
from mash.core.utilities.organizations import make_organization


dist_id = "http://example.com/example#metadata-plaintext"
dataset = models.Dataset(
    identifier="http://example.com/example#metadata",
    title="Example metadata",
    description="Example dataset containing metadata with little",
    conforms_to=[MetadataProfile],  # This is a lie
    information_owner=make_organization("ElBits AS", "931264079"),
    distributions=[models.Distribution(identifier=dist_id, mime_type="text/plain")],
)

mash.save(dataset)
data = b"this is some text"
mash.write_data(data, dist_id)


print(mash.read_data(dist_id).decode())

Write a SPARQL query towards a graph

import mash

rdf_dataset = mash.fetch("urn:uuid:5393e12b-96c3-4fab-8b98-d4ba2e940b94")
m = mash.model_from_dataset(rdf_dataset)
res = m.query("""
    PREFIX cim: <http://iec.ch/TC57/CIM100#>
    SELECT (COUNT(?s) as ?substation_count) WHERE {
        ?s a cim:Substation
    }
""")
print(res)

Mash ORM

Mash features a simple RDF ORM which allows you to extract data from RDF graphs into Pydantic BaseModel classes, and to serialize BaseModel classes into RDF graphs. Using the SemanticModel class and Predicate annotation you can define your own rich metadata classes - either based on existing standards or your own models.

A basic example

from typing import Annotated, ClassVar
from mash import models, serialize, Predicate as P


class Person(models.SemanticModel):
    _type: ClassVar = "foaf:Person"
    name: Annotated[str, P("foaf:name")]
    mail: Annotated[str | None, P("foaf:mbox")] = None
    friends: Annotated[list["Person"], P("mash:friends")] = []


kristoffer_robin = Person(identifier="kr", name="Kristoffer Robin")
ole_brumm = Person(identifier="ob", name="Ole Brumm", mail="ole.brum@100meter.skogen", friends=[kristoffer_robin])
turtle_text = serialize(ole_brumm)
print(turtle_text)

<urn:mash:kr>    a foaf:Person ;
    foaf:name "Kristoffer Robin" .

<urn:mash:ob>    a foaf:Person ;
    mash:friends <urn:mash:kr> ;
    foaf:mbox "ole.brum@100meter.skogen" ;
    foaf:name "Ole Brumm" .

Making data easier to parse with ReverseRelations

Normally in RDF some relations might go in directions that are not pracitical. If you want everything attached to one class rather than to point in all directions from different classes, you can achieve this with ReverseRelations.

Notice how in the previous example Substation has no references to VoltageLevel or SubstationPart since those actually contain the reference to Substation.

from typing import Annotated, ClassVar

import mash

from mash import Predicate as P, ReverseRelation
from mash.grid import get_latest_nemo
from tests.grid.profiles.nemo import IdentifiedObject, VoltageLevel, SubstationPart, NemoProfile


class SubstationEmbed(IdentifiedObject):
    _type: ClassVar = "cim:Substation"
    substation_kind: Annotated[list[str], P("elb:Substation.substationKind")]
    substation_parts: Annotated[list[SubstationPart], ReverseRelation("elb:SubstationPart.Substation")]
    voltage_levels: Annotated[list[VoltageLevel], ReverseRelation("cim:VoltageLevel.Substation")]


latest_nemo = get_latest_nemo()
m = mash.model_from_dataset(latest_nemo)
with NemoProfile:
    substations = mash.instantiate(m, model=SubstationEmbed)
print(substations[0:2])

Use namespaces and models from a profile

You might have conflicting namespaces (links between a prefix like cim and its full URI like http://iec.ch/TC57/CIM100) in different contexts, and you might have different models that represent a resource with the same type.

Using profiles you can group models and namespaces, and tell mash when to use them. This avoids conflicts and confusion in the code.

Looking at a part of the example above, you will see the context manager that activates a profile (with NemoProfile:). Within this block mash will use the namespaces placed on Nemo, and will recognize Nemo models when relevant.

In this particular example the namespaces are important, since the grid module and the NemoProfile have different URI's for the prefix elb.

latest_nemo = get_latest_nemo()
m = mash.model_from_dataset(latest_nemo)
with NemoProfile:
    substations = mash.instantiate_all(m, model=SubstationEmbed)

Data catalog

To open a visual catalog of the data mash has access to, simply run uv run poe catalog and a page will open in your browser.