maplib: High-performance RDF knowledge graph construction, SHACL validation, SPARQL and Datalog in Python
maplib is written in Rust, it is built on Apache Arrow using Pola.rs and uses libraries from Oxigraph for handling linked data as well as parsing SPARQL queries.
maplib allows you to leverage your existing skills with Pandas or Polars to extract and wrangle data from existing databases and spreadsheets, before applying simple templates to them to build a knowledge graph. You can also read knowledge graphs extremely quickly from a wide variety of serialization formats. Using the built-in SPARQL, SHACL and Datalog engines means you can query, inspect, enrich and validate and then serialize the knowledge graph immediately. All query results are Polars Dataframes that are transferred zero-copy from Rust to Python. Currently, maplib is in-memory and supports around 100M triples on 32GB of RAM.
The core functionality of maplib (mapping, querying, serialization) is open source, but SHACL and Datalog functionality are not. Please send us a message, e.g. on LinkedIn (search for Data Treehouse) or on email (magnus at data-treehouse.com) if you want to try out these features. See our roadmap for upcoming features.
Installing
The package is published on PyPi and the API documented here:
pip install maplib
Model
We can easily map DataFrames to RDF-graphs using the Python library. Below is a reproduction of the example in the paper [1]. Assume that we have a DataFrame given by:
import polars as pl
pl.Config.set_fmt_str_lengths(150)
pi = "https://github.com/DataTreehouse/maplib/pizza#"
df = pl.DataFrame({
"p":[pi + "Hawaiian", pi + "Grandiosa"],
"c":[pi + "CAN", pi + "NOR"],
"ings": [[pi + "Pineapple", pi + "Ham"],
[pi + "Pepper", pi + "Meat"]]
})
print(df)
That is, our DataFrame is:
| p | c | ings |
|---|---|---|
| str | str | list[str] |
| "https://.../pizza#Hawaiian" | "https://.../maplib/pizza#CAN" | [".../pizza#Pineapple", ".../pizza#Ham"] |
| "https://.../pizza#Grandiosa" | "https://.../maplib/pizza#NOR" | [".../pizza#Pepper", ".../pizza#Meat"] |
Then we can define a OTTR template, and create our knowledge graph by expanding this template with our DataFrame as input:
from maplib import Model, Prefix, Template, Argument, Parameter, Variable, RDFType, Triple, a
pi = Prefix(pi)
p_var = Variable("p")
c_var = Variable("c")
ings_var = Variable("ings")
template = Template(
iri= pi.suf("PizzaTemplate"),
parameters= [
Parameter(variable=p_var, rdf_type=RDFType.IRI()),
Parameter(variable=c_var, rdf_type=RDFType.IRI()),
Parameter(variable=ings_var, rdf_type=RDFType.Nested(RDFType.IRI()))
],
instances= [
Triple(p_var, a, pi.suf("Pizza")),
Triple(p_var, pi.suf("fromCountry"), c_var),
Triple(
p_var,
pi.suf("hasIngredient"),
Argument(term=ings_var, list_expand=True),
list_expander="cross")
]
)
m = Model()
m.map(template, df)
hpizzas = """
PREFIX pi:<https://github.com/DataTreehouse/maplib/pizza#>
CONSTRUCT { ?p a pi:HeterodoxPizza }
WHERE {
?p a pi:Pizza .
?p pi:hasIngredient pi:Pineapple .
}"""
m.insert(hpizzas)
return m
We can immediately query the mapped knowledge graph:
m.query("""
PREFIX pi:<https://github.com/DataTreehouse/maplib/pizza#>
SELECT ?p ?i WHERE {
?p a pi:Pizza .
?p pi:hasIngredient ?i .
}
""")
The query gives the following result (a DataFrame):
Next, we are able to perform a construct query, which creates new triples but does not insert them.
hpizzas = """
PREFIX pi:<https://github.com/DataTreehouse/maplib/pizza#>
CONSTRUCT { ?p a pi:UnorthodoxPizza }
WHERE {
?p a pi:Pizza .
?p pi:hasIngredient pi:Pineapple .
}"""
res = m.query(hpizzas)
res[0]
The resulting triples are given below:
| subject | verb | object |
|---|---|---|
| str | str | str |
| "https://.../pizza#Hawaiian" | "http://.../22-rdf-syntax-ns#type" | "https://.../pizza#UnorthodoxPizza" |
If we are happy with the output of this construct-query, we can insert it in the model state. Afterwards we check that the triple is added with a query.
m.insert(hpizzas)
m.query("""
PREFIX pi:<https://github.com/DataTreehouse/maplib/pizza#>
SELECT ?p WHERE {
?p a pi:UnorthodoxPizza
}
""")
Indeed, we have added the triple:
| p |
|---|
| str |
| "https://github.com/DataTreehouse/maplib/pizza#Hawaiian" |
API
The API is simple, and contains only one class and a few methods for:
- expanding templates
- querying with SPARQL
- validating with SHACL
- reading JSON to triples using Façade-X
- importing triples (Turtle, RDF/XML, NTriples, JSON-LD, HDT)
- writing triples (Turtle, RDF/XML, NTriples)
- creating a new Model from a named graph
The API is documented HERE
Roadmap of features and optimizations
Spring 2026
- SHACL Rules
- Disk based storage and internal serialization format
- Jelly
Roadmap is subject to changes,particularly user and customer requests.
References
There is an associated paper [1] with associated benchmarks showing superior performance and scalability that can be found here. OTTR is described in [2].
[1] M. Bakken, "maplib: Interactive, literal RDF model model for industry," in IEEE Access, doi: 10.1109/ACCESS.2023.3269093.
[2] M. G. Skjæveland, D. P. Lupp, L. H. Karlsen, and J. W. Klüwer, “Ottr: Formal templates for pattern-based ontology engineering.” in WOP (Book), 2021, pp. 349–377.
Contributing
Any contributions will fall under the Apache 2.0 license of maplib.
Licensing
All code produced since August 1st. 2023 is copyrighted to Data Treehouse AS with an Apache 2.0 license unless otherwise noted.
All code which was produced before August 1st. 2023 copyrighted to Prediktor AS with an Apache 2.0 license unless otherwise noted, and has been financed by The Research Council of Norway (grant no. 316656) and Prediktor AS as part of a PhD Degree. The code at this state is archived in the repository at https://github.com/magbak/maplib.
1 maintainer
Metadata
1 maintainer
Release files for maplib 0.20.26
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| maplib-0.20.26.tar.gz | 427.5 kB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| maplib-0.20.26-cp310-abi3-win_amd64.whl | CPython 3.10 | abi3 | Windows x86-64 | Details |
| maplib-0.20.26-cp310-abi3-manylinux_2_28_x86_64.whl | CPython 3.10 | abi3 | Linux glibc 2.28+ x86-64 | Details |
| maplib-0.20.26-cp310-abi3-manylinux_2_28_aarch64.whl | CPython 3.10 | abi3 | Linux glibc 2.28+ ARM64 | Details |
| maplib-0.20.26-cp310-abi3-macosx_11_0_arm64.whl | CPython 3.10 | abi3 | macOS 11.0+ ARM64 | Details |
| maplib-0.20.26-cp310-abi3-macosx_10_12_x86_64.whl | CPython 3.10 | abi3 | macOS 10.12+ x86-64 | Details |
Total release size: 142.4 MB
Release files / maplib-0.20.26.tar.gz
| Download URL | maplib-0.20.26.tar.gz |
|---|---|
| Size | 427.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4842a5fcb5f40396e9d9185a83ca6b6dba12c799e80675bde97fb5e3bf55753c
|
|
BLAKE2b-256 checksum How to use checksums |
6796d04efa5166fbd4f0d71447c462e7fcf5f0e494a60b666258f1b77b18729b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.9.4
|
Release files / maplib-0.20.26-cp310-abi3-win_amd64.whl
| Download URL | maplib-0.20.26-cp310-abi3-win_amd64.whl |
|---|---|
| Size | 30.6 MB |
| Tags | CPython 3.10 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
a12d6e09532cefa2d9396c8bfe3d0f2ed1c9cc223afb103a7b553fa1088c3b11
|
|
BLAKE2b-256 checksum How to use checksums |
869ab8cb35add41c7df741b198212258625e04e470ab94b80df5c6ac3092ff23
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.9.4
|
Release files / maplib-0.20.26-cp310-abi3-manylinux_2_28_x86_64.whl
| Download URL | maplib-0.20.26-cp310-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 29.6 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
3bb27c01fc95cc47827a60daf1a589241526b6fb734db1c9d1efc97e4b3c4dc1
|
|
BLAKE2b-256 checksum How to use checksums |
fe9dacd3bbf948b47483b7e369afa870b4e7cdfa004580f8926e9a1d262782aa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.9.4
|
Release files / maplib-0.20.26-cp310-abi3-manylinux_2_28_aarch64.whl
| Download URL | maplib-0.20.26-cp310-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 27.5 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
96a2f7215f8c7f17b96b17c5c8daea42efcf348f9301bca7e8cac57d347e2101
|
|
BLAKE2b-256 checksum How to use checksums |
cac42a6000f228b21cbcc423d257725a0399251aa6bb27f6938b283256181b78
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.9.4
|
Release files / maplib-0.20.26-cp310-abi3-macosx_11_0_arm64.whl
| Download URL | maplib-0.20.26-cp310-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 26.2 MB |
| Tags | CPython 3.10 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
86232af69ce92b2271ec46eef17a9b0a1e39c18388e0bb0f56fd5789cf6dccb8
|
|
BLAKE2b-256 checksum How to use checksums |
69fad066dd1fad2ef3a4354d1ac47e3d4bd097ae411d6157c874971af3f7daff
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.9.4
|
Release files / maplib-0.20.26-cp310-abi3-macosx_10_12_x86_64.whl
| Download URL | maplib-0.20.26-cp310-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 28.1 MB |
| Tags | CPython 3.10 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
fb4b1f2befb1acfc0a73d3ee25a16e761bca809143c00f57679fcab2604dcd6f
|
|
BLAKE2b-256 checksum How to use checksums |
1850fb1f853c08ae65beb4e01fd4789f4c57779e71e466cd9607263a3ddab408
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.9.4
|
