fog-rml 1.0.0

fog-rml: An algebra-based RML engine for Knowledge Graph Construction.

fog-rml

A Python implementation of the formal algebra for Knowledge Graph Construction, based on the work of Olaf Hartig. An Algebraic Foundation for Knowledge Graph Construction

---

Table of Contents

• 1. Project Context
• 2. Quick Start
• 3. Features
• 4. Command Line Interface (CLI)
• 5. Python API Usage
• 6. Project Structure
• 7. Testing
• 8. Authors
• 9. License
• 10. Acknowledgements
• 11. Contact

1. Project Context

This library is a research project developed for the "Engineering For Research I" module.

It is part of the M1 Computer Science, SMART Computing Master's Program at Nantes Université.

The project is hosted by the LS2N (Laboratoire des Sciences du Numérique de Nantes), within the GDD (Gestion des Données Distribuées) team.

It serves as the core logical component for the MCP-SPARQLLM project, aiming to translate heterogeneous data sources into RDF Knowledge Graphs via algebraic operators.

---

2. Quick Start

1. Installation

This project is designed to be installed in "editable" mode for development.

# Clone the repository
git clone [https://github.com/Armotik/fog-rml](https://github.com/Armotik/fog-rml)
cd fog-rml

# Create a virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -e '.[test]'

2. Verify Installation

Run the help command to ensure everything is set up correctly:

fog-rml --help

---

3. Features

fog-rml provides a set of composable Python objects representing the core algebraic operators for querying heterogeneous data sources.

Current implementation status covers the foundations required to reproduce Source, Extend, Union, and Project operators as defined in the paper:

• Algebraic Structures: Strict typing for MappingTuple, IRI, Literal, BlankNode, and the special error value EPSILON ($\epsilon$).
• Source Operator:
  • Extracts data from heterogeneous sources (JSON support implemented).
  • Handles iteration logic (JSONPath) and Cartesian Product flattening for multivalued attributes.
• Extend Operator:
  • Dynamically creates new attributes based on expressions (φ).
  • Used to generate IRIs, Literals, or derived values.
• Union Operator:
  • Merges data streams from multiple pipelines.
  • Supports Bag Semantics (preserves duplicates) and maintains tuple order.
• Project Operator:
  • Restricts the relation to a specific subset of attributes.
  • Enforces strict schema validation (P⊆A).
• EquiJoin Operator
  • Joins two data streams based on equality conditions
  • Essential for handling RML Referencing Object Maps (Foreign Keys).
• Expression System ($\varphi$):
  • Composite pattern implementation for recursive expressions.
  • Supports Constant, Reference (attributes), and FunctionCall.
• Built-in Functions:
  • Implementation of Annex B functions: toIRI, toLiteral, concat.
  • Strict error propagation handling (Epsilon).
• RML Mapping Support:
  • RML Parser: Compiles declarative RML mapping files (.ttl) into an optimized algebraic plan.
  • Built-in Functions: Supports standard transformation functions (toIRI, toLiteral, concat, toBNode).
  • Strict Typing: Operates strictly on RDF Terms (IRI, Literal, BlankNode) rather than raw Python strings.
• Pipeline Visualization:
  • explain() method for human-readable pipeline trees
  • explain_json() method for programmatic access to pipeline structure
  • Detailed expression and operator visualization

4. Command Line Interface (CLI)

fog-rml provides a plugin-based CLI architecture for executing mappings and utility tasks.

3.1. Execute a Mapping (run)

The standard command to execute an RML file and generate N-Triples.

fog-rml -v run \
    -m mappings/my_mapping.ttl \
    -o output.nt

• -m / --mapping: Path to the RML mapping file (Turtle format).
• -o / --output: Path to the output N-Triples file.
• -v : Verbose mode (logs info). -vv for debug.
• --explain: Print the algebraic execution plan instead of running it.

3.2. Multi-Repo Aggregation (list-issues)

A specialized command demonstrating the dynamic generation of Knowledge Graphs from multiple GitHub/GitLab repositories.

fog-rml -vv list-issues \
     https://github.com/facebook/react \
     https://github.com/tensorflow/tensorflow \
     https://gitlab.com/gitlab-org/gitlab \
     https://gitlab.com/inkscape/inkscape \
     -m data/mappings/commands/issue_template.ttl \
     -o output_issues.nt

• Auto-Detection: Automatically detects GitHub vs GitLab URLs.
• Aggregation: Fetches data from APIs and merges them into unified sources.
• Templating: Injects data into the provided RML template before execution.

3.3. Run External RML Conformance Tests

Use the conformance runner to execute external RML test cases and compare fog-rml output with expected RDF.

python scripts/run_rml_conformance.py \
    --tests-dir external/rml-test-cases/test-cases \
    --output-dir external/rml-test-cases/results

Optional flags:

• --suite <name>: Restrict execution to a sub-suite/folder.
• --verbose: Print per-case execution details.

Example (single case):

python scripts/run_rml_conformance.py \
    --tests-dir external/rml-test-cases/test-cases/RMLTC0007g-JSON \
    --output-dir external/rml-test-cases/results

3.4. List Articles Use Case (list-articles)

A convenience command to fetch article metadata from several external sources (OpenAlex, HAL, DBLP, SerpAPI), normalize the results, inject them into a provided RML mapping, and materialize RDF quads per-author.

Usage:

fog-rml list-articles "<Author Name>" \
    -m <mapping.ttl> \
    -o <outdir> \
    --sources openalex,hal,dblp,serpapi \
    --start-year 2020 --end-year 2026

Flags and arguments:

• <Author Name>: Positional argument — the author to search for (quoted if it contains spaces).
• -m, --mapping: Path to the RML mapping file (Turtle). The mapping is expected to reference local JSON source files (see tests/use_cases/list_articles/data/).
• -o, --outdir: Directory where output will be written. By default the command writes beside the mapping file.
• --sources: Comma-separated list of sources to fetch. Supported values: openalex, hal, dblp, serpapi.
• --start-year / --end-year: Integer year bounds to filter fetched results.
• --explain: Print the algebraic execution plan for the mapping instead of executing it.
• -v / -vv: Verbosity flags for logging output.

Behavior and notes:

• The command will run fetch scripts (shell or the Python fallback) to populate *.json files expected by the mapping. If SerpAPI is used, set SERPAPI_KEY in the environment or place it in a .env file in the mapping data folder.
• Output filename: the command writes N-Quads named after the sanitized author, e.g. Pascal_Molli.nq, colocated with the mapping by default (or under --outdir if provided). Serializer selection follows the output extension (.nq → N-Quads).
• Extension points: The mapping may call FnML/FnO extension functions; register any required functions (see the functions package) before running.

Example (run for Pascal Molli mapping in tests):

fog-rml list-articles "Pascal Molli" \
    --mapping tests/use_cases/list_articles/data/mapping.ttl \
    --outdir tests/use_cases/list_articles \
    --sources openalex,hal,dblp,serpapi \
    --start-year 2020 --end-year 2026

Behavior summary:

• Detects mapping + expected output files per case.
• Executes fog-rml run with an output format matching the expected file extension.
• Compares output via RDF graph isomorphism (triples/quads aware).
• Uses metadata.csv when available to track expected-error cases.
• Writes per-case artifacts to results/<case>/ (mapping/resources + output_fog-rml.*).

5. Python API Usage

You can embed the engine directly in your own Python scripts.

4.1. Basic Pipeline Execution

from fog-rml.mapping.MappingParser import MappingParser

# 1. Initialize the parser
parser = MappingParser("data/mappings/fusion_mapping.ttl")

# 2. Parse into an algebraic pipeline (Operator Tree)
pipeline = parser.parse()

# 3. (Optional) Visualize the plan
print(pipeline.explain())

# 4. Execute (Lazy Iterator)
results = pipeline.execute()

# 5. Process results
for row in results:
    # row is an immutable MappingTuple
    print(f"{row['subject']} {row['predicate']} {row['object']} .")

4.2. Manual Pipeline Construction

from fog-rml.operators.sources.JsonSourceOperator import JsonSourceOperator
from fog-rml.operators.ExtendOperator import ExtendOperator
from fog-rml.functions.builtins import to_iri, concat
from fog-rml.expressions import FunctionCall, Reference, Constant

# 1. Define Source
source = JsonSourceOperator(
    source_data={"users": [{"id": 1, "name": "Alice"}]},
    iterator_query="$.users[*]",
    attribute_mappings={"uid": "id", "name": "name"}
)

# 2. Add Transformation (URI Generation)
# Expression: to_iri(concat("[http://ex.org/](http://ex.org/)", Ref("uid")))
uri_expr = FunctionCall(
    to_iri,
    [FunctionCall(concat, [Constant("[http://ex.org/](http://ex.org/)"), Reference("uid")])]
)

pipeline = ExtendOperator(source, "subject", uri_expr)

# 3. Execute
for row in pipeline.execute():
    print(row)

4.3. Registering FnO functions (FunctionRegistry)

This project supports FnML/FnO style extension functions by exposing a small plugin registry and runtime resolution mechanism. The goal is to let mapping authors refer to functions by standard FnO URIs while implementers supply Python callables that perform the actual work — without editing core code.

What it does

• Provides a global FunctionRegistry to bind FnO URI strings to Python callables.
• Parses fnml:functionValue / fno:executes mapping constructs into FunctionCall expression nodes.
• Resolves and invokes registered callables at evaluation time, supporting nested function values and reference/constant arguments.

How it works

• Register: call FunctionRegistry.register(uri, callable) from your application or a plugin module.
• Parse: MappingParser converts fnml:functionValue declarations into FunctionCall(func_iri, args...).
• Resolve: when FunctionCall.evaluate() runs, if its function is a string IRI the registry is queried for the corresponding callable; resolution is lazy so registration may occur before or after parsing.
• Execute: the callable is invoked with evaluated positional arguments; its return value is inserted into the algebraic pipeline (preferably as a fog-rml RDF term).

Inputs and outputs

• Mapping-time input: a term map node using fnml:functionValue and an fno:executes URI, plus any predicateObjectMap parameters (constants, rml:reference, nested functionValues).
• Runtime input to the Python callable: evaluated argument values (often fog-rml.algebra.Terms.Literal, IRI, or BlankNode objects). Callables should unwrap .lexical_form / .value as needed.
• Output from the callable: ideally a fog-rml RDF term (IRI, Literal, BlankNode) so the pipeline can use it directly. Returning raw Python primitives is tolerated but may require normalization.

Error semantics

• If an argument evaluates to EPSILON, or if the function lookup fails, or the callable raises an exception, the FunctionCall returns EPSILON to preserve strict error propagation across the algebra.

Security and best practices

• Registered callables execute arbitrary Python code — only register trusted code in production.
• Validate and coerce inputs inside the callable; prefer returning fog-rml RDF term objects.
• Catch exceptions and return EPSILON or a controlled default to avoid crashing the pipeline.

Example

from fog-rml.functions.registry import FunctionRegistry

def my_upper(arg):
        val = getattr(arg, 'lexical_form', None) or getattr(arg, 'value', None) or arg
        return str(val).upper()

FunctionRegistry.register("http://example.com/fn/upper", my_upper)

References / where to look in the codebase

• fog-rml/functions/registry.py — registry API and storage
• fog-rml/expressions/FunctionCall.py — evaluation and EPSILON semantics
• fog-rml/mapping/MappingParser.py — FnML parsing and FunctionCall construction
• tests/test_suite/test_18_fnml_plugins.py and tests/test_suite/test_19_fno.py — concrete examples and tests

Notes

• Registration is lazy — functions can be registered at any time before evaluation.
• Prefer returning RDF term objects from callables to avoid ambiguity and preserve typing.

5.4. SPARQL SERVICE-CALL Example

This project includes a pragmatic SPARQL integration that demonstrates running RML mappings per-repository and querying the materialized named graphs via a SERVICE-CALL-style pattern.

Quick run (from project root):

# activate your virtualenv
source .venv/bin/activate          # Unix/macOS
.venv\Scripts\Activate.ps1       # PowerShell on Windows

# ensure rdflib is installed
pip install rdflib

# run the example demo (prints resulting quads)
python fog-rml/examples/multi_repo_service_demo.py

What the demo does

• Locates example mapping files under fog-rml/examples/data/ for repositories (r1, r2, r3).
• For each ?repo token it runs the corresponding mapping (via MappingParser) and materializes output into a named graph (IRI derived from the repo token).
• Executes a SPARQL query using BIND SERVICE-CALL(?repo, "mapping.ttl") AS ?g plus GRAPH ?g { ... } to read data from the materialized graphs; the demo prints materialized quads in a readable format.

Using the handler from Python code

from rdflib import Dataset
from pathlib import Path
from fog-rml.sparql.service_call import execute_query_with_service_call

ds = Dataset()
query = '''
PREFIX ex: <http://example.org/>
SELECT ?repo ?x ?y ?title WHERE {
    VALUES ?repo { <http://example.org/r1> <http://example.org/r2> <http://example.org/r3> }
    BIND SERVICE-CALL(?repo, "mapping.ttl") AS ?g
    GRAPH ?g { ?x ex:issue ?y . ?y ex:title ?title }
}
'''

res = execute_query_with_service_call(ds, query, Path('fog-rml/examples/data'))
for row in res:
        print(row)

Notes and recommendations

• The handler runs mappings and injects VALUES ?g clauses; when rewritten queries return no rows it falls back to a per-graph evaluation to guarantee results across different SPARQL engine behaviors.
• Ensure the mapping files referenced by SERVICE-CALL exist in the mapping_dir you pass to the handler.
• The demo is an example and can be adapted for production: replace ad-hoc file resolution with repository metadata and register any FnO functions your mappings require.

6. Project Structure

src/fog-rml/
├── algebra/            # Core algebraic definitions
│   ├── Terms.py        # RDF Terms (IRI, Literal, BlankNode)
│   └── Tuple.py        # MappingTuple and Epsilon
├── commands/          # CLI command implementations
│   ├── base.py         # Base command class
│   ├── run.py          # Standard mapping execution command
│   └── list_issues.py  # Multi-repo GitHub/GitLab
├── expressions/        # Recursive expression system 
│   ├── Expression.py   # Abstract base class
│   ├── Constant.py     # Constant values
│   ├── Reference.py    # Attribute references
│   └── FunctionCall.py # Extension function applications
├── functions/          # Extension functions
│   └── builtins.py     # Implementation of toIRI, concat, etc.
├── mapping/            # RML Mapping Parser
│   └── MappingParser.py # Parses RML files into operator pipelines
├── serializers/      # Serialization utilities
│   ├── NTriplesSerializer.py # N-Triples output
│   └── NQuadsSerializer.py # N-Quads output
├── operators/          # Algebraic Operators
│   ├── Operator.py     # Abstract base class for all operators
│   ├── SourceFactory.py # Factory for creating Source operators
│   ├── EquiJoinOperator.py # EquiJoin operator implementation
│   ├── ExtendOperator.py # Extend operator implementation
│   ├── ProjectOperator.py # Project operator implementation
│   ├── UnionOperator.py  # Union operator implementation
│   ├── SourceOperator.py # Abstract Source operator
│   └── sources/        # Source operator implementations
│       └── CsvSourceOperator.py # CSV data source operator
│       └── JsonSourceOperator.py # JSON data source operator
│       └── XmlSourceOperator.py # XML data source operator
│       └── SparqlSourceOperator.py # SPARQL source operator
│       └── MysqlSourceOperator.py # MySQL source operator
│       └── PostgresqlSourceOperator.py # PostgreSQL source operator
│       └── SqlserverSourceOperator.py # SQL Server source operator
├── namespaces.py    # Common RDF namespaces
└── __main__.py       # Entry point for CLI
tests/                  # Unit tests for all components
├── use_cases/        # Example usage scripts
│   └── github_gitlab/ # Example with GitHub and GitLab data
└── test_suite/     # Comprehensive test suite
data/               # Sample data files for testing
└── mappings/       # RML mapping files
    └── commands/   # Mappings for CLI commands
LICENSE                 # MIT License
README.md               # Project documentation
CHANGELOG.md            # Project changelog
pyproject.toml          # Project configuration and dependencies
requirements.txt        # Additional dependencies

7. Testing

To run the test suite, ensure you have installed the testing dependencies and execute:

# Run all tests
pytest tests/

# Run specific category
pytest tests/test_suite/test_13_equijoin_operator.py

6.1. Initial Regression Suite

Run the project regression suite (quiet mode):

python -m pytest -q

6.2. RML Conformance Suite

Run the external RML conformance checks:

python scripts/run_rml_conformance.py \
    --tests-dir external/rml-test-cases/test-cases \
    --output-dir external/rml-test-cases/results

The conformance summary reports:

• total executed cases,
• passed/failed counts,
• expected-error pass/fail counts,
• coverage percentage.

Generated outputs are retained under external/rml-test-cases/results in per-case folders for inspection.

Test Coverage

• Cartesian Product Flattening
• Recursive Expression Evaluation
• Multi-Source Union & Joins
• Strict Mode Projections
• Real-world Data Integration Scenarios

8. Authors

This project is developed by:

• Anthony MUDET
• Léo FERMÉ
• Mohamed Lamine MERAH

8.1. Supervision

This project is supervised by:

• Full Professor Pascal MOLLI
• Full Professor Hala SKAF-MOLLI
• Associate Professor Gabriela MONTOYA

9. License

This project is licensed under the MIT License - see the LICENSE file for details.

10. Acknowledgements

We would like to thank the LS2N and GDD team for their support and resources provided during this project. We also acknowledge the foundational work of Olaf Hartig, which inspired this implementation.

11. Contact

For any questions or contributions, please open an issue or contact the authors directly.