tokamap 0.2.0

A JSON schema-based framework for mapping and structuring experimental data

TokaMap

TokaMap is a JSON schema-based framework for mapping and structuring experimental data. It provides a standardized way to define how experimental measurements, calculations, and data transformations should be organized and validated.

The Python package contains the following components:

• tokamap: A module that provides details about the install, such as the schemas directory
• tokamap.validator: The validation module for ensuring data consistency and correctness

And the following CLI tools:

• tokamap: Command-line interface for querying details about the install, such as the schemas directory
• tokamap-validator: Command-line interface for validating TokaMap mappings

Overview

TokaMap defines a set of JSON schemas that allow researchers and engineers to:

• Map experimental data using structured configuration files
• Define data sources with standardized parameters and arguments
• Create expressions and calculations from mapped data
• Partition and group data based on experimental attributes
• Validate mappings to ensure data consistency and correctness

Installation

From PyPI

pip install tokamap

From Source

git clone https://github.com/ukaea/tokamap.git
pip install .

TokaMap CLI

Finding the schemas directory:

tokamap --schemas-dir

Checking the installed version:

tokamap --version

TokaMap library

Finding the schemas directory:

from tokamap import schemas_dir

print(schemas_dir)

Checking the installed version:

from tokamap import __version__

print(__version__)

Validation

TokaMap includes a Python validator tool to ensure your mapping files conform to the schemas.

Usage

Validate a TokaMap mapping directory:

tokamap-validator /path/to/mapping/directory

For verbose output:

tokamap-validator -v /path/to/mapping/directory

Check the installed version:

tokamap-validator --version

The validator will:

1. Check that the configuration file (mappings.cfg.json) exists and is valid
2. Validate the top-level globals file
3. Validate each mapping group's globals and mappings files

Validator Requirements

• Python >= 3.13
• jsonschema >= 4.25.0

Architecture

The TokaMap system consists of three main schema components:

1. Mappings Schema (mappings.schema.json)

Defines the structure for mapping experimental data with five main mapping types:

• DIMENSION: Maps dimensional probe data
• VALUE: Maps static values (numbers, strings, arrays, objects)
• DATA_SOURCE: Maps external data sources with configurable arguments, offsets, scaling, and slicing
• EXPR: Maps mathematical expressions with parameters
• CUSTOM: Maps custom functions from external libraries

2. Globals Schema (globals.schema.json)

Defines global configuration settings, particularly data source configurations with their associated arguments.

3. Configuration Schema (mappings.cfg.schema.json)

Defines the top-level configuration structure including:

• Metadata: Experiment information, author, and version
• Partitions: Data partitioning rules with selectors (max_below, min_above, exact, closest)
• Groups: Array of group identifiers

Schema Structure

Mapping Types

Each mapping entry can be one of several types:

Dimension Mapping

{
  "MAP_TYPE": "DIMENSION",
  "DIM_PROBE": "probe_identifier",
  "COMMENT": "Optional description"
}

Value Mapping

{
  "MAP_TYPE": "VALUE",
  "VALUE": "any_value_type",
  "COMMENT": "Optional description"
}

Data Source Mapping

{
  "MAP_TYPE": "DATA_SOURCE",
  "DATA_SOURCE": "source_name",
  "ARGS": {
    "arg1": "value1",
    "arg2": true
  },
  "OFFSET": 0,
  "SCALE": 1.0,
  "SLICE": "0:10",
  "COMMENT": "Optional description"
}

Expression Mapping

{
  "MAP_TYPE": "EXPR",
  "EXPR": "mathematical_expression",
  "PARAMETERS": {
    "param1": "value1"
  },
  "COMMENT": "Optional description"
}

Custom Function Mapping

{
  "MAP_TYPE": "CUSTOM",
  "LIBRARY": "library_name",
  "FUNCTION": "function_name",
  "INPUTS": {
    "input1": "value1"
  },
  "PARAMETERS": {},
  "COMMENT": "Optional description"
}

Directory Structure

A typical TokaMap project follows this structure:

mapping_root/
├── mappings.cfg.json          # Main configuration file
├── globals.json               # Global settings
├── mapping_group_1/
│   ├── globals.json          # Group-specific globals
│   └── mappings.json         # Group mappings
└── mapping_group_2/
    ├── globals.json
    └── mappings.json

Getting Started

1. Create a configuration file (mappings.cfg.json) with your experiment metadata:

{
  "metadata": {
    "experiment": "your_experiment_name",
    "author": "Your Name",
    "version": "1.0.0"
  },
  "partitions": [
    {
      "attribute": "time",
      "selector": "closest"
    }
  ],
  "groups": ["group1", "group2"]
}

2. Define global settings in globals.json:

{
  "DATA_SOURCE_CONFIG": {
    "your_data_source": {
      "ARGS": {
        "connection_string": "your_connection"
      }
    }
  }
}

3. Create mapping directories for each group with their respective mappings.json and globals.json files.

4. Validate your configuration using the TokaMap validator.

Schema Versioning

TokaMap schemas follow semantic versioning. The current version is 1.0 as indicated in the JSON Schema $id fields.

Contributing

TokaMap is developed to support experimental data mapping workflows. The schemas are designed to be extensible while maintaining validation integrity.

License

See the main project LICENSE file for license information.