fw-gear-session-validator 0.2.2

Validates a session

Session Validator (session-validator)

Overview

This gear is used to validate a Flywheel session container according to a user provided JSONSchema file. The gear can currently only be run at the session level.

Usage

FAQ

Summary

Validates a session based on a provided validation schema.

License

License: MIT

Classification

Category: Analysis

Gear Level:

• Project
• Subject
• Session
• Acquisition
• Analysis

---

[[TOC]]

---

Inputs

• validation_schema
  • Name: validation_schema
  • Type: file
  • Optional: false
  • Description: The schema file to validate the session against.

Config

• save_validation_report

  • Name: save_validation_report
  • Type: boolean
  • Description: Save the validation report
  • Default: false

• force_rerun

  • Name: force_rerun
  • Type: boolean
  • Description: Force re-validation even if the session has already passed validation
  • Default: false

• debug

  • Name: debug
  • Type: boolean
  • Description: Log debug messages
  • Default: false

Outputs

Files

• validation_report
  • Name: validation_report
  • Type: file
  • Optional: true
  • Description: JSON file including full error report from validation.
  • Notes: This file will be empty if validation passes.

Metadata

The gear creates a session-level qc metadata object called "session-validator" that includes information on the gear run (job_info) and the validation report (jsonschema-validation). The jsonschema-validation metadata object includes the overall state of the validation (PASS|FAIL) and a data object including individual entries for each error reported by the JSONSchema Validator.

The qc metadata object has the below structure:

qc:
  session-validator:
    job_info:
      config:
        debug: <boolean>
        save_validation_report: <boolean>
        force_rerun: <boolean>
      inputs:
        validation_schema:
          file_id: <string>
          file_name: <string>
          parent: <string>
          version: <integer>
      job_id: <string>
      version: <string>
    jsonschema_validation:
      state: <PASS|FAIL>
      data: [ "<list of error objects>" ]

Each error included in the data list includes the following abbreviated information from the standard JSONSchema Validator error dictionary:

"error_message": "str - error message from JSONSchema validation"
"error_type": "str - key in JSONSchema where error occurred"
"error_value": "obj - details of JSONSchema where error occurred"
"item": "<path-to-container-giving-error>"

For detailed examples of validation errors and how to interpret them, see the examples documentation.

Pre-requisites

The gear technically has no prerequisites, as validation can be performed completely on container labels and file names. However, both file-metadata-importer and file-classifier should be run if it is desired to perform validation on container metadata (e.g. file classification or header values).

Prerequisite Gear Runs

A list of gears, in the order they need to be run:

1. File Metadata Importer
   • Level: File
2. File Classifier
   • Level: File

Prerequisite Files

The session-validator gear requires no additional files beyond the user-provided JSONSchema input file.

Prerequisite Metadata

The gear requires no prerequisite metadata, however, users may wish to include metadata fields created by the file-metadata-importer and file-classifier gears in their JSONSchema.

Usage

Description

This gear can be used to validate a Flywheel session container against a JSONSchema. The JSONSchema file is required input to the gear and can include validation criteria at the session, acquisition, and/or file level metadata. The Flywheel session container at which the gear is launched will be validated against whether it meets all of the criteria included in the JSONSchema. Validation criteria can be general (e.g., PatientAge is populated with a number) or more specific (e.g., session container contains files with specific classifications or metadata values).

For practical examples, see the examples documentation which includes:

• Sample JSONSchema files for common use cases
• Example validation error outputs

File Specifications

validation_schema

The validation schema must be a 2020-12 compatible JSONSchema. The input file will be validated at the start of the gear run. The validation schema should assume the Flywheel hierarchy, with the root level being the Flywheel session container, acquisitions being inside the session container, and any files inside the acquisition containers.

See the below for example JSONSchemas for common Use Cases.

Workflow

graph LR;
  A[validation_json]:::input --> C[Validate JSON is Non-Empty]:::container

  subgraph Gear;
    C --> D{Is JSON Valid?}
    D -->|No| F[Return Error: Empty or Invalid JSON]:::container
    D -->|Yes| S{Is Container Type 'Session'?}
    S -->|No| Y[Error: Must Run on Session Container]:::container
    S -->|Yes| P{Is Session Already Tagged PASS?}
    P -->|Yes| FR{Force Rerun Enabled?}
    FR -->|No| Z[Exit: Session Already Validated]:::container
    FR -->|Yes| R[Remove PASS Tag]:::container --> E
    P -->|No| E[Validate Session Container\nAgainst Schema]:::container
  E --> T[Tag Session with QC Result]:::container --> X[QC Metadata Ready]:::container
  end

  X --> H[Output QC Metadata]:::container
  X -.-> V[Optional: Save Validation Report]:::container

  classDef container fill:#57d,color:#fff
  classDef input fill:#7a9,color:#fff
  classDef gear fill:#659,color:#fff

Description of workflow

1. Upload JSONSchema file to project-level attachments (assuming same JSONSchema for all sessions in a project)
2. Select file as input to gear
3. Gear checks if the input JSONSchema is valid
4. Gear checks if the Flywheel container being validated is a session
5. Gear checks if session container has already passed validation
6. If session has already passed validation and force_rerun is disabled (default), gear exits without re-validation
7. If session has already passed validation and force_rerun is enabled, gear removes the PASS tag and proceeds with validation
8. Gear runs the JSONSchema validation if validation has not been run, prior validation has failed, or force_rerun is enabled
9. Gear creates the QC metadata object and tags session based on the validation results
10. Gear updates session-level metadata with the QC metadata object
11. Gear saves optional validation report output file in session-level analysis container

Use Cases

The conditions listed for each use case are not mutually exclusive, as the various validation criteria shown in the below use case examples can be combined. Rather, the conditions listed for each use case indicate what container properties are used in the example schemas.

While, there are several ways of setting up a valid JSONSchema (2020-12) for validating the session container, Flywheel recommends the framework shown below separating the rule definitions for file and acquisition containers from the validation logic block. This framework should generally make the JSONSchema file easier to maintain.

Use Case 1: Validate against acquisition labels

*Conditions*:

• Validate using session container properties
• Validate using acquisition container properties
• Validate using file level properties

For well-behaved studies, it is possible to validate against acquisition container labels to quickly determine if a session is missing a required acquisition.

In the below, example jsonschema, we are checking that a session has at minimum one acquisition with label exactly equal to "T1" and one acquisition with label that contains the string "T2". To make maintaining the jsonschema a bit easier, the example adopts the framework of defining rules in the "$defs" section and then leverage the definitions when creating the validation logic in the subsequent block starting with "type": "object".

In the two definitions, the example uses "const" to require that the acquisition label be exactly "T1" in the first definition, and "pattern" to require that the acquisition label contain the string "T2" somewhere in the second definition. In the case of "pattern", regular expressions can also be used. For example, regular expressions can be leveraged to require the acquisition label to start with a string (e.g., "^T2") or end with a string (e.g., "T2$").

The validation block uses "allOf" to require that the session container being validated must meet all of the rules in the list. This is more rigid than "anyOf", where the session container would be considered valid if it met at least one of the rules in the list. The validation block also leverages "minContains" and "maxContains" to require a session only has one acquisition that satisfies each rule.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "description": "Schema to validate acquisition labels within a session object",

  "$defs": {
    "acquisition_with_label_T1": {
      "description": "An acquisition whose label must be exactly 'T1'",
      "type": "object",
      "properties": {
        "label": { "const": "T1" }
      },
      "required": ["label"]
    },
    "acquisition_with_label_T2": {
      "description": "An acquisition whose label contains 'T2'",
      "type": "object",
      "properties": {
        "label": { "pattern": "T2" }
      },
      "required": ["label"]
    }
  },

  "type": "object",
  "properties": {
    "acquisitions": {
      "description": "List of acquisitions in this session",
      "type": "array",
      "items": { "type": "object" },
      "allOf": [
        {
          "description": "Require exactly one acquisition with label 'T1'",
          "contains": { "$ref": "#/$defs/acquisition_with_label_T1" },
          "minContains": 1,
          "maxContains": 1
        },
        {
          "description": "Require exactly one acquisition with label 'T2'",
          "contains": { "$ref": "#/$defs/acquisition_with_label_T2" },
          "minContains": 1,
          "maxContains": 1
        }
      ]
    }
  },

  "required": ["acquisitions"]
}

Use Case 2: Validate against file modality and classification

*Conditions*:

• Validate using session container properties
• Validate using acquisition container properties
• Validate using file level properties

To validate that a session has files with expected modalities and/or classifications, it is possible to leverage the output metadata from the file-metadata-importer and file-classifier gears.

Using the framework of having rule definitions separate, means that as validation criteria are added to the rules, the overall validation block stays relatively fixed.

In the below example, now each acquisition is required to have at least one file with the specified modality and classification. Using the rules framework means that we can make one-to-one matches between the file and acquisition rules.

Note that as validation criteria are added to the "properties", the values for the respective "required" keys need to be updated to ensure that the schema first checks that the properties exist in the data object being validated before assessing the rules. In this way, the error for a required property not existing will be separate from a session that is missing a file or acquisition without the properties defined in the rules.

Also, note that this example removes the "label" requirement from the acquisition rules. This schema will simply check if there is an acquisition whose file(s) meets the file rules without imposing the requirement that the acquisition label be something specific. It is easy enough to combine the logic from the previous use case to add the acquisition label requirement back in if desired.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "description": "Schema to validate acquisition labels and require files with modality and classification",

  "$defs": {
    "T1_file": {
      "description": "A T1 file with required modality and classification",
      "type": "object",
      "properties": {
        "modality": { "const": "MR" },
        "classification": {
          "type": "object",
          "properties": {
            "Measurement": {
              "description": "Measurment array must contain the value 'T1'",
              "type": "array",
              "items": { "const": "T1" },
              "minItems": 1,
              "maxItems": 1
            },
            "Intent": {
              "description": "Intent array must contain the value 'Structural'",
              "type": "array",
              "items": { "const": "Structural" },
              "minItems": 1,
              "maxItems": 1
            },
            "Features": {
              "description": "Features array must contain both 'MPRAGE' and '3D'",
              "type": "array",
              "minItems": 2,
              "items": { "type": "string" },
              "allOf": [
                { "contains": { "const": "MPRAGE" } },
                { "contains": { "const": "3D" } }
              ]
            }
          },
          "required": ["Measuremet", "Intent", "Features"]
        },
      },
      "required": ["modality", "classification"]
    },

    "T2_file": {
      "description": "A T2 file with required modality and classification",
      "type": "object",
      "properties": {
        "modality": { "const": "MR" },
        "classification": {
          "type": "object",
          "properties": {
            "Measurement": {
              "description": "Measurment array must contain the value 'T2'",
              "type": "array",
              "items": { "const": "T2" },
              "minItems": 1,
              "maxItems": 1
            },
            "Intent": {
              "description": "Intent array must contain the value 'Structural'",
              "type": "array",
              "items": { "const": "Structural" },
              "minItems": 1,
              "maxItems": 1
            },
            "Features": {
              "description": "Features array must contain 'Spin-Echo'",
              "type": "array",
              "items": { "const": "Spin-Echo" },
              "minItems": 1
            }
          },
          "required": ["Measuremet", "Intent", "Features"]
        },
      },
      "required": ["modality", "classification"]
    },

    "acquisition_with_label_T1": {
      "description": "An acquisition that contains at least one T1 file",
      "type": "object",
      "properties": {
        "files": {
          "type": "array",
          "contains": { "$ref": "#/$defs/T1_file" },
          "minContains": 1
        }
      },
      "required": ["files"]
    },

    "acquisition_with_label_T2": {
      "description": "An acquisition that contains at least one T2 file",
      "type": "object",
      "properties": {
        "files": {
          "type": "array",
          "contains": { "$ref": "#/$defs/T2_file" },
          "minContains": 1
        }
      },
      "required": ["files"]
    }
  },

  "type": "object",
  "properties": {
    "acquisitions": {
      "description": "List of acquisitions in this session",
      "type": "array",
      "items": { "type": "object" },
      "allOf": [
        {
          "description": "Require exactly one acquisition with label 'T1' containing at least one MR file",
          "contains": { "$ref": "#/$defs/acquisition_with_label_T1" },
          "minContains": 1,
          "maxContains": 1
        },
        {
          "description": "Require exactly one acquisition with label 'T2' containing at least one MR file",
          "contains": { "$ref": "#/$defs/acquisition_with_label_T2" },
          "minContains": 1,
          "maxContains": 1
        }
      ]
    }
  },

  "required": ["acquisitions"]
}

Use Case 3: Validate against file metadata

*Conditions*:

• Validate using session container properties
• Validate using acquisition container properties
• Validate using file level properties

While the dicom-qc gear is specifically suited to validate the full header extracted by file-metadata-importer, it is possible to validate a session against file metadata to determine if it contains files with the desired imaging parameters or header information.

In the below partial jsonschema example, we show how to specify a file rule to check that certain properties of the file.info.header.dicom metadata object are the specified types or values. Again, since the framework used in these examples separates out the file and acquisition validation criteria into separate, defined rules, no additional changes would need to be made to the acquisition rule associated with this file rule or to the overall jsonschema validation block.

For the added criteria of checking against file.info.header.dicom key-value pairs, the below example shows how to check if the "SeriesDescription" is a non-empty string. The combination of "minLength": 1 and "pattern": "\\S" tells the jsonschema to make sure that "SeriesDescription" contains at least one visible character. For "Columns", the schema simply checks that the value is an integer. And, finally, "ImageType" is checked to make sure that the string "ORIGINAL" is contained somewhere in the array. As with the examples in Use Cases 1 and 2, it is also possible to validate against specific values (const) or patterns (pattern) in file.info.header.dicom key-value pairs.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$defs": {
    "CT_file": {
      "type": "object",
      "properties": {
        "modality": { "const": "CT" },
        "classification": {
          "type": "object",
          "properties": {
            "Contrast": {
              "type": "array",
              "items": { "const": "No Contrast" },
              "minItems": 1,
              "maxItems": 1
            },
            "Scan Orientation": {
              "type": "array",
              "items": { "const": "Axial" },
              "minItems": 1,
              "maxItems": 1
            }
          },
          "required": ["Contrast", "Scan Orientation"]
        },
        "info": {
          "type": "object",
          "properties": {
            "header": {
              "type": "object",
              "properties": {
                "dicom": {
                  "type": "object",
                  "properties": {
                    "SeriesDescription": {
                      "description": "Must be string and have at least one character",
                      "type": "string",
                      "minLength": 1,
                      "pattern": "\\S"
                    },
                    "Columns": {
                      "description": "Must be an integer",
                      "type": "integer"
                      },
                    "ImageType": {
                      "description": "Must contain 'ORIGINAL'",
                      "type": "array",
                      "contains": { "const": "ORIGINAL" }
                    }
                  },
                  "required": ["SeriesDescription", "Columns", "ImageType"]
                }
              },
              "required": ["dicom"]
            }
          },
          "required": ["header"]
        }
      },
      "required": ["modality", "classification", "info"]
    },
    ...
  }
}

Use Case 4: Validate different sessions with different criteria

*Conditions*:

• Validate using session container properties
• Validate using acquisition container properties
• Validate using file level properties

For projects that include multi-session data, it is possible to create a single JSONSchema that includes conditional logic to assess different sessions against different validation criteria.

This example schema shows how to add conditional logic to first check the session label, and then based on the session label call specific rules. So, for this artificial example, we assume that each subject has two sessions, "T1_session" and "T2_session". Since, we are not updating the file or acquisition rules, all of the changes are in the schema validation block. First, we add this: "label": { "type": "string" } to the session properties. Then, inside of the "allOf" block, we first add an "if" statement to check what the session label is; a "then" block is then added to specify which acquisition rule(s) should be used for validating the session.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "description": "Schema to validate acquisition labels and require files with modality and classification in specific sessions",

  "$defs": {
    ...
  },

  "type": "object",
  "properties": {
    "label": { "type": "string" },
    "acquisitions": {
      "type": "array",
      "items": { "type": "object" }
    }
  },

  "allOf": [
    {
      "if": { "properties": { "label": { "const": "T1_session" } } },
      "then": {
        "properties": {
          "acquisitions": {
            "type": "array",
            "contains": { "$ref": "#/$defs/acquisition_with_T1_file" },
            "minContains": 1,
            "maxContains": 1
          }
        },
        "required": ["acquisitions"]
      }
    },
    {
      "if": { "properties": { "label": { "const": "T2_session" } } },
      "then": {
        "properties": {
          "acquisitions": {
            "type": "array",
            "contains": { "$ref": "#/$defs/acquisition_with_T2_file" },
            "minContains": 1,
            "maxContains": 1
          }
        },
        "required": ["acquisitions"]
      }
    }
  ],

  "required": ["label", "acquisitions"]
}

Use Case 5: Validate against session metadata

*Conditions*:

• Validate using session container properties
• Validate using acquisition container properties
• Validate using file level properties

It is also possible to include more general validation criteria for session-level metadata to validate whether the value is of the correct type (e.g., string, number, boolean, etc.). Unlike the previous use cases, where the schema imposes required values and/or patterns of values, with session-level metadata like age and weight, it is more useful to just check that the metadata exists and is of a specific type.

In the below example jsonschema, we extend the first use case to add a requirement in the validation block that in addition to the session having the required acquisitions, the age metadata object is also of type number and has a minimum value of 0. The schema could have also leveraged "maximum" to set a maximum value.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "description": "Schema to validate acquisition labels and session-level metadata",

  "$defs": {
    ...
  },

  "type": "object",
  "properties": {
    "age": {
      "description": "Age of the subject at time of session in years",
      "type": "number",
      "minimum": 0
    },
    "acquisitions": {
      "description": "List of acquisitions in this session",
      "type": "array",
      "items": { "type": "object" },
      "allOf": [
        {
          "description": "Require exactly one acquisition with label 'T1'",
          "contains": { "$ref": "#/$defs/acquisition_with_label_T1" },
          "minContains": 1,
          "maxContains": 1
        },
        {
          "description": "Require exactly one acquisition with label 'T2'",
          "contains": { "$ref": "#/$defs/acquisition_with_label_T2" },
          "minContains": 1,
          "maxContains": 1
        }
      ]
    }
  },

  "required": ["age", "acquisitions"]
}

Interpreting Validation Results

Where to Find Validation Errors

After running the Session Validator gear, you can find validation errors in two places:

1. validation_report.json (optional output file)

If you enable save_validation_report in the gear config, the full error report is saved as validation_report.json in the analysis container.

Structure:

[
  {
    "error_message": "...",
    "instance_path": "...",
    ...
  },
  {
    "error_message": "...",
    "instance_path": "...",
    ...
  }
]

2. Session QC Metadata

Errors are also stored in the session's QC metadata at:

session.qc.session-validator.jsonschema_validation.data

Structure:

qc:
  session-validator:
    jsonschema_validation:
      state: "FAIL"
      data:
        - error_message: "..."
          instance_path: "..."
          ...

Understanding Error Structure

Each validation error has the following fields:

{
  "error_message": "Human-readable description of what failed",
  "instance_path": "Location in your data where error occurred",
  "label": "Human-facing identifier (e.g., acquisition label or file name)",
  "schema_path": "Location in your schema where the rule is defined",
  "validator_type": {
    "type": "Type of validation that failed",
    "details": {
      "validator-specific": "Additional context"
    }
  },
  "validator_value": "The constraint value from your schema",
  "schema_definition": "Optional: $ref definition name if applicable"
}

Tips for Debugging Validation Errors

1. Start with instance_path

This tells you exactly where in your session the problem is:

• root → Session level
• acquisitions.0 → First acquisition in the JSONSchema
• acquisitions.0.files.1 → Second file in first acquisition
• acquisitions.0.files.1.modality → The modality property of that file

2. Check schema_definition for complex errors

When present, this tells you which rule in your $defs section failed:

{
  "schema_definition": "acquisition_with_label_T1"
}

Go to your schema's $defs section and find acquisition_with_label_T1 to see exactly what it requires.

3. Use the label field to find items in Flywheel

The label field provides human-readable identifiers:

• For acquisitions: the acquisition label
• For files: the file name
• For other items: relevant contextual information

This helps you quickly locate the problematic item in the Flywheel UI.

4. Review validator_type.details

This provides validator-specific context that varies by error type:

Validator Type	Details Fields	What They Mean
required	missing	Which fields are missing
type	expected, found	Expected vs actual data type
pattern	pattern, preview	Required regex and actual value
const / enum	expected, found	Expected vs actual value
contains	found, minContains	Items checked vs required
minimum / maximum	found, min/max	Actual value vs constraint

5. Understand the schema_path

This shows where in your JSONSchema the validation rule is defined:

$schema → $defs → T1_file → properties → modality → const

This path tells you: "Go to the $defs section, find T1_file, look in properties, then modality, and you'll see a const constraint."

For concrete examples of validation errors and how to interpret them, see the examples documentation.

Logging

The gear logs at the information level, including general information as it starts and completes steps of the workflow. The gear will also log information on warnings and errors.

FAQ

FAQ.md

Contributing

[For more information about how to get started contributing to that gear, checkout CONTRIBUTING.md.]