File Classifier (file-classifier)

Overview

The File Classifier gear, maintained by Flywheel and part of the Utility suite, classifies files based on 1) their associated metadata and 2) a classification profile containing declarative rules that act on that metadata. The profile is highly customizable and can act on any metadata field available on the input file. The gear ships with default profiles for CT, MR, PT and MG DICOM images and is built on top of the fw-classification toolkit. It is designed to run after the file-metadata-importer gear, typically as a gear rule on file upload.

Summary

Classifies a file by applying a profile of declarative rules against its existing metadata, such as DICOM header fields. Ships with default profiles for CT, MR, PT, and MG modalities, and accepts a custom profile for other file types or classification schemes. Writes classification fields to the file's metadata. Requires metadata to already be populated on the file in advance, for example by file-metadata-importer.

Cite

N/A

License

MIT

Classification

Category: Utility

Gear Level:

• Project
• Subject
• Session
• Acquisition
• Analysis

Inputs

• file-input
  • Name: file-input
  • Type: file
  • Optional: False
  • Classification: DICOM, DICOM ZIP archive or NIfTI (see File Specifications)
  • Description: File to classify.
  • Notes: DICOM files must already have their header extracted by file-metadata-importer. NIfTI files are classified from a matching JSON sidecar.
• profile
  • Name: profile
  • Type: file
  • Optional: True
  • Classification: source code
  • Description: Optional classification profile.
  • Notes: A YAML profile that replaces the default profiles. See Custom Classification Profiles.

Config

• debug
  • Name: debug
  • Type: boolean
  • Description: Log debug messages.
  • Default: false
• remove_existing
  • Name: remove_existing
  • Type: boolean
  • Description: Clear existing classification.
  • Default: false
• tag
  • Name: tag
  • Type: string
  • Description: The tag to be added on input file upon run completion.
  • Default: file-classifier
• validate
  • Name: validate
  • Type: boolean
  • Description: If true the gear will validate the classification schema.
  • Default: false

Outputs

Files

This gear does not produce output files.

Metadata

• File classification: the classification set by the profile is written to the input file's classification (file.classification), in accordance with the classification schema defined on the Flywheel instance. For example, the classification appended might appear similar to:

  {
      "file": {
          "classification": {
              "Intent": "Structural",
              "Measurement": "T1"
          }
      }
  }
  

• Custom metadata: custom classification keys (e.g. file.classification.Custom) and other file or parent container fields can be set by actions defined in the classification profile.

• NIfTI companions: when classifying a NIfTI, the same classification and modality are also applied to its JSON sidecar and to any matching .bval/.bvec files.

• QC: a classification QC result is added to the input file. It is PASS, with the list of profile blocks that evaluated to true, when at least one block matched, and FAIL otherwise.

• Tag: the value of the tag config option is added to the input file.

Pre-requisites

Prerequisite Gear Runs

• file-metadata-importer: DICOM files are classified from the DICOM header metadata that the file-metadata-importer gear stores under file.info.header.dicom. It must be run on DICOM files before this gear.

Prerequisite Files

For NIfTI inputs, a JSON sidecar matching the NIfTI must be present in the same container (see Classifying NIfTI Files from a JSON Sidecar). Otherwise, this gear has no prerequisite files.

Prerequisite Metadata

The file must carry the metadata that the classification profile rules act on (e.g. file.info.header.dicom for the default DICOM profiles). The Flywheel instance must be version 21.6 or later; the gear exits with an error on older instances.

Usage

Description

The gear loads a classification profile, either the default main.yaml profile bundled with the gear, which covers the CT, MR, PT and MG modalities, or the custom profile provided as the profile input. Any contextual classification rules stored in the project custom information are appended to the end of the profile. The profile is then evaluated against the input file's metadata, and the resulting classification and modality are written back to the file.

By default, the provided classification profiles classify medical DICOM images with the following modalities:

• Computed Tomography (CT)
• Magnetic Resonance (MR)
• Positron Emission Tomography (PT)
• Mammography (MG)

Custom classification profiles can be created to classify other types of files.

If remove_existing is enabled, the existing file classification is cleared before classifying. If validate is enabled, the gear checks that the file modality is defined on the instance and that the instance's modality classification schema matches the schema bundled with the gear, and fails otherwise. See how to modify the classification schema.

File Specifications

file-input

Currently the gear supports classification of the following file types:

• DICOM and DICOM ZIP archive, via the file.info.header.dicom namespace which is populated using the file-metadata-importer gear.
• NIfTI, classified from a JSON sidecar that shares the NIfTI's basename and is in the same container as the input. See NIfTI Classification via JSON Sidecar.

profile

A YAML classification profile. Detailed documentation for writing classification profiles can be found in the profile section of the fw-classification library documentation. See Custom Classification Profiles.

NIfTI Classification via JSON Sidecar

DICOM files are classified from the DICOM header metadata that the file-metadata-importer gear stores under file.info.header.dicom. NIfTI files do not carry that metadata, so the gear classifies a NIfTI from a JSON sidecar instead.

When the input file is a NIfTI, the gear locates a JSON sidecar in the same container whose basename matches the NIfTI (for example REST_10m_bold.json for REST_10m_bold.nii.gz) and whose Flywheel file type is source code. It reads the sidecar, converts BIDS timing fields (RepetitionTime, EchoTime, InversionTime, ExposureTime, FrameDuration) from seconds to milliseconds, and uses those values as the input to classification. The resulting classification and modality are written to the NIfTI and mirrored to the sidecar .json and any matching .bval/.bvec files.

Note: The sidecar's contents are used to derive the classification; they are not stored as metadata on the NIfTI. The gear does not import the JSON's fields into the file's metadata. If you need the sidecar's values imported as metadata, use a curation gear such as file-curator with a curation script.

See Classifying NIfTI Files from a JSON Sidecar for the setup requirements and a full walkthrough.

Custom Classification Profiles

Detailed documentation for creating classification profiles can be found in the profile section of the fw-classification library documentation.

Custom classification profiles can be included several ways.

1. Custom Profile File (Recommended)

You can create a custom classification profile and include it as the profile input file when configuring the gear. This completely overrides the default classification profiles, so you may want to include the default profiles, or sections of the profiles in your custom profile if you wish to retain some of the default behavior. You can find the default profiles in the fw-classification-profiles library.

The custom profile can reference the default profiles bundled with the gear using relative paths in the includes section. At runtime, relative includes are resolved against the gear's built-in classification_profiles/ directory, so you do not need to provide absolute paths or git URLs.

Example: Extend all default profiles with a custom rule

This profile loads all default classification profiles (MR, CT, PT, MG) via main.yaml, then adds a custom block that flags DICOM files as Deleted when the ProtocolName header field is set to Deleted.

name: Custom classifier
includes:
  - main.yaml  # Loads all default profiles (MR, CT, PT, MG)

profile:
  - name: set_custom_deleted
    description: |
      Set custom deleted classification if ProtocolName was deleted
    rules:
      - match_type: 'all'
        match:
          - key: file.type
            is: dicom
          - key: file.info.header.dicom.ProtocolName
            is: 'Deleted'
        action:
          - key: file.classification.Custom
            add: 'Deleted'

You can also include only a specific modality profile instead of all defaults by referencing it directly (e.g., MR/main.yaml, CT/main.yaml, PT.yaml, MG.yaml).

Example: Include a profile from a git URL

You can include profiles directly from a git repository using the format git_repo_url$path/to/file or git_repo_url$path/to/file$branch. The profile is cloned locally at runtime. This is useful when you want to reference a specific version of the default profiles or a profile hosted in a different repository.

name: Custom classifier with remote include
includes:
  - https://gitlab.com/flywheel-io/scientific-solutions/lib/fw-classification-profiles$classification_profiles/MR/main.yaml

profile:
  # ... custom blocks go here

2. Contextual Classifications via Project Custom Information

You can also include custom classification blocks by adding them to the project custom information under the classifications key. This is useful when you want to extend the current classification with a small set of rules for a given project. In this case, the rules are appended after the profiles have been selected (either default or by the custom profile method above). This can be done via the UI or the SDK.

Note: This method appends rules to the classification profile after profile selection, which means that they have the highest priority.

Example: Adding a ProtocolName block via the SDK

import flywheel
fw = flywheel.Client()
proj = fw.get_project(<proj_id>) # or use lookup()
existing_info = proj.info

# Initialize context classifications if they don't exist
existing_info.setdefault('classifications', [])
existing_info['classifications'].append(
    {
        'match': [
            {
                'key': 'file.type',
                'is': 'dicom',
            },
            {
                'key': 'file.info.header.dicom.ProtocolName',
                'is': 'deleted',
            }
        ],
        'action': [
            {'key': 'file.classification.Custom', 'add': 'Deleted'},
        ]
    }
)
proj.replace_info(existing_info)

Example: Adding a ProtocolName block via the UI

Navigate to the project "Information" tab for your project, then add the following block to the "Custom Information" section:

Workflow

flowchart LR
    A[Input file] --> B[Load profile:<br/>default or custom]
    B --> C[Append project<br/>context rules]
    C --> D[Evaluate rules<br/>on metadata]
    D --> E[Write classification,<br/>modality and QC]
    E --> F[Tag input file]

1. The gear checks the Flywheel instance version and, if enabled, clears the existing classification and validates the modality classification schema.
2. The classification profile is loaded: the profile input if provided, otherwise the default main.yaml profile.
3. Contextual classification rules from project.info.classifications are appended to the profile.
4. The profile is evaluated against the file metadata (for NIfTI, against the matching JSON sidecar).
5. The classification, modality and any other fields set by the profile are written back to the file (and NIfTI companions), and a classification QC result is recorded.
6. The input file is tagged with the tag config value.

Use Cases

Use Case 1: Synergy with other gears

The file-classifier gear is part of a suite of gears designed to work synergistically to help you curate and manage your files. The gear is designed to run after the file-metadata-importer gear to classify files and augment the metadata extracted by the file-metadata-importer gear.

Use Case 2: Gear Rules

The file-classifier gear is a good candidate to be run as a gear rule automatically on file upload (triggered after the file-metadata-importer gear). This also allows for more advanced workflows to run via gear rules triggered on specific classifications. For example, if you have a rule that classifies a file as a T1-weighted MRI image, you can then trigger a gear rule to automatically run a gear that processes T1-weighted MRI images.

Use Case 3: Classifying NIfTI Files from a JSON Sidecar

Unlike DICOM, a NIfTI file does not store rich acquisition metadata in its header. To classify a NIfTI, the gear reads a JSON sidecar (for example, a BIDS sidecar produced by dcm2niix) that sits alongside the NIfTI in the same container.

Requirements

For the gear to find and use the sidecar, all of the following must be true:

• Same container. The NIfTI and its JSON sidecar must be in the same acquisition (or other container). The gear only looks for the sidecar among the files of the NIfTI's own parent container.
• Matching basename. The sidecar must share the NIfTI's basename, differing only by extension. REST_10m_bold.json pairs with REST_10m_bold.nii.gz (or REST_10m_bold.nii); a differently named sidecar such as REST_bold.json will not pair with REST_10m_bold.nii.gz.
• source code file type. The sidecar must have the Flywheel file type source code, which is the default type assigned to .json files. If the JSON has been reclassified as another type, the gear will not recognize it as a sidecar.
• Run on the NIfTI. Run the gear (or configure the gear rule) with the NIfTI as the input file. The gear finds the sidecar itself; running it on the JSON file does not trigger the sidecar lookup.

Example

Consider an acquisition that contains a NIfTI and its matching sidecar:

• REST_10m_bold.nii.gz, with Flywheel file type nifti
• REST_10m_bold.json, with Flywheel file type source code

When the gear runs on REST_10m_bold.nii.gz, it:

1. Finds REST_10m_bold.json in the same acquisition by matching basename.
2. Reads the sidecar and converts BIDS timing fields (RepetitionTime, EchoTime, InversionTime, ExposureTime, and FrameDuration) from seconds to milliseconds.
3. Classifies the NIfTI from the sidecar values using the active profile and writes the result to the file classification. The same classification and modality are also applied to the sidecar .json and to any matching .bval/.bvec files.

A second pair, such as Coronal_Baseline_T1.nii.gz with Coronal_Baseline_T1.json, works the same way.

Note: The gear classifies the NIfTI from the sidecar; it does not copy the sidecar's fields onto the NIfTI as metadata. See the FAQ for what is and isn't imported.

Warning: If the sidecar is not yet present in the container when the gear runs, the gear waits and retries for a short time (about 30 seconds) before failing with "Could not find NIfTI sidecar." When using a gear rule, make sure the sidecar lands in the same container as the NIfTI.

Logging

This gear uses standard Python logging via the fw-gear library. Set the debug config option to true for verbose debug output. The logs report which blocks of the classification profile evaluated to true, which helps troubleshoot unexpected classifications.

FAQ

Q: How do I create a custom classification profile?

A: Follow the instructions in the profile section of the fw-classification library documentation, and see Custom Classification Profiles.

Q: I ran the classifier gear but my file is not classified. What's wrong?

A: The first thing you should check is the logs of the gear job to see if there were any obvious errors that would cause the gear to fail. If there are no errors, the logs also produce messages which indicate which portions of the classification profile evaluated to true. If the issue is still unclear, check whether the file has the necessary metadata for the profiles to work. If the file does not have the necessary metadata, the classifier gear will not be able to classify the file. Often, this is because either the file-metadata-importer has not been run on the file, or the file has been stripped of metadata in some way (e.g. anonymized/de-identified).

Q: What resources are needed to run the gear?

A: This gear can be run on any instance type and does not require any specialized hardware.

Q: How long does it take to run the gear?

A: This gear usually takes a few seconds (<10s) to run.

Q: How are NIfTI files classified, and is my JSON imported?

A: NIfTI files are classified from a JSON sidecar (for example a BIDS sidecar) that shares the NIfTI's basename, lives in the same container, and has the Flywheel file type source code, for example REST_10m_bold.json next to REST_10m_bold.nii.gz. The gear reads the sidecar to derive the classification and writes the result to the file classification. It does not copy the sidecar's fields onto the NIfTI as metadata. To import the JSON's values as metadata, use a curation gear such as file-curator with a curation script. See Classifying NIfTI Files from a JSON Sidecar for details.

Contributing

For information on how to contribute to this gear, please see CONTRIBUTING.md.