Metadata for User facility Template Transformations (MUTTs)

Table of Contents

• Metadata for User facility Template Transformations (MUTTs)
  • Table of Contents
  • Introduction
  • MUTTs User Documentation
    • Prerequisites
    • Installation
    • Updating to the Latest Version
    • Usage
      • Example 1: Generate a JGI Metagenome spreadsheet
      • Example 2: Generate a JGI Metagenome v15 spreadsheet
      • Example 3: Generate an EMSL spreadsheet
      • Command Options
  • MUTTs Developer Documentation
    • Software Requirements
    • Development Installation
    • Running Tests
    • Creating Custom Mapper Files

Introduction

The programs bundled in this repository automatically retrieve Biosample metadata records for studies submitted to NMDC through the NMDC Submission Portal, and convert the metadata into Excel spreadsheets that are accepted by DOE user facilities.

MUTTs User Documentation

The documentation and setup instructions in this section are meant for any user who would like to install the MUTTs Python package and use it's transformation capabilities to convert data from the NMDC Submission Portal into an Excel spreadsheet that follows a template, based on the MUTTs JSON mapper file that is used.

Prerequisites

• Python 3.12 or higher
• An NMDC user account with an API access token

To create an NMDC user account you will need to sign up at the above link by clicking on the 'ORCID LOGIN' button/link at the top right corner of the NMDC site, and signing in appropriately with your ORCID credentials

Setting up your API access token

This is required for running the examples in the Usage section below (after going through all the Installation steps).

Create a .env file in your working directory with the following environment variables:

echo "DATA_PORTAL_REFRESH_TOKEN=your_token_here" > .env
echo "SUBMISSION_PORTAL_BASE_URL=https://data.microbiomedata.org" >> .env

To get your access token:

1. Visit https://data.microbiomedata.org/user
2. Copy your Refresh Token
3. Replace your_token_here in the .env file with your token

Installation

1. Create a virtual environment (recommended)

python -m venv mutts-env
source mutts-env/bin/activate  # On Windows: mutts-env\Scripts\activate

2. Install the MUTTs package from PyPI

pip install mutts

3. Download any of the MUTTs JSON mapper configuration files

Note: It is not mandatory that you need to download/use any of the pre-existing/already defined JSON mapper files that are present in this repository. You can always define your own custom JSON mapper files that follow a format similar to the ones defined in this repo.

Create a directory for your mapper files and download them from this repository:

mkdir input-files
cd input-files

Download the mapper files you need from the input-files directory:

• For EMSL: emsl_header.json
• For JGI Metagenome: jgi_mg_header.json or jgi_mg_header_v15.json
• For JGI Metatranscriptome: jgi_mt_header.json or jgi_mt_header_v15.json

Updating to the Latest Version

To ensure you have the latest features and bug fixes, you can upgrade the MUTTs package from PyPI:

pip install --upgrade mutts

To check your currently installed version:

pip show mutts

You can also install a specific version if needed:

pip install mutts==<version>

Usage

Run the mutts command with the required options:

mutts --help

Note: In the below examples there is a --sample-set argument that requires you to pass it an NMDC Sample Set UUID as value. Each NMDC Submission has one or more Sample Sets associated with it. You can find the Sample Set UUIDs for your Submission by visiting the NMDC Submission Portal and navigating to the Sample Metadata page for a Sample Set within a Submission. The URL will contain the Sample Set UUID.

An example would look like below:

https://data.microbiomedata.org/submission/<submission-uuid>/sample_set/<sample-set-uuid>/samples

Example 1: Generate a JGI Metagenome spreadsheet

mutts --sample-set <sample-set-uuid> \
      --user-facility jgi_mg \
      --mapper input-files/jgi_mg_header.json \
      --output my-samples_jgi.xlsx

Example 2: Generate a JGI Metagenome v15 spreadsheet

mutts --sample-set <sample-set-uuid> \
      --user-facility jgi_mg \
      --mapper input-files/jgi_mg_header_v15.json \
      --output my-samples_jgi_v15.xlsx

Example 3: Generate an EMSL spreadsheet

mutts --sample-set <sample-set-uuid> \
      --user-facility emsl \
      --mapper input-files/emsl_header.json \
      --header \
      --output my-samples_emsl.xlsx

Command Options

• -s, --sample-set: Your NMDC sample set UUID (required)
• -u, --user-facility: Target facility (required): emsl, jgi_mg, jgi_mg_lr, or jgi_mt
• -m, --mapper: Path to the JSON mapper file (required)
• -o, --output: Output Excel file path (required)
• -h, --header: Include headers in output (use for EMSL, omit for JGI)

MUTTs Developer Documentation

The documentation and setup instructions in this section are largely meant for any developer/programmer whose primary use case is to extend/improve/build upon the current capabilities of the MUTTs software.

The software consists of two main components:

1. JSON Mapper Configuration Files

• Controls/specifies the mapping between columns from the NMDC Submission Portal and column names used in the output spreadsheets
• Top-level keys indicate main headers in the output
• Numbered keys add clarifying header information
• The header keyword allows custom column names
• The sub_port_mapping keyword specifies mappings between Submission Portal columns/slots (as dictated by the NMDC submission schema) and user facility template columns
• Examples available in input-files/

2. mutts CLI

• Command-line application that performs the metadata conversion
• Consumes mapper files and submission sample set data as inputs

Software Requirements

• Poetry
• Python 3.12 or higher

Development Installation

1. Clone this repository

git clone https://github.com/microbiomedata/metadata-for-user-facility-template-transformations.git
cd metadata-for-user-facility-template-transformations

2. Install dependencies with Poetry

poetry install

This installs the mutts package in development mode and creates the mutts command-line tool.

3. Set up your .env file

cp .env.example .env  # if available, or create a new .env file

Add your NMDC API token and submission portal base URL:

DATA_PORTAL_REFRESH_TOKEN=your_token_here
SUBMISSION_PORTAL_BASE_URL=https://data.microbiomedata.org

Get your token from: https://data.microbiomedata.org/user

4. Run the CLI in development mode

poetry run mutts --help

Running Tests

Unit Tests

The unit tests are isolated from NMDC services, so they do not require an API token, a .env file, or network access. This is also the test suite run for pull requests.

Run the complete unit test suite:

poetry run pytest

Run a single test file:

poetry run pytest tests/test_dataframe.py

Run a single test function:

poetry run pytest tests/test_dataframe.py::test_merges_environmental_records_by_sample_name

Integration Test

The integration test makes authenticated, read-only requests to the configured nmdc-server instance. It refreshes an access token, reads one stable sample set, and verifies that MUTTs can produce a DataFrame containing an expected sample.

In addition to the URL and token environment variables described above, the integration test requires three additional environment variables. Add these to your .env file to run the integration test locally:

INTEGRATION_TEST_SAMPLE_SET_ID=<test-sample-set-id>
INTEGRATION_TEST_USER_FACILITY=<test-user-facility>
INTEGRATION_TEST_EXPECTED_SAMPLE_NAME=<test-sample-name>

Then run the integration test with:

poetry run pytest -m integration

Pytest excludes integration tests by default. Selecting the integration marker explicitly overrides that default.

The Dev nmdc-server integration tests GitHub Actions workflow runs nightly and can also be started manually. It is configured through repository secrets and variables to talk to the deployed dev nmdc-server instance. The workflow tests both the default-branch (main) source and the latest published MUTTs package:

Failures create or update one GitHub issue. A successful recovery comments on and closes that issue.

Creating Custom Mapper Files

To create a custom mapper for a new user facility, refer to the existing examples:

• emsl_header.json - EMSL configuration
• jgi_mg_header.json - JGI Metagenome configuration
• jgi_mt_header.json - JGI Metatranscriptome configuration
• jgi_mg_header_v15.json - JGI Metagenome v15 configuration
• jgi_mt_header_v15.json - JGI Metatranscriptome v15 configuration