A collection of scripts for sharing/uploading and listing data from XNAT repositories.
Project description
Introduction
The XNAT_CLI_Toolkit is a command-line interface (CLI) tool designed to interact with XNAT (an imaging informatics platform) seamlessly. This toolkit offers a variety of functionalities that allow users to:
List available projects.
Upload data to XNAT, including prearchiving, archiving, and updating demographics.
Share data across different users or systems.
Display a comprehensive list of available commands via the xnat-toolkit command.
The tool is split into different modules, each responsible for specific tasks to ensure a streamlined interaction with XNAT. The command-line interface allows for flexibility and efficiency in handling large datasets and performing administrative tasks on XNAT.
Core Functionalities:
List Projects: Retrieve and display a list of available XNAT projects or subjects from a specific project.
- Upload Data: Upload data to XNAT in a structured manner:
Prearchiving
Archiving
Demographics update
Sharing Data: Share project data with collaborators or other systems.
Help Command: Display a list of available commands with xnat-toolkit.
Dependencies
Python 3.x: Required to run the scripts.
python --version
Checks for the python version installed on your machine.
PIP
py get pip.py
Click:
pip install click
click: For building command-line interfaces.
After installing click there are some more libraries
XNAT:
pip install xnat
After Installation of xnat
- Libraries and Modules:
xnat: A Python library for interacting with XNAT servers. This is essential for connecting to XNAT, handling authentication, and managing the upload, download, and processing of imaging data.
logging: For logging messages related to script execution, such as successes, errors, and debugging information (part of the Python standard library).
json: For working with JSON files, such as reading and writing data (part of the Python standard library).
os: For file system navigation and file handling (part of the Python standard library).
- authenticate: Custom module for managing XNAT credentials and handling expired credentials.
- This module includes:
store_credentials():A funtion to store the credentials, Using xnat-authenticate users can store their credentials(Auto-expire in 1 Hour.)
get_credentials(): A function to retrieve stored XNAT credentials.
CredentialExpiredError: A custom exception for managing credential expiration.
Optional Libraries (in case not pre-installed):
requests: A common HTTP library for making requests to the XNAT server, often required for interacting with APIs.
Installation
pip install requests
Installation Command
You can install all the necessary libraries using the following command:
pip install click xnat requests
Installation
To install the XNAT_CLI_Toolkit, use the following command:
pip install XNAT_CLI_Toolkit
Once installed, you can start using the command-line tool by typing xnat-toolkit in your terminal.
Authentication Module
The authenticate.py module is responsible for securely handling user credentials. It stores credentials for the XNAT server, reuses sessions, and manages credential expiration. Below is a detailed description of how the authentication process works and the options available.
Authentication Process
The authenticate.py module offers functionality for securely storing and retrieving XNAT server credentials. This ensures that you don’t need to re-enter your credentials for every command, making the interaction more efficient.
Command for Storing Credentials
To store your XNAT credentials (host, username, password), use the following command:
xnat-authenticate --host <XNAT_SERVER_URL> --username <USERNAME> --password <PASSWORD>
This command saves your credentials securely to your .netrc file and stores a timestamp for session tracking.
Key Functions
- store_credentials(host, username, password):
Saves the host and user credentials to the .netrc file.
Stores a timestamp for the credentials in a JSON file (~/.last_credentials.json), allowing the system to verify the session validity.
- get_credentials():
Retrieves the last stored credentials from the JSON file.
Checks if the credentials have expired. If the session has expired (after 30 minutes), it prompts the user to re-enter their credentials.
Handling Expired Credentials
When credentials are older than the expiration time (60 minutes), the system will prompt you to enter new ones. If an attempt is made to access XNAT with expired credentials, the following exception will be raised:
CredentialExpiredError: "Credentials have expired. Please enter new ones through xnat-authenticate."
Project Listing Module
The list.py module provides functionality to list the available projects or Subjects from a specific project on the XNAT server. This command uses the click package to create a command-line interface that connects to the server, fetches a list of projects or subjects and displays them. If credentials (server URL, username, and password) are not provided via the command-line, it retrieves them from previously stored credentials using the authenticate.py module.
Command for Listing Projects
The xnat-list command fetches and displays the list of projects or subjects from a specific project stored on the XNAT server. If credentials are not passed in via the command-line options, the stored credentials from .netrc or the last credentials JSON file are used.
Usage
To list all the projects available on your XNAT server, use the following command:
xnat-list --server <XNAT_SERVER_URL> --username <USERNAME> --password <PASSWORD>
If you have already stored your credentials using the xnat-authenticate command, you can omit the server, username, and password, and the tool will automatically fetch the saved credentials:
xnat-list
This will securely retrieve your credentials from the stored .netrc file and connect to the XNAT server.
To list all the subjects from a specific project, use the following command:
xnat-list --server <XNAT_SERVER_URL> --username <USERNAME> --password <PASSWORD> --project-id <PROJECT-ID>
OR, If you have already stored your credentials using the xnat-authenticate command, you can omit the server, username, and password, and the tool will automatically fetch the saved credentials:
xnat-list --project-id <PROJECT-ID>
Key Features
- Command-line options: The xnat-list command supports passing in the server URL, username, and password directly via the command line.
--server or -s: The XNAT server URL (e.g., http://localhost).
--username or -u: The username for the XNAT server.
--password or -p: The password for the XNAT server. If not provided, credentials will be fetched from stored values.
--project-id or -pid: The project id of the project whose subjects you want to be displayed. If not provided, it will give the list of all the available projects on the XNAT Server.
- Logging:
Logs are created in the logs folder in the current working directory.
The log filename is generated using the current date and time, ensuring that each session has its own log file. The log captures important events such as connecting to the server, fetching projects, and errors if any occur during the process.
Error Handling
If the connection to the XNAT server fails, or if there is an issue with the provided credentials, an appropriate error message is displayed. Errors and exceptions are also logged into the log file for future reference.
Example
Here’s an example of listing projects from the XNAT server without passing credentials (assuming they have been previously stored):
$ xnat-list Using credentials from .netrc... server: http://xnat.example.com, Username: myuser Project ID: Project1 Project ID: Project2 ...
In case of a failure in connecting to the server or retrieving projects, the error will be logged, and an error message will be printed to the console:
$ xnat-list Error: Failed to connect to the XNAT server.
Logging Example
Upon execution, logs are generated with timestamps for every action performed, for example:
2024-10-15 13:45:22 - INFO - Using credentials from .netrc. 2024-10-15 13:45:24 - INFO - Fetched projects from the XNAT server. 2024-10-15 13:45:27 - ERROR - Error: Connection refused.
File Structure
xnat-list: Command to list projects from the XNAT server.
Logging: Stored in a log file under the logs directory with a timestamp in the filename.
Error Handling: Includes error messages for failed connections and logs them.
Upload and Archive Module
The upload.py module provides a command-line interface to upload DICOM files to an XNAT project, archive the uploaded files, and update demographic information for the subjects. The module uses the click package for argument parsing and xnat for connecting and interacting with the XNAT server.
Command for Uploading and Archiving Files The xnat-upload command uploads ZIP files containing DICOM data to the XNAT server’s prearchive, archives them into the specified project, and updates subject demographic information.
Usage
To upload and archive DICOM files, use the following command:
xnat-upload --project <XNAT_PROJECT> --server <XNAT_SERVER_URL> --username <USERNAME> --source <SOURCE_DIR>
If you have previously stored your credentials (server, username, and password), you can omit the credentials from the command, and they will be fetched from the stored credentials:
xnat-upload --project <XNAT_PROJECT> --source <SOURCE_DIR>
The <SOURCE_DIR> should contain ZIP files of the DICOM datasets to be uploaded.
Command-line Options
--project or -d: The XNAT project where the files will be archived. This option is required.
--username or -u: Username for XNAT. If not provided, the stored username is used.
--server or -s: The XNAT server URL (e.g., http://localhost). If not provided, the stored server URL is used.
--password or -p: Password for XNAT. If not provided, the stored password is used.
--source or -x: Directory containing the ZIP files to be uploaded. This option is required.
Workflow
- Credential Retrieval:
If username, server, or password is not provided, the get_credentials function from the authenticate.py module retrieves the stored credentials.
- File Upload:
The module traverses the specified source directory and uploads all ZIP files to the XNAT prearchive.
- Archiving Files:
Once the files are uploaded, they are archived into the specified XNAT project, and an experiment label is generated based on subject, study date, study time, and modality.
- Demographic Update:
After archiving, the subject demographic variables such as age, date of birth (DOB), and gender are updated using DICOM tags if available.
Logging
Logs are created in the logs folder in the current working directory. The log file is named based on the current date and time, e.g., 2024-10-15_13-45-22_share.log.
Example log entry:
2024-10-15 13:45:22 - INFO - Connected to XNAT http://xnat.example.com 2024-10-15 13:45:24 - INFO - Uploading /path/to/file.zip 2024-10-15 13:45:27 - ERROR - Error uploading /path/to/file.zip: Connection refused.
Example
Here’s an example of uploading and archiving DICOM files to an XNAT project:
$ xnat-upload --project MyProject --source /path/to/zip/files Using credentials from .netrc... Connected to XNAT http://xnat.example.com Uploading /path/to/file1.zip Uploading /path/to/file2.zip ... Successfully Archived: Subject001_20241015T1330_CT to Project: MyProject Archive completed. Now updating Demographic Variables. Subject: Subject001; Age: 30; Gender: Male
Error Handling
Errors during upload or archiving are logged and displayed. If any files fail to upload, they are listed at the end of the process.
File Structure
upload_and_archive: Command to upload and archive DICOM files to XNAT.
Logging: Stored in a log file under the logs directory.
Error Handling: Logs and prints errors during connection, upload, and archiving.
Pre-Archive Module
Overview
The upload_to_prearchive.py script is designed to facilitate the uploading of DICOM files (in ZIP format) to the prearchive area of a specified XNAT (eXtensible Neuroimaging Archive Toolkit) project. The script connects to the XNAT server using user-provided or stored credentials, scans a given directory for ZIP files, and uploads them to the prearchive.
Usage
To run the script, use the following command in the terminal:
xnat-prearchive --project <project_name> --username <username> --server <server_url> --source <source_directory> --password <password>
Command-Line Options
--project or -d: (Required) The name of the destination XNAT project where the DICOM files will be uploaded.
--username or -u: (Optional) The username for XNAT. If not provided, the script will attempt to fetch stored credentials.
--server or -s: (Optional) The URL of the XNAT server. If not provided, the script will attempt to fetch stored credentials.
--source or -x: (Required) The directory containing the source ZIP files to be uploaded.
--password``or ``-p: (Optional) The password for XNAT. If not provided, the script will attempt to fetch stored credentials.
Workflow
- Credential Management:
The script checks if username, server, and password are provided as command-line arguments.
If any are missing, it attempts to retrieve stored credentials using the get_credentials() function from the authenticate module. If the credentials are expired, it raises a CredentialExpiredError.
- Server Connection:
The script connects to the XNAT server using the provided or retrieved credentials. If the connection fails, an error message is logged.
- File Collection:
The script scans the specified source directory for all ZIP files and compiles a list of their paths. It utilizes os.walk() to traverse subdirectories and ensure that all relevant files are included.
- File Uploading:
- For each ZIP file in the collected list:
The script attempts to upload the file to the prearchive area of the specified XNAT project.
It logs the upload process and tracks any errors that occur during the upload.
- Subject ID Extraction:
The script extracts the subject ID from the uploaded ZIP file’s name (by default, it assumes the subject ID is the filename without the .zip extension). This may be modified if a different extraction method is needed.
- Logging:
The script logs all actions and errors, providing a detailed record of the upload process. Logging is set to the INFO level by default, and logs are formatted to include timestamps and log levels.
- Saving New Subjects:
If any new subjects are added during the upload process, their IDs are saved to a temporary JSON file (new_subjects.json). This file can be utilized for further processing or record-keeping.
- Error Handling:
The script handles various exceptions, logging errors that occur during the connection to XNAT, file retrieval, and uploading processes.
If any uploads fail, the script logs the filenames of the failed uploads.
Example
Here’s an example command to run the script:
xnat-prearchive --project BrainStudy --username johndoe --server https://xnat.example.com --source /path/to/zip/files --password mypassword
This command will upload all ZIP files located in /path/to/zip/files to the BrainStudy project on the specified XNAT server.
Archiving Module
Overview
The archive_to_xnat.py script is designed to archive files from the prearchive area to a specified XNAT (eXtensible Neuroimaging Archive Toolkit) project. It connects to the XNAT server using user-provided or stored credentials, retrieves files from the prearchive, extracts relevant metadata, and uploads the files to the specified project in XNAT.
Usage
To run the script, use the following command in the terminal:
xnat-archive --project <project_name> --username <username> --server <server_url> --password <password>
Command-Line Options
--project or -d: (Required) The name of the destination XNAT project where files will be archived.
--username or -u: (Optional) The username for XNAT. If not provided, the script will attempt to fetch stored credentials.
--server or -s: (Optional) The URL of the XNAT server. If not provided, the script will attempt to fetch stored credentials.
--password or -p: (Optional) The password for XNAT. If not provided, the script will attempt to fetch stored credentials.
Workflow
- Credential Management:
The script checks if username, server, and password are provided as command-line arguments.
If any are missing, it attempts to retrieve stored credentials using the get_credentials() function from the authenticate module. If the credentials are expired, it raises a CredentialExpiredError.
- Server Connection:
The script connects to the XNAT server using the provided or retrieved credentials. If the connection fails, an error message is logged.
- Prearchive File Retrieval:
The script retrieves all sessions from the XNAT prearchive. For each session, it extracts the subject and scans.
- DICOM Metadata Extraction:
- For each scan in the session, the script extracts the following DICOM metadata:
Study Date
Study Time
Modality
This information is used to create a unique experiment label.
- Archiving Process:
The script archives each session to the specified XNAT project using the generated experiment label. If any errors occur during archiving, they are logged.
- Logging:
The script logs all actions and errors, providing a record of the archiving process. Logging is set to the INFO level by default, and logs are formatted to include timestamps and log levels.
Example
Here’s an example command to run the script:
xnat-archive --project BrainStudy --username johndoe --server https://xnat.example.com --password mypassword
This command will archive files from the prearchive area into the BrainStudy project on the specified XNAT server.
Update_Demographics Module
Overview
The update_demographics.py script is designed to update demographic variables (age, date of birth, and gender) for newly added subjects within a specified XNAT (eXtensible Neuroimaging Archive Toolkit) project. The script connects to the XNAT server, retrieves DICOM data for each new subject, and updates the demographic fields in the project accordingly. Usage ~~~~~ To run the script, use the following command in the terminal:
xnat-updatedemographics --project <project_name> --username <username> --server <server_url> --password <password> --new_subjects_file <path_to_new_subjects_file>
Command-Line Options
--project or -d: (Required) The name of the destination XNAT project where the demographic updates will occur.
--username or -u: (Optional) The username for XNAT. If not provided, the script will attempt to fetch stored credentials.
--server or -s: (Optional) The URL of the XNAT server. If not provided, the script will attempt to fetch stored credentials.
--password or -p: (Optional) The password for XNAT. If not provided, the script will attempt to fetch stored credentials.
--new_subjects_file or -n: (Optional) The path to the JSON file containing newly added subjects. Defaults to new_subjects.json.
Workflow
- Credential Management:
The script checks if username, server, and password are provided as command-line arguments.
If any are missing, it attempts to retrieve stored credentials using the get_credentials() function from the authenticate module. If the credentials are expired, it raises a CredentialExpiredError.
- Server Connection:
The script connects to the specified XNAT server using the provided or retrieved credentials. If the connection fails, an error message is logged.
- Loading New Subjects:
The script attempts to load the list of newly added subjects from the specified JSON file. If the file cannot be found or has an invalid format, an error message is displayed.
- Demographic Data Retrieval:
- For each subject in the loaded list:
The script retrieves the subject object from the specified project.
It then iterates through the experiments and scans associated with the subject to extract demographic information from the DICOM data.
- The following DICOM tags are used to obtain demographic variables:
Subject Age: Retrieved from the tag (0010,1010).
Subject Date of Birth: Retrieved from the tag (0010,0030).
Subject Gender: Retrieved from the tag (0010,0040).
- Updating Demographics:
- The script updates the demographic fields for each subject:
Age: If available, the age is converted to an integer (removing the ‘Y’ suffix) or set to 0 if not specified.
Date of Birth: Updated with the retrieved value.
Gender: Mapped to descriptive strings (‘Male’, ‘Female’, ‘Other’) based on the retrieved value.
- Logging:
Throughout the execution, the script logs actions and results, providing a clear record of the updates made to the demographic variables.
Example
Here’s an example command to run the script:
xnat-updatedemographics --project BrainStudy --username johndoe --server https://xnat.example.com --password mypassword --new_subjects_file new_subjects.json
This command updates the demographic variables for the subjects listed in new_subjects.json in the BrainStudy project on the specified XNAT server.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file xnat_cli_toolkit-0.1.1.tar.gz.
File metadata
- Download URL: xnat_cli_toolkit-0.1.1.tar.gz
- Upload date:
- Size: 26.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.12.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a757b933b15b67e55dfc4ab9a208e98ba700db6faaea4ccff9e36f3281c7b63c
|
|
| MD5 |
3f53e43b37e540fa852afd528b7558c1
|
|
| BLAKE2b-256 |
37d51c7dfb5916c18764d3925bc4c930bf458cbceae80402bb9f72a46dbd07ba
|
File details
Details for the file XNAT_CLI_Toolkit-0.1.1-py3-none-any.whl.
File metadata
- Download URL: XNAT_CLI_Toolkit-0.1.1-py3-none-any.whl
- Upload date:
- Size: 23.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.12.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
71b66f5ddc89ef2b0f404bbfb7fcf21aaa633a1424eafdf3a4d41c138dc64ac3
|
|
| MD5 |
f93ed0f0232785011e8ad2b865af7138
|
|
| BLAKE2b-256 |
9a1ffab44ae9d82c1a391ef92ea887778322e2a2bad98d93ee4ba3cfd1041a43
|
