Skip to main content

Monitoring a clinically deployed AI device to detect performance drift is an essential step to ensure the safety and effectiveness of AI.

AIM-CU is a statistical tool for AI monitoring based on a cumulative sum (AIM-CU) approach.

AIM-CU computes:

  • The parameter choices for change-point detection based on an acceptable false alarm rate

  • Detection delay estimates for a given displacement of the performance metric from the target for those parameter choices.

System setup

Make sure R is installed in the system. There is no specific version that this relies on. Here we have used the version 4.1.2 (2021-11-01). Instructions for Linux (the below setup is only performed in Linux):

wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc |  tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc
add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/"
apt-get install -y --no-install-recommends r-base r-base-dev

# setup R configs
echo "r <- getOption('repos'); r['CRAN'] <- 'http://cran.us.r-project.org'; options(repos = r);" > ~/.Rprofile
Rscript -e "install.packages('ggplot2')"
Rscript -e "install.packages('hexbin')"
Rscript -e "install.packages('lazyeval')"
Rscript -e "install.packages('cusumcharter')"
Rscript -e "install.packages('RcppCNPy')"
Rscript -e "install.packages('spc')"

Code execution

Clone AIM-CU repository.

git clone https://github.com/DIDSR/AIM-CU.git

Run the following commands to install required dependencies (Python = 3.10 is used).

apt-get -y install python3
apt-get -y install pip
cd AIM-CU
pip install -r requirements.txt

Run AIM-CU.

cd src/package
python3 app.py

Open the URL http://0.0.0.0:7860 that is running the AIM-CU locally.

Example code execution

Example code can be run through Jupyter Notebook. Do this by entering the jupyter notebook command from the /src/package/ directory. The tool is designed to be used through a GUI, not from the console.

Demo

AIM-CU can also be run through the demo available at https://huggingface.co/spaces/didsr/AIM-CU. If Space is paused, click on Restart button. Note: this Space uses a custom Docker container; build may break due to latest package updates pulled by HuggingFace.

Disclaimer

About the Catalog of Regulatory Science Tools

The enclosed tool is in preparation for the Catalog of Regulatory Science Tools. Note that this software is not (yet) a standalone RST but rather is to be included as a reference tool to support other Regulatory Science Tools (such as synthetic datasets) for reproducibility.

This catalog collates a variety of regulatory science tools that the FDA’s Center for Devices and Radiological Health’s (CDRH) Office of Science and Engineering Labs (OSEL) developed. These tools use the most innovative science to support medical device development and patient access to safe and effective medical devices. If you are considering using a tool from this catalog in your marketing submissions, note that these tools have not been qualified as Medical Device Development Tools and the FDA has not evaluated the suitability of these tools within any specific context of use. You may request feedback or meetings for medical device submissions as part of the Q-Submission Program.

For more information about the Catalog of Regulatory Science Tools, email RST_CDRH@fda.hhs.gov.

Metadata

Release files for aim-cu 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aim-cu 0.2.1
File Size Uploaded
aim_cu-0.2.1.tar.gz 16.0 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for aim-cu 0.2.1
File Interpreter ABI Platform
aim_cu-0.2.1-py3-none-any.whl Python 3 none any Details

Total release size: 31.9 kB

Release files / aim_cu-0.2.1.tar.gz

Download URL aim_cu-0.2.1.tar.gz
Size 16.0 kB
Tags Source
SHA-256 checksum
How to use checksums
fd94a63bffca823b9a09ae0c2d04a3a5a6c1970bf3a9a50518e07f288373ae07
BLAKE2b-256 checksum
How to use checksums
34fe27c4ac320d22f3ad9a4e239d717846263f7a760839967f5fc09e9f6df67d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.2

Release files / aim_cu-0.2.1-py3-none-any.whl

Download URL aim_cu-0.2.1-py3-none-any.whl
Size 15.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
02d2450529bc7fa7e070a32773543369e81c4a18da6f940479043d8688875cba
BLAKE2b-256 checksum
How to use checksums
43d82ae7549c8d282972c3aad01e2347e6260c76eef2b3f60b3d4c3af58d1278
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.2

Release history Release notifications | RSS feed

This release

0.2.1 This release

2 release files

0.2.0

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page