Monitoring a clinically deployed AI device to detect performance drift is an essential step to ensure the safety and effectiveness of AI.
AIM-CU is a statistical tool for AI monitoring based on a cumulative sum (AIM-CU) approach.
AIM-CU computes:
The parameter choices for change-point detection based on an acceptable false alarm rate
Detection delay estimates for a given displacement of the performance metric from the target for those parameter choices.
System setup
Make sure R is installed in the system. There is no specific version that this relies on. Here we have used the version 4.1.2 (2021-11-01). Instructions for Linux (the below setup is only performed in Linux):
wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc | tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc
add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/"
apt-get install -y --no-install-recommends r-base r-base-dev
# setup R configs
echo "r <- getOption('repos'); r['CRAN'] <- 'http://cran.us.r-project.org'; options(repos = r);" > ~/.Rprofile
Rscript -e "install.packages('ggplot2')"
Rscript -e "install.packages('hexbin')"
Rscript -e "install.packages('lazyeval')"
Rscript -e "install.packages('cusumcharter')"
Rscript -e "install.packages('RcppCNPy')"
Rscript -e "install.packages('spc')"
Code execution
Clone AIM-CU repository.
git clone https://github.com/DIDSR/AIM-CU.git
Run the following commands to install required dependencies (Python = 3.10 is used).
apt-get -y install python3
apt-get -y install pip
cd AIM-CU
pip install -r requirements.txt
Run AIM-CU.
cd src/package
python3 app.py
Open the URL http://0.0.0.0:7860 that is running the AIM-CU locally.
Example code execution
Example code can be run through Jupyter Notebook. Do this by entering the jupyter notebook command from the /src/package/ directory. The tool is designed to be used through a GUI, not from the console.
Demo
AIM-CU can also be run through the demo available at https://huggingface.co/spaces/didsr/AIM-CU. If Space is paused, click on Restart button. Note: this Space uses a custom Docker container; build may break due to latest package updates pulled by HuggingFace.
Disclaimer
About the Catalog of Regulatory Science Tools
The enclosed tool is in preparation for the Catalog of Regulatory Science Tools. Note that this software is not (yet) a standalone RST but rather is to be included as a reference tool to support other Regulatory Science Tools (such as synthetic datasets) for reproducibility.
This catalog collates a variety of regulatory science tools that the FDA’s Center for Devices and Radiological Health’s (CDRH) Office of Science and Engineering Labs (OSEL) developed. These tools use the most innovative science to support medical device development and patient access to safe and effective medical devices. If you are considering using a tool from this catalog in your marketing submissions, note that these tools have not been qualified as Medical Device Development Tools and the FDA has not evaluated the suitability of these tools within any specific context of use. You may request feedback or meetings for medical device submissions as part of the Q-Submission Program.
For more information about the Catalog of Regulatory Science Tools, email RST_CDRH@fda.hhs.gov.
Metadata
Release files for aim-cu 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| aim_cu-0.2.1.tar.gz | 16.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| aim_cu-0.2.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 31.9 kB
Release files / aim_cu-0.2.1.tar.gz
| Download URL | aim_cu-0.2.1.tar.gz |
|---|---|
| Size | 16.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fd94a63bffca823b9a09ae0c2d04a3a5a6c1970bf3a9a50518e07f288373ae07
|
|
BLAKE2b-256 checksum How to use checksums |
34fe27c4ac320d22f3ad9a4e239d717846263f7a760839967f5fc09e9f6df67d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.2
|
Release files / aim_cu-0.2.1-py3-none-any.whl
| Download URL | aim_cu-0.2.1-py3-none-any.whl |
|---|---|
| Size | 15.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
02d2450529bc7fa7e070a32773543369e81c4a18da6f940479043d8688875cba
|
|
BLAKE2b-256 checksum How to use checksums |
43d82ae7549c8d282972c3aad01e2347e6260c76eef2b3f60b3d4c3af58d1278
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.2
|