A hybrid CNN-Swin Transformer network as deep learning model observer to predict human observer performance in 2AFC trial

Muhan Shao; Jhimli Mitra; Darrin W. Byrd; Craig K. Abbey; Fatemeh Behnia; Jean H. Lee; Amir Iravani; Murat Sadic; Delphine L. Chen; Paul E. Kinahan; Sangtae Ahn

doi:10.1117/12.3005656

29 March 2024 A hybrid CNN-Swin Transformer network as deep learning model observer to predict human observer performance in 2AFC trial

Muhan Shao, Jhimli Mitra, Darrin W. Byrd, Craig K. Abbey, Fatemeh Behnia, Jean H. Lee, Amir Iravani, Murat Sadic, Delphine L. Chen, Paul E. Kinahan, Sangtae Ahn

Author Affiliations +

Proceedings Volume 12929, Medical Imaging 2024: Image Perception, Observer Performance, and Technology Assessment; 129290B (2024) https://doi.org/10.1117/12.3005656
Event: SPIE Medical Imaging, 2024, San Diego, California, United States

Abstract

Model observers designed to predict human observers in detection tasks are important tools for assessing task-based image quality and optimizing imaging systems, protocol, and reconstruction algorithms. Linear model observers have been widely studied to predict human detection performance, and recently, deep learning model observers (DLMOs) have been developed to improve the prediction accuracy. Most existing DLMOs utilize convolutional neural network (CNN) architectures, which are capable of learning local features while not good at extracting long-distance relations in images. To further improve the performance of CNN-based DLMOs, we investigate a hybrid CNN-Swin Transformer (CNN-SwinT) network as DLMO for PET lesion detection. The hybrid network combines CNN and SwinT encoders, which can capture both local information and global context. We trained the hybrid network on the responses of 8 human observers including 4 radiologists in a two-alternative forced choice (2AFC) experiment with PET images generated by adding simulated lesions to clinical data. We conducted a 9-fold cross-validation experiment to evaluate the proposed hybrid DLMO, compared to conventional linear model observers such as a channelized Hotelling observer (CHO) and a non-prewhitening matched filter (NPWMF). The hybrid CNN-SwinT DLMO predicted human observer responses more accurately than the linear model observers and DLMO with only the CNN encoder. This work demonstrates that the proposed hybrid CNN-SwinT DLMO has the potential as an improved tool for task-based image quality assessment.

Conference Presentation

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Muhan Shao, Jhimli Mitra, Darrin W. Byrd, Craig K. Abbey, Fatemeh Behnia, Jean H. Lee, Amir Iravani, Murat Sadic, Delphine L. Chen, Paul E. Kinahan, and Sangtae Ahn "A hybrid CNN-Swin Transformer network as deep learning model observer to predict human observer performance in 2AFC trial", Proc. SPIE 12929, Medical Imaging 2024: Image Perception, Observer Performance, and Technology Assessment, 129290B (29 March 2024); https://doi.org/10.1117/12.3005656

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available