Investigation of methods for analyzing location specific observer performance data

D. P. Chakraborty; Hong-Jun Yoon

doi:10.1117/12.770531

6 March 2008 Investigation of methods for analyzing location specific observer performance data

D. P. Chakraborty, Hong-Jun Yoon

Proceedings Volume 6917, Medical Imaging 2008: Image Perception, Observer Performance, and Technology Assessment; 69170C (2008) https://doi.org/10.1117/12.770531
Event: Medical Imaging, 2008, San Diego, California, United States

Abstract

We examined the statistical powers of three methods for analyzing FROC mark-rating data, namely ROC, JAFROC and IDCA. Two classes of observers were simulated: a designer-level CAD algorithm and a human observer. A search-model based simulator was used with the average numbers of false positives per image ranging from 0.21 for the human observer to 10 for CAD. Model parameters were chosen to yield 80% and 85% areas under the predicted ROC curves for both classes of observers and inter-image and inter-modality correlations of 0.1, 0.5 and 0.9 were investigated. The area under the FROC curve up to abscissa α (ranging from 0.18 to 6.7) was used as the IDCA figure-of-merit; the other methods used their well-known figures of merit. For IDCA power increased with α so it should be chosen as large as possible consistent with the need for overlap of the two FROC curves in the x-direction. For CAD the IDCA method yielded the highest statistical power. Surprisingly, JAFROC yielded the highest statistical power for human observers, even greater than IDCA which, unlike JAFROC, uses all the marks. The largest difference occurred for conservative reporting styles and high data correlation: e.g., 0.3453 for JAFROC vs. 0.2672 for IDCA. One reason is that unlike IDCA, the JAFROC figure of merit is sensitive to unmarked normal images and unmarked lesions. In all cases the ROC method yielded the least statistical power and entailed a substantial statistical power penalty (e.g., 24% for ROC vs. 41% for JAFROC). For human observers JAFROC should be used and for designer-level CAD data IDCA should be used and use of the ROC method for localization studies is discouraged.

Citation Download Citation

D. P. Chakraborty and Hong-Jun Yoon "Investigation of methods for analyzing location specific observer performance data", Proc. SPIE 6917, Medical Imaging 2008: Image Perception, Observer Performance, and Technology Assessment, 69170C (6 March 2008); https://doi.org/10.1117/12.770531

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available