Guest editors Claudia Mello-Thoms, Craig K. Abbey, and Elizabeth A. Krupinski conclude the JMI Special Series on 2D and 3D Imaging, with commentary on the contributions.

Purpose: We propose a deep learning-based anthropomorphic model observer (DeepAMO) for image quality evaluation of multi-orientation, multi-slice image sets with respect to a clinically realistic 3D defect detection task.

Approach: The DeepAMO is developed based on a hypothetical model of the decision process of a human reader performing a detection task using a 3D volume. The DeepAMO is comprised of three sequential stages: defect segmentation, defect confirmation (DC), and rating value inference. The input to the DeepAMO is a composite image, typical of that used to view 3D volumes in clinical practice. The output is a rating value designed to reproduce a human observer’s defect detection performance. In stages 2 and 3, we propose: (1) a projection-based DC block that confirms defect presence in two 2D orthogonal orientations and (2) a calibration method that “learns” the mapping from the features of stage 2 to the distribution of observer ratings from the human observer rating data (thus modeling inter- or intraobserver variability) using a mixture density network. We implemented and evaluated the DeepAMO in the context of  Tc99m-DMSA SPECT imaging. A human observer study was conducted, with two medical imaging physics graduate students serving as observers. A 5  ×  2-fold cross-validation experiment was conducted to test the statistical equivalence in defect detection performance between the DeepAMO and the human observer. We also compared the performance of the DeepAMO to an unoptimized implementation of a scanning linear discriminant observer (SLDO).

Results: The results show that the DeepAMO’s and human observer’s performances on unseen images were statistically equivalent with a margin of difference (ΔAUC) of 0.0426 at p  <  0.05, using 288 training images. A limited implementation of an SLDO had a substantially higher AUC (0.99) compared to the DeepAMO and human observer.

Conclusion: The results show that the DeepAMO has the potential to reproduce the absolute performance, and not just the relative ranking of human observers on a clinically realistic defect detection task, and that building conceptual components of the human reading process into deep learning-based models can allow training of these models in settings where limited training images are available.

Purpose: To assess the comparative diagnostic performance of digital periapical (PA) radiography and cone-beam computed tomography (CBCT) imaging on endodontic disease detection and to provide study methodology and design recommendations for future studies comparing the diagnostic performance of imaging modalities on endodontic disease detection.

Approach: A search of the Medline, Embase, Scopus, Web of Science, and the Cochrane Central Register of Controlled Trials databases was conducted. Studies that compared the performance of CBCT to digital PA radiography for detecting endodontic disease had an independent reference standard determining the presence of endodontic disease and conducted data analysis including either sensitivity, specificity, receiver operating characteristic (ROC) analysis or free response operating characteristic analysis were included. Of the 20,530 identified studies, only 3 fulfilled the inclusion criteria.

Results: Most studies assessed for eligibility were excluded due to limitations and biases in study design—15 of 18 studies had no reference standard. Only one retrospective clinical study reported on the diagnostic performance of CBCT and showed a sensitivity of 86% and specificity of 26%. Two cadaver studies reported sensitivity ranging from 60% to 100%, specificity ranging from 79% to 100%, and an area under the ROC curve of 0.943 for CBCT. The reported sensitivity for digital PA radiography ranged from 27% to 60%, specificity was 99%, and the area under the ROC curve was 0.629.

Conclusions: There is a lack of quality evidence and insufficient data to compare diagnostic performance of digital PA and CBCT imaging. This emphasizes the need for well-designed studies to inform clinicians about the relative diagnostic performance of these imaging modalities.

Purpose: Three-dimensional “volumetric” imaging methods are now a common component of medical imaging across many imaging modalities. Relatively little is known about how human observers localize targets masked by noise and clutter as they scroll through a 3D image and how it compares to a similar task confined to a single 2D slice.

Approach: Gaussian random textures were used to represent noisy volumetric medical images. Subjects were able to freely inspect the images, including scrolling through 3D images as part of their search process. A total of eight experimental conditions were evaluated (2D versus 3D images, large versus small targets, power-law versus white noise). We analyze performance in these experiments using task efficiency and the classification image technique.

Results: In 3D tasks, median response times were roughly nine times longer than 2D, with larger relative differences for incorrect trials. The efficiency data show a dissociation in which subjects perform with higher statistical efficiency in 2D tasks for large targets and higher efficiency in 3D tasks with small targets. The classification images suggest that a critical mechanism behind this dissociation is an inability to integrate across multiple slices to form a 3D localization response. The central slices of 3D classification images are remarkably similar to the corresponding 2D classification images.

Conclusions: 2D and 3D tasks show similar weighting patterns between 2D images and the central slice of 3D images. There is relatively little weighting across slices in the 3D tasks, leading to lower task efficiency with respect to the ideal observer.

Purpose: Automation of organ segmentation, via convolutional neural networks (CNNs), is key to facilitate the work of medical practitioners by ensuring that the adequate radiation dose is delivered to the target area while avoiding harmful exposure of healthy organs. The issue with CNNs is that they require large amounts of data transfer and storage which makes the use of image compression a necessity. Compression will affect image quality which in turn affects the segmentation process. We address the dilemma involved with handling large amounts of data while preserving segmentation accuracy.

Approach: We analyze and improve 2D and 3D U-Net robustness against JPEG 2000 compression for male pelvic organ segmentation. We conduct three experiments on 56 cone beam computed tomography (CT) and 74 CT scans targeting bladder and rectum segmentation. The two objectives of the experiments are to compare the compression robustness of 2D versus 3D U-Net and to improve the 3D U-Net compression tolerance via fine-tuning.

Results: We show that a 3D U-Net is 50% more robust to compression than a 2D U-Net. Moreover, by fine-tuning the 3D U-Net, we can double its compression tolerance compared to a 2D U-Net. Furthermore, we determine that fine-tuning the network to a compression ratio of 64:1 will ensure its flexibility to be used at compression ratios equal or lower.

Conclusions: We reduce the potential risk involved with using image compression on automated organ segmentation. We demonstrate that a 3D U-Net can be fine-tuned to handle high compression ratios while preserving segmentation accuracy.

Purpose: Experienced radiologists have enhanced global processing ability relative to novices, allowing experts to rapidly detect medical abnormalities without performing an exhaustive search. However, evidence for global processing models is primarily limited to two-dimensional image interpretation, and it is unclear whether these findings generalize to volumetric images, which are widely used in clinical practice. We examined whether radiologists searching volumetric images use methods consistent with global processing models of expertise. In addition, we investigated whether search strategy (scanning/drilling) differs with experience level.

Approach: Fifty radiologists with a wide range of experience evaluated chest computed-tomography scans for lung nodules while their eye movements and scrolling behaviors were tracked. Multiple linear regressions were used to determine: (1) how search behaviors differed with years of experience and the number of chest CTs evaluated per week and (2) which search behaviors predicted better performance.

Results: Contrary to global processing models based on 2D images, experience was unrelated to measures of global processing (saccadic amplitude, coverage, time to first fixation, search time, and depth passes) in this task. Drilling behavior was associated with better accuracy than scanning behavior when controlling for observer experience. Greater image coverage was a strong predictor of task accuracy.

Conclusions: Global processing ability may play a relatively small role in volumetric image interpretation, where global scene statistics are not available to radiologists in a single glance. Rather, in volumetric images, it may be more important to engage in search strategies that support a more thorough search of the image.

Purpose: A recently proposed model observer mimics the foveated nature of the human visual system by processing the entire image with varying spatial detail, executing eye movements, and scrolling through slices. The model can predict how human search performance changes with signal type and modality (2D versus 3D), yet its implementation is computationally expensive and time-consuming. Here, we evaluate various image quality metrics using extensions of the classic index of detectability expression and assess foveated model observers for search tasks.

Approach: We evaluated foveated extensions of a channelized Hotelling and nonprewhitening matched filter model with an eye filter. The proposed methods involve calculating a model index of detectability (d^′) for each retinal eccentricity and combining these with a weighting function into a single detectability metric. We assessed different versions of the weighting function that varied in the required measurements of the human observers’ search (no measurements, eye movement patterns, size of the image, and median search times).

Results: We show that the index of detectability across eccentricities weighted using the eye movement patterns of observers best predicted human performance in 2D versus 3D search performance for a small microcalcification-like signal and a larger mass-like. The metric with a weighting function based on median search times was the second best predicting human results.

Conclusions: The findings provide a set of model observer tools to evaluate image quality in the early stages of imaging system evaluation or design without implementing the more computationally complex foveated search model.

Purpose: The objective of this study was to investigate the effects of (i) the difference in the beam qualities on the presampled modulation transfer function (MTF) using the edge method and (ii) the readout direction of the edge profile for the photostimulable phosphor (PSP) system.

Approach: The International Electrotechnical Commission (IEC) defined a technique using the “radiation qualities based on a phantom made up of an aluminum added filter” (RQA). A general radiographic system with a tube voltage of 50 kV and 9.7 mm of additional aluminum filtration was used to conform the x-ray to the IEC-specified beam quality definition RQA3. Additionally, we employed two different beams with tube voltages of 60 and 70 kV using a dental x-ray unit. The MTF was measured in the readout direction from low-to-high exposure regions and vice versa with respect to the scanning and subscanning directions of the PSP system.

Results: The difference in the 50%MTF value for all directions between 60 and 70 kV averaged less than 0.05. The 50%MTF of RQA3 was on average 0.2 lower than the value for 60 kV for each direction. For all beam qualities, no difference was observed between the MTFs measured in the readout direction from the low-to-high exposure regions and vice versa.

Conclusions: The MTFs, measured using the dental x-ray unit, were unaffected by the tube voltage, and they were slightly higher than those measured using the RQA3. Furthermore, the MTF was unaffected by the differences in the readout directions of the edge profile.

Purpose: Osteoarthritis (OA) is a common degenerative disease involving a variety of structural changes in the affected joint. In addition to narrowing of the articular space, recent studies involving statistical shape analysis methods have suggested that specific bone shapes might be associated with the disease. We aim to investigate the feasibility of using the recently introduced framework of functional shapes (Fshape) to extract morphological features of OA that combine shape variability of articular surfaces of the tibia (or femur) together with the changes of the joint space.

Approach: Our study uses a dataset of three-dimensional cone-beam CT volumes of 17 knees without OA and 17 knees with OA. Each knee is then represented as an object (Fshape) consisting of a triangulated tibial (or femoral) articular surface and a map of joint space widths (JSWs) measured at the points of this surface (joint space map, JSM). We introduce a generative atlas model to estimate a template (mean) Fshape of the sample population together with template-centered variables that model the transformations from the template to each subject. This approach has two potential advantages compared with other statistical shape modeling methods that have been investigated in knee OA: (i) Fshapes simultaneously consider the variability in bone shape and JSW, and (ii) Fshape atlas estimation is based on a diffeomorphic transformation model of surfaces that does not require a priori landmark correspondences between the subjects. The estimated atlas-to-subject Fshape transformations were used as input to principal component analysis dimensionality reduction combined with a linear support vector machine (SVM) classifier to identify the morphological features of OA.

Results: Using tibial articular surface as the shape component of the Fshape, we found leave-one-out cross validation scores of ≈91.18  %   for the classification based on the bone surface transformations alone, ≈91.18  %   for the classification based on the residual JSM, and ≈85.29  %   for the classification using both Fshape components. Similar results were obtained using femoral articular surfaces. The discriminant directions identified in the statistical analysis were associated with medial narrowing of the joint space, steeper intercondylar eminence, and relative deepening of the medial tibial plateau.

Conclusions: The proposed approach provides an integrated framework for combined statistical analysis of shape and JSPs. It can successfully extract features correlated to OA that appear consistent with previous studies in the field. Although future large-scale study is necessary to confirm the significance of these findings, our results suggest that the functional shape methodology is a promising new tool for morphological analysis of OA and orthopedics data in general.

Purpose: Caused by brain trauma or blood vessel abnormality, intracerebral hemorrhage and secondary ischemia have become prevalent and severe neurological diseases. The timely and accurate detection of disease is essential for the recovery of patients. As an emerging visualization technique, electrical impedance tomography (EIT) offers an alternative. It is able to reconstruct the conductivity distribution that reflects the pathological variation of human tissue.

Approach: In the EIT-based detection, electrodes are usually in uniform arrangement, which may be not suitable in some conditions. To enhance sensitivity in the region of interest, EIT with a novel offset arrangement of boundary electrodes is proposed to image a simulated frontal lobe hemorrhage and secondary ischemia. To cope with the ill-posed inverse problem, the L1 regularization method is developed during the reconstruction. In addition, the impact of noise with a signal-to-noise ratio of 56 dB is studied.

Results: Compared with the traditional uniform electrode arrangement, the results demonstrate that EIT with the proposed offset arrangement of electrodes is more advantageous for imaging frontal lobe disease.

Conclusions: The proposed offset arrangement of electrodes is superior to the traditional uniform arrangement in imaging frontal lobe disease, especially under the impact of noise.

Purpose: Explainable AI aims to build systems that not only give high performance but also are able to provide insights that drive the decision making. However, deriving this explanation is often dependent on fully annotated (class label and local annotation) data, which are not readily available in the medical domain.

Approach: This paper addresses the above-mentioned aspects and presents an innovative approach to classifying a lung nodule in a CT volume as malignant or benign, and generating a morphologically meaningful explanation for the decision in the form of attributes such as nodule margin, sphericity, and spiculation. A deep learning architecture that is trained using a multi-phase training regime is proposed. The nodule class label (benign/malignant) is learned with full supervision and is guided by semantic attributes that are learned in a weakly supervised manner.

Results: Results of an extensive evaluation of the proposed system on the LIDC-IDRI dataset show good performance compared with state-of-the-art, fully supervised methods. The proposed model is able to label nodules (after full supervision) with an accuracy of 89.1% and an area under curve of 0.91 and to provide eight attributes scores as an explanation, which is learned from a much smaller training set. The proposed system’s potential to be integrated with a sub-optimal nodule detection system was also tested, and our system handled 95% of false positive or random regions in the input well by labeling them as benign, which underscores its robustness.

Conclusions: The proposed approach offers a way to address computer-aided diagnosis system design under the constraint of sparse availability of fully annotated images.

Purpose: Radiologists sometimes fail to report clearly visible, clinically significant findings. Eye tracking can provide insight into the causes of such errors.

Approach: We tracked eye movements of 17 radiologists, searching for masses in 80 mammograms (60 with masses).

Results: Errors were classified using the Kundel et al. (1978) taxonomy: search errors (target never fixated), recognition errors (fixated <500  ms), or decision errors (fixated >500  ms). Error proportions replicated Krupinski (1996): search 25%, recognition 25%, and decision 50%. Interestingly, we found few differences between experts and residents in accuracy or eye movement metrics. Error categorization depends on the definition of the useful field of view (UFOV) around fixation. We explored different UFOV definitions, based on targeting saccades and search saccades. Targeting saccades averaged slightly longer than search saccades. Of most interest, we found that the probability that the eyes would move to the target on the next saccade or even on one of the next three saccades was strikingly low (∼33  %  , even when the eyes were <2  deg from the target). This makes it clear that observers do not fully process everything within a UFOV. Using a probabilistic UFOV, we find, unsurprisingly, that observers cover more of the image when no target is present than when it is found. Interestingly, we do not find evidence that observers cover too little of the image on trials when they miss the target.

Conclusions: These results indicate that many errors in mammography reflect failed deployment of attention; not failure to fixate clinically significant locations.

Purpose: Currently, functional magnetic resonance imaging (fMRI) is the most commonly used technique for obtaining dynamic information about the brain. However, because of the complexity of the data, it is often difficult to directly visualize the temporal aspect of the fMRI data.

Approach: We outline a t-distributed stochastic neighbor embedding (t-SNE)-based postprocessing technique that can be used for visualization of temporal information from a 4D fMRI data. Apart from visualization, we also show its utility in detection of major changes in the brain meta-states during the scan duration.

Results: The t-SNE approach is able to detect brain-state changes from task to rest and vice versa for single- and multitask fMRI data. A temporal visualization can also be obtained for task and resting state fMRI data for assessing the temporal dynamics during the scan duration. Additionally, hemodynamic delay can be quantified by comparison of the detected brain-state changes with the experiment paradigm for task fMRI data.

Conclusion: The t-SNE visualization can visualize help identify major brain-state changes from fMRI data. Such visualization can provide information about the degree of involvement and attentiveness of the subject during the scan and can be potentially utilized as a quality control for subject’s performance during the scan.

The erratum corrects an error in Fig. 6 of the originally published article.