This PDF file contains the front matter associated with SPIE Proceedings Volume 11314, including the title page, copyright information, table of contents, and author and conference committee lists.

Early detection of breast cancer through screening mammography yields a 20-35% increase in survival rate; however, there are not enough radiologists to serve the growing population of women seeking screening mammography. Although commercial computer aided detection (CADe) software has been available to radiologists for decades, it has failed to improve the interpretation of full-field digital mammography (FFDM) images due to its low sensitivity over the spectrum of findings. In this work, we leverage a large set of FFDM images with loose bounding boxes of mammographically significant findings to train a deep learning detector with extreme sensitivity. Building upon work from the Hourglass architecture, we train a model that produces segmentation- like images with high spatial resolution, with the aim of producing 2D Gaussian blobs centered on ground-truth boxes. We replace the pixel-wise L2 norm with a weak-supervision loss designed to achieve high sensitivity, asymmetrically penalizing false positives and false negatives while softening the noise of the loose bounding boxes by permitting a tolerance in misaligned predictions. The resulting system achieves a sensitivity for malignant findings of 0.99 with only 4.8 false positive markers per image. When utilized in a CADe system, this model could enable a novel workflow where radiologists can focus their attention with trust on only the locations proposed by the model, expediting the interpretation process and bringing attention to potential findings that could otherwise have been missed. Due to its nearly perfect sensitivity, the proposed detector can also be used as a high-performance proposal generator in two-stage detection systems.

Detecting microcalcification clusters in mammograms is important to the diagnosis of breast diseases. Previous studies which mainly focused on supervised methods require abundant annotated training data but these data are usually hard to acquire. In this work, we leverage unsupervised convolutional autoencoders and structural similarity (SSIM) based post-processing to detect and localize microcalcification clusters in full-field digital mammograms (FFDMs). Our models were trained by patches extracted from 3,632 normal cases, in total with 16,702 mammograms. Evaluations were conducted in three aspects, including patch-based anomaly detection, pixel-wise microcalcification localization, and microcalcification cluster detection. Specifically, the receiver operating characteristic (ROC) analysis was used for patch-based anomaly detection. Then, a pixel-wise ROC analysis and a cluster-based free-response ROC (FROC) analysis were performed to assess our detection algorithms of individual microcalcifications and microcalcification clusters, respectively. We achieved a pixel-wise AUC of 0.97 as well as a cluster-based sensitivity of 0.62 at 1 false positive per image and 0.75 at 2.5 false positives per image. Both qualitative and quantitative results demonstrated the effectiveness of our method.

Despite the prominent success of deep learning (DL) in medical imaging for tasks such as computer-aided detection and diagnosis, the field faces a number of challenging problems. An important issue is that of mismatch of data distributions between different data sources, also known as a distribution shift. Distribution shifts may also be present between different subpopulations or subgroups. Distribution shifts that are not easily detectable can prevent the successful deployment of DL models in medical imaging. We use variational inference to create subsets of a given dataset while enforcing artificial distribution shifts between these subsets, thus creating subsets with different characteristics that represent different pseudo “data sources”. By training and testing ROI-based malignant/benign lesion classification models over these pseudo data sources, we evaluate the extent to which distribution shift could deteriorate the performance of popular DL models. We show that distribution shift indeed poses a serious concern for malignant/benign lesion classification in mammography, and we show that the algorithmically created pseudo data sources may not correspond to any recorded clinical or image characteristics. This study shows a potential method for evaluating deep learning algorithms for robustness against distribution shifts. Furthermore, our technique can serve as a benchmark method for development of new models which aim to be robust to distribution shift.

We proposed a two-branch multitask learning convolutional neural network to solve two different but related tasks at the same time. Our main task is to predict occult invasive disease in biopsy proven Ductal Carcinoma in-situ (DCIS), with an auxiliary task of segmenting microcalcifications (MCs). In this study, we collected digital mammography from 604 patients, 400 of which were DCIS. The model used patches with size of 512×512 extracted within a radiologist masked ROIs as input, with outputs including noisy MC segmentations obtained from our previous algorithms, and classification labels from final diagnosis at patients’ definite surgery. We utilized a deep multitask model by combining both Unet segmentation networks and prediction classification networks, by sharing first several convolutional layers. The model achieved a patch-based ROC-AUC of 0.69, with a case-based ROC-AUC of 0.61. Segmentation results achieved a dice coefficient of 0.49.

Mammography-based screening has helped reduce the breast cancer mortality rate, but has also been associated with potential harms due to low specificity, leading to unnecessary exams or procedures, and low sensitivity. Digital breast tomosynthesis (DBT) improves on conventional mammography by increasing both sensitivity and specificity and is becoming common in clinical settings. However, deep learning (DL) models have been developed mainly on conventional 2D full-field digital mammography (FFDM) or scanned film images. Due to a lack of large annotated DBT datasets, it is difficult to train a model on DBT from scratch. In this work, we present methods to generalize a model trained on FFDM images to DBT images. In particular, we use average histogram matching (HM) and DL fine-tuning methods to generalize a FFDM model to the 2D maximum intensity projection (MIP) of DBT images. In the proposed approach, the differences between the FFDM and DBT domains are reduced via HM and then the base model, which was trained on abundant FFDM images, is fine-tuned. When evaluating on image patches extracted around identified findings, we are able to achieve similar areas under the receiver operating characteristic curve (ROC AUC) of ~ 0:9 for FFDM and ~ 0:85 for MIP images, as compared to a ROC AUC of ~ 0:75 when tested directly on MIP images.

We propose a fast few-shot learning framework that uses transfer learning to identify different lung and chest diseases and conditions from chest x-rays. Our model can be trained with as few as five training examples, making it potentially applicable for diagnosis of rare diseases. In this work, we divide different chest diseases into two disjoint categories: (i) base classes (with large training set) and (ii) novel classes (with a few training examples per class). Our method consists of two steps, namely feature extraction and classification. For the feature extraction, we employ a deep convolutional neural network, customized for chest x-rays. We train the feature extractor with data only from base classes. So the novel classes are unseen to the feature extractor during training. However, we use the feature extractor for extracting features from the data of novel classes resulting in transfer learning. Our classifier, on the other hand, uses the data only from the novel classes for training. We introduce the idea of autoencoder ensemble to design the classifier. Only a few feature vectors from each of the novel classes are used for training the classifier making it a few-shot learner. Incorporating new novel classes require training only the classifier which makes the entire process extremely fast. The performance of the classifier is evaluated on the test data from the novel classes. Experiments show _ 18% improvement in the F1 score compared to the baseline on identifying the novel diseases from publicly available chest x-ray dataset.

Weakly supervised disease classification of CT imaging suffers from poor localization owing to case-level annotations, where even a positive scan can hold hundreds to thousands of negative slices along multiple planes. Furthermore, although deep learning segmentation and classification models extract distinctly unique combinations of anatomical features from the same target class(es), they are typically seen as two independent processes in a computer-aided diagnosis (CAD) pipeline, with little to no feature reuse. In this research, we propose a medical classifier that leverages the semantic structural concepts learned via multi-resolution segmentation feature maps, to guide weakly supervised 3D classification of chest CT volumes. Additionally, a comparative analysis is drawn across two different types of feature aggregation to explore the vast possibilities surrounding feature fusion. Using a dataset of 1593 scans labeled on a case-level basis via rule-based model, we train a dual-stage convolutional neural network (CNN) to perform organ segmentation and binary classification of four representative diseases (emphysema, pneumonia/atelectasis, mass and nodules) in lungs. The baseline model, with separate stages for segmentation and classification, results in AUC of 0.791. Using identical hyperparameters, the connected architecture using static and dynamic feature aggregation improves performance to AUC of 0.832 and 0.851, respectively. This study advances the field in two key ways. First, case-level report data is used to weakly supervise a 3D CT classifier of multiple, simultaneous diseases for an organ. Second, segmentation and classification models are connected with two different feature aggregation strategies to enhance the classification performance.

Dual-energy (DE) chest radiography provides the capability of selectively imaging two clinically relevant materials, namely soft tissues, and osseous structures, to better characterize a wide variety of thoracic pathology and potentially improve diagnosis in posteroanterior (PA) chest radiographs. However, DE imaging requires specialized hardware and a higher radiation dose than conventional radiography, and motion artifacts some- times happen due to involuntary patient motion. In this work, we learn the mapping between conventional radiographs and bone suppressed radiographs. Specifically, we propose to utilize two variations of generative adversarial networks (GANs) for image-to-image translation between conventional and bone suppressed radio- graphs obtained by DE imaging technique. We compare the effectiveness of training with patient-wisely paired and unpaired radiographs. Experiments show both training strategies yield radio-realistic" radiographs with suppressed bony structures and few motion artifacts on a hold-out test set. While training with paired images yields slightly better performance than that of unpaired images when measuring with two objective image quality metrics, namely Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR), training with unpaired images demonstrates better generalization ability on unseen anteroposterior (AP) radiographs than paired training.

Low-dose thoracic CT (LDCT) screening has provided a low risk method of obtaining useful clinical information with lower quality images. Coronary artery calcium (CAC), a major indicator of cardiovascular disease, can be visualized on LDCT images. Additionally, the U-Net architecture has shown outstanding performance in a variety of medical imaging tasks, including image segmentation. Thus, the purpose of this study is to analyze the potential of the U-Net in the classification and localization of CAC in LDCT images. This study was performed with 814 LDCT cases with radiologist-determined CAC severity scores. A total of 3 truth masks per image were manually created for training of 3 U-Nets that were used to define the CAC search region, identify CAC candidates, and eliminate false positives (namely, aortic valve calcifications). Additionally, a single network tasked with only CAC candidate identification was tested to assess the need for different sections of the cascade of U-Nets. All CAC segmentation tasks were assessed using ROC analysis in the task of determining whether or not a case contained any CAC. The area under the ROC curve (AUC) as a performance metric and preliminary analysis showed potential for extension to a full classification task. CAC detection through the total cascade of 3 networks achieved and AUC of 0.97 +/- 0.01. Overall, this study shows significant promise in the localization and classification of CAC in LDCT images using a cascade of U-Nets.

Fibrosing idiopathic interstitial pneumonia (IIP) is a subclass of interstitial lung diseases manifesting as progressive worsening of lung function. Such degradation is a continuous and irreversible process which requires quantitative follow-up of patients to assess the pathology occurrence and extent in the lung. The development of automated CAD tools for such purpose is oriented today towards machine learning approaches and in particular convolutional neural networks. The difficulty remains in the choice of the network architecture that best fit to the problem, in straight relationship with available databases for training. We follow-up our work on lung texture analysis and investigate different CNN architectures and training strategies in the context of a limited database, with high class imbalance and subjective and partial annotations. We show that increased performances are achieved using an end-to-end architecture versus patch-based, but also that naive implementation in the former case should be avoided. The proposed solution is able to leverage global information in the scan and shows a high improvement in the F1 scores of the predicted classes and visual results of predictions in better accordance with the radiologist expectations.

Pneumothorax, presenting as a fine line at the edge of the lung and a change in texture outside the lung, is a particularly difficult condition to detect on chest radiographs due to its wide range of sizes and subtle visual signs. Deep learning methods can be applied to chest radiographs to assist in the detection and localization of pneumothorax. The visual signs of pneumothorax are usually unable to be seen at typical neural network input sizes (256 x 256 or 224 x 224); therefore, increasing the resolution of the input images is expected to be beneficial for deep learning detection of pneumothorax. In this work, chest radiographs were separated into two apex images (top third of lung) and then 256 x 256 patches were extracted from the apex images. VGG19 neural networks were fine-tuned for the task of distinguishing between images with and without pneumothorax. One network was fine-tuned with the apex images (downsized to 256 x 256) and another fine-tuned with 256 x 256 patches within the apex images. These fine-tuned networks were tested on an independent test set and ROC analysis performed. The apex-based network yielded an AUC of 0.80 (95% confidence interval (CI): 0.79, 0.81) and the patch-based network yielded an AUC of 0.73 (95% CI: 0.71, 0.74) in the task of distinguishing between images with and without pneumothorax. When the outputs from the two networks were merged via soft voting, a statistically significant increase in performance was observed as compared to either network alone (AUC=0.83, p<0.001).

Brain hemorrhage (BH) is a severe type of stroke resulting in high mortality and morbidity. Detection and diagnosis of BH is commonly performed using neuroimaging tools such as Computed Tomography (CT). We compare and contrast symmetry-aware, symmetry-naive feature representations and their combination for the detection of BH using CT imaging. One of the proposed architectures, e-DeepSymNet, achieves AUC 0.99 [0.97- 1.00] for BH detection. An analysis of the activation values shows that both symmetry-aware and symmetry-naive representations offer complementary information with symmetry-aware representation naive contributing 20% towards the final predictions.

Imbalanced training data introduce important challenge into medical image analysis where a majority of the data belongs to a normal class and only few samples belong to abnormal classes. We propose to mitigate the class imbalance problem by introducing two generative adversarial network (GAN) architectures for class minority oversampling. Here, we explore balancing data distribution 1) by generating new sample from unsupervised GAN or 2) synthesize missing image modalities from semi-supervised GAN. We evaluated the effect of the synthetic unsupervised and semi-supervised GAN methods by use of 1,500 MR images for brain disease diagnosis, where the classification performance of a residual network was compared between unbalanced datasets, classic data augmentation, and the proposed new GAN-based methods.The evaluation results showed that the synthesized minority samples generated by GAN improved classification accuracy up to 18% in term of Dice score.

Retrospective neuro-oncology imaging research relies on standardization of large, heterogeneous sets of clinical images. In particular, many tumor segmentation algorithms require pre- and post- Gadolinium (Gd) T1-weighted MRI scans. Since the presence of contrast agent cannot be reliably inferred from image metadata, we propose an automatic imagebased classifier for this purpose. We proceed with aligning a T1-weighted MR image to a standard atlas space, by selecting one of eight affine transforms produced using different registration parameters and atlases. After resampling to the standard space, we normalize the intensity distribution and compute intensity characteristics inside a pre-built binary mask of likely enhancement. Using a labeled set of 1892 scans, we evaluated logistic regressions with mean, standard deviation, and 95th percentile as possible factors. A univariable logistic regression with standard deviation as factor was most accurate at 98.9% on testing data. The slope coefficient was highly robust with p<1e-6 and Cramer-Rao bound on variance of 1%. The resulting classification script is completely unsupervised. Accuracy on two validation datasets from different sources (totaling 1328 scans) was over 99% on scans with isotropic sampling. Accuracy was lower on highly anisotropic or otherwise lower quality scans. To our knowledge, this is the first attempt to build an automated Gd enhancement classifier for big data applications. We plan to integrate it into XNAT platform for automatic labeling, to enable Gd enhanced image search. The proposed detector performed well on a wide variety of acquisition parameters. Image anisotropy and acquisition artifacts may interfere with accurate detection.

Evaluating size of hyperacute stroke lesions speedily is an essential procedure before physicians make treatment decisions. For a patient with brain stroke suspicion, noncontrast computerized tomography (NCCT) is firstly taken for initial infarction assessment. However, in a lot of cases, because CT hypoattenuation and texture variation caused by hyperacute ischemia are subtle, besides local intensities and texture, physicians usually compare the difference between right and left sides based on the symmetric characteristic of brain anatomy not to miss the subtle lesions. In this paper, we propose a novel 3D U-Net architecture that integrates the comparison knowledge to automatically segment hyperacute stroke lesions on NCCT. To effectively capture right and left comparison features, we introduced a horizontal flip operation into 3D UNet. We also applied gradient-based sensitivity map method to our trained model in order to visualize how much each voxel contributes to segmentation results. Experimental results showed that the proposed architecture improved segmentation accuracy. Dice similarity coefficient (DSC) was improved from 0.44 to 0.54. Sensitivity and specificity was also improved from 0.80 to 1.00 and from 0.90 to 0.98 respectively. Sensitivity maps derived from our trained model demonstrated that both the right and left sides were utilized more effectively to successfully segment ischemic lesions.

One of the most challenging problems encountered in deep learning-based brain tumor segmentation models is the misclassification of tumor tissue classes due to the inherent imbalance in the class representation. Consequently, strong regularization methods are typically considered when training large-scale deep learning models for brain tumor segmentation to overcome undue bias towards representative tissue types. However, these regularization methods tend to be computationally exhaustive, and may not guarantee the learning of features representing all tumor tissue types that exist in the input MRI examples. Recent work in context encoding with deep CNN models have shown promise for semantic segmentation of natural scenes, with particular improvements in small object segmentation due to improved representative feature learning. Accordingly, we propose a novel, efficient 3DCNN based deep learning framework with context encoding for semantic brain tumor segmentation using multimodal magnetic resonance imaging (mMRI). The context encoding module in the proposed model enforces rich, class-dependent feature learning to improve the overall multi-label segmentation performance. We subsequently utilize context augmented features in a machine-learning based survival prediction pipeline to improve the prediction performance. The proposed method is evaluated using the publicly available 2019 Brain Tumor Segmentation (BraTS) and survival prediction challenge dataset. The results show that the proposed method significantly improves the tumor tissue segmentation performance and the overall survival prediction performance.

In this work, we present a memory-efficient fully convolutional network (FCN) incorporated with several memory-optimized techniques to reduce the run-time GPU memory demand during training phase. In medical image segmentation tasks, subvolume cropping has become a common preprocessing. Subvolumes (or small patch volumes) were cropped to reduce GPU memory demand. However, small patch volumes capture less spatial context that leads to lower accuracy. As a pilot study, the purpose of this work is to propose a memory-efficient FCN which enables us to train the model on full size CT image directly without subvolume cropping, while maintaining the segmentation accuracy. We optimize our network from both architecture and implementation. With the development of computing hardware, such as graphics processing unit (GPU) and tensor processing unit (TPU), now deep learning applications is able to train networks with large datasets within acceptable time. Among these applications, semantic segmentation using fully convolutional network (FCN) also has gained a significant improvement against traditional image processing approaches in both computer vision and medical image processing fields. However, unlike general color images used in computer vision tasks, medical images have larger scales than color images such as 3D computed tomography (CT) images, micro CT images, and histopathological images. For training these medical images, the large demand of computing resource become a severe problem. In this paper, we present a memory-efficient FCN to tackle the high GPU memory demand challenge in organ segmentation problem from clinical CT images. The experimental results demonstrated that our GPU memory demand is about 40% of baseline architecture, parameter amount is about 30% of the baseline.

A persistent issue in deep learning (DL) is the inability of models to function in a domain in which they were not trained. For example, a model trained to segment an organ in MRI scans often dramatically fails when tested in the domain of computed tomography (CT) scans. Since manual segmentation is extremely timeconsuming, it is often not feasible to acquire an annotated dataset in the target domain. Domain adaptation allows transfer of knowledge about a labelled source domain into a target domain. In this work, we attempt to address the differences in model performance when segmenting from intravenous contrast (IVC) enhanced or from non-contrast (NC) CT scans. Most of the publicly available, large-scale, annotated CT datasets are IVCenhanced. However, physicians frequently use NC scans in clinical practice. This necessitates methods capable of reliably functioning across both domains. We propose a novel DL framework that can segment the pancreas from non-contrast CT scans through training with the help of IVC-enhanced CT scans. Our method first utilizes a CycleGAN to create synthetic NC (s-NC) variants from IVC scans. Subsequently, we introduce a multilevel 3D UNet architecture to perform pancreas segmentation. The proposed method significantly outperforms the baseline. Experimental results show 6.2% percent improvement compared to the baseline model in terms of the Dice coefficient. To our knowledge, this method is the first of its kind in pancreas segmentation from NC CTs.

Automatic hepatic vessel segmentation from computed tomography (CT) images is essential in computer-assisted liver surgery. However, because of the error-prone and time-consuming manual annotation, it is impractical to obtain the fully correct training labels of hepatic vein (HV) and highly branched portal vein (PV), which largely restricts the development of deep learning methods on hepatic vessel segmentation. To reduce the noise label interference, this paper builds a robust hepatic vessel segmentation model via analyzing the probability distribution relationship between noisy annotation labels and unobserved correct ones, and apply it to deep neural networks (DNNs). Meanwhile, for inferior vena cava (IVC) close to liver and PV in extrahepatic area, segmentation methods are also represented to enhance the completeness of hepatic vessel structure. Experiments, which are conducted on a public hepatic vessel dataset with noise interference, indicates that our model can decrease misclassified regions and increase the vessel recognition probability, simultaneously.

Bi-parametric MRI (bpMRI: T2W MRI and Apparent Diffusion Coefficient maps (ADC) derived from diffusion weighted imaging) is increasingly being used to characterize prostate cancer (PCa). However, inter- and intrareader variability hinders interpretation of MRI. Deep learning networks may aid in PCa characterization and may allow for non-invasively distinguishing clinically significant (csPCa: GGG<1) and insignificant (ciPCa: GGG=1) PCa. Recent studies have shown that signatures from peri-tumoral (PT) region on imaging add significant value to those from intra-tumoral (IT) region for disease detection and characterization. In this work, we present a multi-sequence multi-instance learning convolutional neural network trained using 2D patches extracted from PCa regions of interest (ROIs) on prostate bpMRI to distinguish csPCa and ciPCa. The trained classifier is used to extract pooled features from both the IT and PT ROIs, which are then used to train a random forest classifier to distinguish csPCa and ciPCa. We train and test our models using patient studies from two different institutions (n=298) with GGG obtained either from post-surgical specimens or biopsies. Model built using IT (D_IT) and PT (D_PT) deep features alone resulted in an area under the curve (AUC) of 0.83 and 0.73 respectively, while models computed from IT (R_IT) and PT (R_PT) radiomic features resulted in an AUC of 0.77 and 0.75 respectively. The models D_IP and R_IP trained on combination of IT and PT deep features and radiomic features resulted in an AUC of 0.86 and 0.80 respectively. In both cases, we observe that combining IT and PT features helps in improving the overall classifier performance in distinguishing csPCa and ciPCa.

Colorectal cancer (CRC) remains one of the leading causes of cancer deaths today. Since precancerous colorectal polyps slowly progress into cancer, screening methods are highly effective in reducing the overall mortality rate of CRC by removing them before developing into later stages. The two current screening modalities, optical colonoscopy (OC) and virtual tomographic colonography (CTC), are both effective at detecting polyps, but the diagnostic performance from each has lagged behind detection. In this paper, we propose a texture analysis-based approach for integrating the complementary information from these two screening modalities. We use a set of well-established texture features including gray-level co-occurrence matrix features, gray-level run-length matrix features, local binary pattern features, first order histogram features, and more. To maximize the amount of textures extracted to examine the tissue heterogeneities between polyp pathologies, these textures are also computed on the higher order derivative images of the CTC polyp images and on the Hue/Saturation/Value color-space of the optical polyp images. The dataset used consisted of 165 polyps taken from 113 patients who underwent standard clinical prep prior to the procedures. Patients first had the CTC scan followed by the OC procedure, where the polyps where registered between imaging modalities and were pathologically confirmed for ground truth. Using a random forest classifier with a greedy feature selection algorithm, we find that the combination of using both CTC and OC texture features can improve the diagnostic performance by area under the receiver operating characteristic (AUC) score by upwards of 3%.

The vertebral levels of the spine provide a useful coordinate system when making measurements of plaque, muscle, fat, and bone mineral density. Correctly classifying vertebral levels with high accuracy is challenging due to the similar appearance of each vertebra, the curvature of the spine, and the possibility of anomalies such as fractured vertebrae, implants, lumbarization of the sacrum, and sacralization of L5. The goal of this work is to develop a system that can accurately and robustly identify the L1 level in large heterogeneous datasets. The first approach we study is using a 3D U-Net to segment the L1 vertebra directly using the entire scan volume to provide context. We also tested models for two class segmentation of L1 and T12 and a three class segmentation of L1, T12 and the rib attached to T12. By increasing the number of training examples to 249 scans using pseudo-segmentations from an in-house segmentation tool we were able to achieve 98% accuracy with respect to identifying the L1 vertebra, with an average error of 4.5 mm in the craniocaudal level. We next developed an algorithm which performs iterative instance segmentation and classification of the entire spine with a 3D U-Net. We found the instance based approach was able to yield better segmentations of nearly the entire spine, but had lower classification accuracy for L1.

Osteoarthritis (OA) is a chronic degenerative disorder of joints and is the most common reason leading to total knee joint replacement (TKR). In this paper, we implemented a semi-supervised learning approach based on Unsupervised Data Augmentation (UDA) along with valid perturbations for radiographs to enhance the performance of supervised TKR outcome prediction model. Our results suggest that the use of semi-supervised approach provides superior results compared to the supervised approach (AUC of 0.79 ± 0.04 vs 0.74 ± 0.04).

Despite its superior soft tissue contrast, conventional MRI is qualitative in nature and this presents a bottleneck problem in quantitative image analysis and data-driven medicine. Various investigations have been devoted to overcoming this limitation, but practical solutions remain elusive. Leveraging from the unique ability of emerging deep learning in feature extraction, we investigate a data-driven strategy of separating contributions of various contributing factors intertwined in a single T₁ weighted image to derive quantitative T₁ and ρ maps without any additional image acquisition. Furthermore, in the proposed deep learning framework, compensation for radiofrequency field inhomogeneities is automatically achieved without extra measurement of B₁ map. The tasks are accomplished using self-attention deep convolutional neural networks, which make efficient use of both local and non-local information. The premise of the approach is that qualitative and quantitative MRI, named Q²MRI, can be attained simultaneously without changing the existing imaging protocol. Q²MRI lays foundation for next generation of digital medicine and provides a promising quantitative imaging tool for a wide spectrum of biomedical applications, ranging from disease diagnosis, treatment planning, prognosis to assessment of therapeutic response.

Knee osteoarthritis (OA) is a prevalent and disabling degenerative joint disease. Objectively identifying knee OA severity is challenging given significant inter-reader variability due to human interpretation factors. The Kellgren-Lawrence (KL) grading system is a commonly used scale to quantitatively characterize the severity of knee OA in knee radiographs. It is important to reliably identify severe knee OA since total knee arthroplasty (TKA) can provide significant improvement in patient quality of life for patients with severe knee OA. In this study, we demonstrate a deep learning approach to automatically assessing KL grades. Our approach uses faster R-CNN object detection network to identify the knee region and deep convolutional neural network for classification. We used a dataset of 7962 knee radiographs for each posteroanterior (PA) and lateral (LAT) views, to develop and evaluate our approach. Images with their corresponding KL grades were obtained from the Multicenter Osteoarthritis Study (MOST) dataset. Our network showed multi-class classification accuracy of 69.15 % when the assessment was made based on PA views and accuracy of 56.68 % when LAT views were used. The developed network may play a significant role in surgical decision-making regarding knee replacement surgery.

The confident detection and monitoring of metastatic bone disease remains one of the major unfulfilled needs in oncology. Whole-body MRI offers excellent resolution and sensitivity for the detection of neoplastic cells within the bone marrow using so-called anatomical sequences. In combination with whole-body diffusion-weighted functional sequences, it has shown a great potential in the assessment of patient tumor involvement. However, metastatic bone disease can lead to a large amount of bone lesions spread across the skeleton, making it impractical and labor demanding to manually delineate by a radiologist. Computer-aided detection could alleviate the workflow, enabling automatic, accurate and reproducible study of the patient tumor load. In this paper, we propose a fully automated computer-aided detection system for bone metastases composed of two steps. First, whole-body multi-modal MR image preprocessing is performed consisting of intra- and inter-modality image spatial registration, intensity standardization and atlas-based segmentation of the skeleton. The second stage detects the metastases candidates using random forest voxel classification algorithm. The system is evaluated on the dataset of 6 male advanced prostate cancer patients with metastases to the bone using a leave-one-patient-out cross-validation with manual segmentation of the metastases as the reference standard. The proposed system showed metastases detection sensitivity of 0.74 with a median false positive rate of 9.67. In clinical workflow the system could potentially be used as the initial screening and treatment response assessment tool for whole-body multi-modal MRI of any advanced cancer with metastases to the bone

We developed and evaluated the effect of U-Net-based radiomic features, called U-radiomics, on the prediction of the overall survival of patients with idiopathic pulmonary fibrosis (IPF). To generate the U-radiomics, we retrospectively collected lung CT images of 72 patients with interstitial lung diseases. An experienced observer delineated regions of interest (ROIs) from the lung regions on the CT images, and labeled them into one of four interstitial lung disease patterns (ground-glass opacity, reticulation, consolidation, and honeycombing) or a normal pattern. A U-Net was trained on these images for classifying the ROIs into one of the above five lung tissue patterns. The trained U-Net was applied to the lung CT images of an independent test set of 75 patients with IPF, and a U-radiomics vector for each patient was identified as the average of the bottleneck layer of the U-Net across all the CT images of the patient. The U-radiomics vector was subjected to a Cox proportional hazards model with elastic-net penalty for predicting the survival of the patient. The evaluation was performed by using bootstrapping with 500 replications, where concordance index (C-index) was used as the comparative performance metric. The preliminary results showed the following C-index values for two clinical biomarkers and the U-radiomics: (a) composite physiologic index (CPI): 64.6%, (b) gender, age, and physiology (GAP) index: 65.5%, and (c) U-radiomics: 86.0%. The U-radiomics significantly outperformed the clinical biomarkers in predicting the survival of IPF patients, indicating that the U-radiomics provides a highly accurate prognostic biomarker for patients with IPF.

Studies investigating variations in radiomic features due to changes in image acquisition and reconstruction parameters have focused on non-human phantoms to limit the exposure to human subjects. This study investigated such variations in the computed tomography (CT) scans of a cadaveric human liver. A reference CT scan of a normal cadaveric liver and 16 modified scans were acquired with one parameter changed from the reference for each modified scan: two iterative reconstruction strength levels, one reconstruction kernel, one slice thickness, one coronal view, one sagittal view, one field of view (FOV), two pitch levels, one slice interval, two CTDIvol levels, three kV settings, and one scan acquired using a scanner from another manufacturer. The liver was segmented in each scan, and 142 radiomic features were calculated on each slice across all scans. Unpaired Student’s t-tests assessed significant differences in feature distributions between reference and modified scans for all features after correcting for multiple comparisons (p<0.0004). Assessments were repeated after normalizing pixel value histograms for each scan. Variations in tube voltage, pitch, and slice interval resulted in the smallest number of features reflecting significant differences. Variations in FOV and scan orientation resulted in the largest number of features reflecting significant differences. Fractal features were relatively robust to differences among parameters, while first-order features were sensitive to these differences by comparison. After normalization, first-order features improved agreement between modified and reference scans, while higher-order features were not greatly affected.

Tumor downstaging after neoadjuvant chemoradiation (CRT) in rectal cancer patients is typically assessed via Magnetic Resonance Imaging (MRI) in order to determine follow-up surgical interventions, but is associated with marked inter-reader variability and limited performance. While radiomic features have shown promise for evaluating chemoradiation response and tumor stage in rectal cancers, there is a need to determine how reproducible these features are across different MRI scanners and acquisitions. In this study, we evaluated radiomic feature reproducibility in terms of feature instability within a uniquely curated true healthy" rectum cohort in order to construct a stability-informed radiomic classifier for differentiating poorly from markedly down-staged rectal tumors after chemoradiation in a multi-site setting. We utilized a cohort of 156 patients, with (a) 74 MRIs visualizing the healthy rectum, (b) 52 post-CRT MRI scans in the discovery cohort, and (c) 30 post-CRT MRI scans in a second-site validation cohort; the latter 2 being from rectal cancer patients. 764 radiomic features were extracted from within the entire rectal wall on each MRI scan. Feature instability was used to quantify how reproducible each radiomic feature was between the discovery cohort and the healthy rectum cohort, using locations along the rectum that were spatially distinct from the treated tumor region. From the resulting stability-informed" feature set, the most relevant features were identified to distinguish pathologic tumor stage groups in the discovery cohort via a QDA classifier with cross-validation to ensure robustness. The top 4 radiomic features were then evaluated in hold-out fashion on scans from the validation cohort. We found that utilizing a stability-informed radiomic model (which comprised features that were reproducible in 100% of all comparisons) was significantly more accurate in identifying pathological tumor stage regression in both discovery (AUC=0:66 ± 0:09) and validation (AUC=0.73) cohorts, compared to a basic radiomic model that used all extracted features (AUC=0:60 ± 0:07 in discovery, AUC=0.62 in validation). Evaluating feature instability with respect to healthy rectal tissue may thus enhance the performance of radiomic models in characterizing pathologic downstaging in rectal cancers, via MRI.

Studies have shown that combining calculations of radiomic features with estimates of mammographic density results in an even better assessment of breast cancer risk than density alone. However, to ensure that risk assessment calculations are consistent across different imaging acquisition settings, it is important to identify features that are not overly sensitive to changes in these settings. In this study, digital mammography (DM) images of an anthropomorphic phantom (“Rachel”, Gammex 169, Madison, WI) were acquired at various technique settings. We varied kV and mAs, which control contrast and noise, respectively. DM images in women with negative screening exams were also analyzed. Radiomic features were calculated in the raw (“FOR PROCESSING”) DM images; i.e., grey-level histogram, co-occurrence, run length, fractal dimension, Gabor Wavelet, local binary pattern, Laws, and co-occurrence Laws features. For each feature, the range of variation across technique settings in phantom images was calculated. This range was scaled against the range of variation in the clinical distribution (specifically, the range corresponding to the middle 90% of the distribution). In order for a radiomic feature to be considered robust, this metric of imaging acquisition variation (IAV) should be as small as possible (approaching zero). An IAV threshold of 0.25 was proposed for the purpose of this study. Out of 341 features, 284 features (83%) met the threshold IAV ≤ 0.25. In conclusion, we have developed a method to identify robust radiomic features in DM.

Radiomic features extracted from dynamic contrast-enhanced magnetic resonance (DCE-MR) images of breast lesions can be used in computer-aided diagnosis. However, some features depend upon field strength of acquisition. This motivates investigation of harmonizing the features using a method to address this “batch” effect of field strength of image acquisition. In this study, thirty-two radiomic features were extracted from DCE-MR images of 1,164 lesions (264 benign, 900 cancers) of the breast acquired at 1.5 T or 3.0 T, after segmentation using a fuzzy C-means method. ComBat harmonization was applied in terms of feature categories of morphology, enhancement texture, and most kinetic curve features, due to their potential intrinsic dependence upon field strength of image acquisition. The covariate was status of lesions as benign or malignant. Changes to features were investigated with the Kolmogorov-Smirnov test statistic and the Davies-Bouldin index for degree of clustering using features reduced from 32 to 2 via t-SNE. Classification performance in the task of distinguishing lesions as benign or malignant was evaluated using ten-fold cross-validation and a random forest classifier. The area under the receiver operating characteristic curve (AUC) was used as figure of merit, and classification performance using features in their raw form and in a set using some harmonized features was deemed to be statistically significantly different if p < 0.05. The Kolmogorov-Smirnov test statistic demonstrated that feature value distributions changed the most in features extracted from images acquired at 3.0 T. The Davies-Bouldin index was 6% and 7% respective for benign lesions and cancers, showing that the features became more similar as a result of harmonization. AUC pre- and post-harmonization [95% CI] was 0.84 [0.81, 0.87] and 0.86 [0.83, 0.88] respectively (p = 0.0012). These results suggest that harmonization of radiomic features across field strength of image acquisition may improve the classification performance of computer-aided diagnosis using datasets acquired at different imaging field strengths.

Lung cancer is a fatal disease, non-small cell lung cancer (NSCLC) being the most prevalent type. One of the main purposes of researching NSCLC is identifying patients at high risk for recurrence after surgical resection so that specific and suitable treatments can be found for them. The classification of cancer by anatomic disease extent, that is, by tumor-size (T stage) and nodal-involvement (N stage), is the most widely accepted determinant of appropriate treatment and prognosis among practicing clinicians. However, TN stage-based risk prediction can be inaccurate, as there is moderate observer variability when reporting the size of the lesion. Here, we propose a lung cancer–recurrence prediction model using principal component analysis (PCA) and machine learning (ML) techniques and considering radiomic features and clinical data, including the TN stage. After being filtered by a statistical model, the principal components, including Tand N-stage data and the handcrafted radiomic features from CT images, were applied to various ML models (i.e., random forests, support vector machines, naive Bayesian classifiers, and both boosting). We conducted this study, not only on recurrence, but also recurrence within two years of surgical resection, since more than 80% of recurrence occurs within this time frame. In both cases, the experimental results showed that combining radiomic features and clinical data improves the prediction of lung-cancer recurrence over that of models that only use TN stage data in terms of the 5-fold cross-validation accuracy mean, the receiver operating characteristic (ROC), the area under the ROC curve (AUC), and Kaplan-Meier curves. Finally, this model has been embedded in a website and is being prepared for the Ministry of Food and Drug Safety (MFDS) medical device registration and approval in South Korea.

Deep-learning convolutional neural networks (DCNNs) are the most commonly used approach in medical image analysis tasks at present; however, they have largely been used as black-box predictors, lacking explanation for the underlying reasons. Explainable artificial intelligence (XAI) is an emerging subfield of AI seeking to understand how models make their decisions. In this work, we applied XAI visualization to gain an insight into the features learned by a DCNN trained to classify estrogen receptor status (ER+ vs ER-) based on dynamic contrast-enhanced magnetic resonance imaging (DCEMRI) of the breast. Our data set contained 1395 ER+ regions-of-interest (ROIs) and 729 ER- ROIs from 148 patients, each with a pre-contrast scan and a minimum of two post-contrast scans. We developed a novel transfer-trained dual-domain DCNN architecture derived from the AlexNet model trained on ImageNet data that received the spatial (across the volume) and dynamic (across the acquisition sequence) components of each DCE-MRI ROI as input. The network’s performance was evaluated with the area under the receiver operating characteristic curve (AUC) from leave-one-case-out crossvalidation. To visualize the DCNN learning, we applied XAI techniques, including the Integrated Gradients attribution method and the SmoothGrad noise reduction algorithm, to the ROIs from the training set. We observed that our DCNN learned relevant features from the spatial and dynamic domains, but there were differences in the contributing features from the two domains. We also visualized DCNN learning from irrelevant features resulting from pre-processing artifacts. These observations motivate new approaches to pre-processing our data and training our DCNN.

The purpose of this study was to assess long short-term memory networks in the prediction of recurrence-free survival in breast cancer patients using features extracted from MRIs acquired during the course of neoadjuvant chemotherapy. In the I-SPY1 dataset, up to 4 MRI exams were available per patient acquired at pre-treatment, early-treatment, interregimen, and pre-surgery time points. Breast cancers were automatically segmented and 8 features describing kinetic curve characteristics were extracted. We assessed performance of long short-term memory networks in the prediction of recurrence-free survival status at 2 years and at 5 years post-surgery. For these predictions, we analyzed MRIs from women who had at least 2 (or 5) years of recurrence-free follow-up or experienced recurrence or death within that timeframe: 157 women and 73 women, respectively. One approach used features extracted from all available exams and the other approach used features extracted from only exams prior to the second cycle of neoadjuvant chemotherapy. The areas under the ROC curve in the prediction of recurrence-free survival status at 2 years post-surgery were 0.80, 95% confidence interval [0.68; 0.88] and 0.75 [0.62; 0.83] for networks trained with all 4 available exams and only the ‘early’ exams, respectively. Hazard ratios at the lowest, median, and highest quartile cut -points were 6.29 [2.91; 13.62], 3.27 [1.77; 6.03], 1.65 [0.83; 3.27] and 2.56 [1.20; 5.48], 3.01 [1.61; 5.66], 2.30 [1.14; 4.67]. Long short-term memory networks were able to predict recurrence-free survival in breast cancer patients, also when analyzing only MRIs acquired ‘early on’ during neoadjuvant treatment.

In this study, we aim to develop a multiparametric breast MRI computer-aided diagnosis (CADx) methodology using residual neural network (ResNet) deep transfer learning to incorporate information from both dynamic contrast-enhanced (DCE)-MRI and T2-weighted (T2w) MRI in the task of distinguishing between benign and malignant breast lesions. This retrospective study included 927 unique lesions from 616 women who underwent breast MR exams. A pre-trained ResNet50 was used to extract features from the maximum intensity projection (MIP) images of the second postcontrast subtraction DCE series and the center slice of the T2w series separately. Support vector machine classifiers were trained on the ResNet features to differentiate between benign and malignant lesions. The benefit of pooling features extracted from multiple levels of the network was examined on DCE MIPs. Three multiparametric methods were investigated, where information from the two sequences was integrated at the image level, feature level, or classifier level. Classification performances were evaluated with five-fold cross-validation using the area under the receiver operating characteristic curve (AUC) as the figure of merit. Using pooled features extracted from multiple layers of the ResNet statistically significantly outperformed only using features extracted from the end of the network (P = .002, 95% CI of ▵AUC: [0.007, 0.029]). The multiparametric classifiers using pooled features yielded AUC_ImageFusion=0.85±0.01, AUC_{FeatureFusion}=0.87±0.01, and AUC_{ClassifierFusion}=0.86±0.01, respectively. The feature fusion method statistically significantly outperformed using DCE alone (P = .01, 95% CI of ▵AUC: [0.004, 0.022]), and all three methods statistically significantly outperformed using T2w alone (P < .001).

Breast density, which is the ratio between fibroglandular tissue (FGT) and total breast volume, can be assessed qualitatively by radiologists and quantitatively by computer algorithms. These algorithms often rely on segmentation of breast and FGT volume. In this study, we propose a method to directly assess breast density on MRI, and provide interpretations of these assessments. We assessed breast density in 506 patients with breast cancer using a regression convolutional neural network (CNN). The input for the CNN were slices of breast MRI of 128 × 128 voxels, and the output was a continuous density value between 0 (fatty breast) and 1 (dense breast). We used 350 patients to train the CNN, 75 for validation, and 81 for independent testing. We investigated why the CNN came to its predicted density using Deep SHapley Additive exPlanations (SHAP). The density predicted by the CNN on the testing set was significantly correlated with the ground truth densities (N = 81 patients, Spearman's ρ = 0:86, P < 0:001). When inspecting what the CNN based its predictions on, we found that voxels in FGT commonly had positive SHAP-values, voxels in fatty tissue commonly had negative SHAP-values, and voxels in non-breast tissue commonly had SHAP-values near zero. This means that the prediction of density is based on the structures we expect it to be based on, namely FGT and fatty tissue. To conclude, we presented an interpretable deep learning regression method for breast density estimation on MRI with promising results.

Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is a valuable modality for evaluating breast abnormalities found in mammography and performing early disease detection in high-risk patients. However, images produced by various MRI scanners (e.g., GE Healthcare & Siemens) differ in terms of intensity and other image characteristics such as noise distribution. This is a challenge both for the evaluation of images by radiologists and for the computational analysis of images using radiomics or deep learning. For example, an algorithm trained on a set of images acquired by one MRI scanner may perform poorly on a dataset produced by a different scanner. Therefore, there is an urgent need for image harmonization. Traditional image to image translation algorithms can be used to solve this problem, but they require paired data (i.e. the same object imaged using different scanners). In this study, we utilize a deep learning algorithm that uses unpaired data to solve this problem through a bi-directional translation between MRI images. The proposed method is based on a cycle-consistent adversarial network (CycleGAN) that uses two generator-discriminator pairs. The original CycleGAN struggles in preserving the structure (i.e. breast tissue characteristics and shape) during the translation. To overcome this, we modified the discriminator architecture and forced the penalization based on the structure at the scale of smaller patches. This allows the network to focus more on features pertaining to breast tissue. The results demonstrate that the transformed images are visually realistic, preserve the structure and harmonize intensity across images from different scanners.

The initial assessment of skin lesions is typically based on dermoscopic images. As this is a difficult and time-consuming task, machine learning methods using dermoscopic images have been proposed to assist human experts. Other approaches have studied electrical impedance spectroscopy (EIS) as a basis for clinical decision support systems. Both methods represent different ways of measuring skin lesion properties as dermoscopy relies on visible light and EIS uses electric currents. Thus, the two methods might carry complementary features for lesion classification. Therefore, we propose joint deep learning models considering both EIS and dermoscopy for melanoma detection. For this purpose, we first study machine learning methods for EIS that incorporate domain knowledge and previously used heuristics into the design process. As a result, we propose a recurrent model with state-max-pooling which automatically learns the relevance of different EIS measurements. Second, we combine this new model with different convolutional neural networks that process dermoscopic images. We study ensembling approaches and also propose a cross-attention module guiding information exchange between the EIS and dermoscopy model. In general, combinations of EIS and dermoscopy clearly outperform models that only use either EIS or dermoscopy. We show that our attention-based, combined model outperforms other models with specificities of 34:4% (CI 31:3-38:4), 34:7% (CI 31:0-38:8) and 53:7% (CI 50:1-57:6) for dermoscopy, EIS and the combined model, respectively, at a clinically relevant sensitivity of 98 %.

Rosacea is a common cutaneous disorder characterized by facial redness, swelling, and flushing, and it is usually diagnosed by a dermatologist after a visual examination. Qualitative human assessment often results in relatively high intra- and interobserver variability, which can negatively affect patient outcomes. Computer-assisted image analysis may improve visual assessment by human observers because it enables quantitative, consistent, and accurate analysis. Here, we combine classical multidimensional scaling (MDS) with deep convolutional neural networks (CNNs) to create an efficient framework to identify rosacea lesions. MDS is utilized to determine an appropriate amount of training data, which are used to train Inception-ResNet-v2 to classify facial images into rosacea and non-rosacea regions. Using a leave-one-patient-out cross-validation scheme with 128 × 128 non-overlapping image patches, the method resulted in a class weighted average Dice coefficient (DC) of 82.1% ± 2.4% and accuracy of 85.0% ± 0.6%. While this average performance is almost identical to our previous results (81.7% ± 2.7% and 84.9% ± 0.6% for DC and accuracy, respectively), with the new scheme, we use approximately 90% less data to train the system. We also report the results of quantitative experiments with overlapping patches with a stride of 50 pixels. With the same experimental setup, speedups of 25.6 times (128 × 128), 23.4 times (192 × 192), and 23.2 times (256 × 256) have been observed when the network is trained with the entire training data as the baseline. The class weighted average DC for this experiment with the proposed method is 83.9% ± 2.1% as in the case of 192 × 192 pixels overlapping patches, while it is 84.4% ± 2.2% when the entire set is trained at each fold. We conclude that the proposed method can be an efficient way to train deep neural networks using only a small subset of the training data.

With the renewed interest in developing machine learning methods for medical imaging using deep-learning approaches, it is essential to reexamine data leakage. In this study, we simulated data leakage in the form of feature leakage, where a classifier was trained on the training set, but the feature selection was influenced by the performance on the validation set. A pre-trained deep-learning convolutional neural network (DCNN) without fine-tuning was used as a feature extractor for malignant and benign mass classification in mammography. A feature selection algorithm was trained in the wrapper mode with a cost function tuned to follow the performance metric on the validation set. Linear discriminant analysis (LDA) classifier was trained to classify masses on mammographic patches. Mammograms from 1,882 patient cases with 4,577 unique patches were partitioned by patient into 3,222 for training and 508 for validation, while 847 were sequestered as unseen independent test set to evaluate the generalization error. The effects of the finite sample size on data leakage were studied by varying the training and validation set sizes from 10% to 100% of the available sets. The area under the receiver operating characteristic curve (AUC) was used as the performance metric. The results show that the performance on the validation set could be overestimated, having AUCs of 0.75 to 0.99 for various sample sizes, whereas the independent test performance could realistically only reach an AUC of 0.72. The analysis indicates that deep learning can risk a high inflation in performance and proper housekeeping rules should be followed when designing and developing deep learning methods in medical imaging.

Architectural distortion (AD) is one of the most important potentially ominous signs of breast cancer. As a 3D imaging, digital breast tomosynthesis (DBT) is an accurate tool to detect AD. We developed a deep learning approach for AD detection guided by mammary gland spatial pattern (MGSP) in DBT. The approach consists of two stages: 2D detection and 3D aggregation. In 2D detection, prior MGSP information is obtained first. It includes 1) magnitude image and orientation field map produced from Gabor filters and 2) mammary gland convergence map. Second, Faster-RCNN detection network is employed. Region proposal network extracts features and determines locations of AD candidates and the soft classifier is used for reducing false positives. In 3D aggregation, a region fusion strategy is designed to fuse 2D candidates into 3D candidates. For evaluation, 265 DBT volumes (138 with ADs and 127 without any lesion) were collected from 68 patients. Free response receiver operating characteristic curve was obtained and the mean true positive fraction (MTPF) was used as the figure-of-merit of model performance. Compared with a baseline model based on convergence measure, the six-fold cross validation results showed that our proposed approach achieved MTPF of 0.50 ± 0.04, while the baseline achieved 0.37 ± 0.03. The improvement of our approach was statistically significant (p≪0.001).

We are developing a computerized method to detect mammographically-occult (MO) breast cancers in screening mammograms. The technique exploits asymmetries between mammograms of the left and right breasts. In this study we investigated whether using a conditional generative adversarial network (CGAN) to produce simulated images of the contralateral breast can provide additional information as to breast cancer occurrence (supplementing the left-right mammogram comparison). We trained the CGAN using 1366 normal screening mammograms to simulate the opposite breast, by using the left-right pair as input. After training, we found increased similarity (mean squared error (MSE) and 2D-correlation) between the pair of simulated contralateral and actual (real) mammograms (SR) compared to that of the pair of actual (real left and real right) mammograms (RR). We then applied the CGAN on the independent screening mammogram dataset of 333 women with dense breasts, containing 97 unilateral MO cancer. We computed the similarity measures on the SR and RR pairs. The similarity between the SR pairs of MO cases was smaller than that of controls, while the similarity between the RR pairs of MO cases and controls was similar to each other. We trained a logistic regression classifier using similarity measures as markers for finding MO cancer. Using 10-folder cross-validation, the AUC of the SR+RR classifier was 0.67±0.09 compared to 0.57±0.1 for the RR classifier (p=0.032). We conclude that by comparing a mammogram with simulated images can provide additional information than obtained by comparing pairs of actual mammograms.

In US breast tumor diagnosis, machine learning approaches for the malignancy classification and the mass localization have been attracting many researchers to improve the diagnostic sensitivity and specificity while reducing the image interpretation time. Recently, fully-supervised deep learning methods showed their promising results in those tasks. However, the full supervision for the localization requires human efforts and time to annotate ground truth regions. In this paper, we present a weakly-supervised deep network which can localize breast masses in US images from only diagnostic labels (i.e., malignant and benign). Specifically, we exploit a flexible convolution method, which learns the size and offset of the convolution kernel, in the classification network to detect more relevant regions of breast masses against their various size and shape. Experimental results show that the proposed network outperform conventional CNN models, such as VGG-16 and VGG-16 with dilated convolution. The proposed model achieved 89.03% in the binary classification accuracy. To evaluate the localization performance with weakly-supervised manners, we also compared class activation maps for each instance with manual masks of breast mass in terms of the Dice similarity coefficient and localization recall. The experimental results also demonstrate that the deep network with the adjustable convolution layers can clinically relevant features of breast mass and its surrounding area for both benign and malignant cases.

Artificial intelligence (AI) algorithms, especially deep learning methods have proven to be successful in many medical imaging applications. Computerized breast cancer image analysis can improve diagnosis accuracy. Digital Breast Tomosynthesis (DBT) imaging is a new modality and more advantageous compared to classical digital mammography (DM). Therefore, development of new deep learning algorithms compatible with DBT modality are potent to improve DBT imaging reading time efficiency and increase accuracy for breast cancer diagnosis when used as additional tool for radiologists. In this work, we aimed to build a 3D deep learning model to distinguish malignancy and benign breasts using DBT images. We also investigated effects of different loss functions in our deep learning models. We implemented and evaluated our method on a large data set of 546 patients (205 malignancy and 341 benign). Our results showed that different loss functions lead to an influence on the models performance in our classification tasks, and specific loss function may be selected or customized to adjust a specific performance metric for concrete applications.

The assessment of lymph nodes in CT examinations of cancer patients is essential for cancer staging with direct impact on therapeutic decisions. Automated detection and segmentation of lymph nodes is challenging, especially, due to significant variability in size, shape and location coupled with weak and variable image contrast. In this paper, we propose a joint detection and segmentation approach using a fully convolutional neural network based on 3D foveal patches. To enable network training, 89 publicly available CT data sets were carefully re-annotated yielding an extensive set of 4351 voxel-wise segmentations of thoracic lymph nodes. Based on these annotations, the 3D network was trained to perform per voxel classification. For enlarged potentially malignant lymph nodes, a detection rate of 79% with 8.0 false-positive detections per volume was obtained. A DICE of 0.44 was achieved on average.

Manual detection of lymph nodes by a radiologist is time-consuming, error-prone and suffers from interobserver variability. We propose a mostly generic computer-aided detection system, which can be trained in an end-to-end fashion, to automatically detect axillary lymph nodes using state of the art fully convolutional neural networks. We aim at a system that can be easily transferred to other body regions such as the mediastinal region. Our pipeline is a two-stage approach, where first a volume of interest (VOI) (axillary region) is localized and then axillary lymph node detection is performed within the VOI. The training was done on 58 CT volumes from 36 patients comprising 300 axillary lymph nodes. On our test dataset, consisting of 75 axillary lymph nodes in the size range 5–10 mm and 17 larger than 10 mm from 30 different patients, we achieved a sensitivity of 83% with 6.7 FPs per volume on average.

The suppression of lung vessels in chest computed tomography (CT) scans can enhance the conspicuity of lung nodules, thereby may improve the detection rate of early lung cancer. This study aimed to verify the effect of lung vessel suppression on the performance of the lung nodule detector. Firstly, a lung vessel suppression technique was developed to remove the vessels while preserving the nodules. Then, a lung nodule detector was developed with two stages: nodule candidate generation and false positive reduction. The vessel suppression and nodule detection methods were validated respectively in 50 three-dimensional (3D) chest CT images with manually-labeled vessel trees and 888 3D chest CT images with manually-located nodules (LUNA16). The lung vessel suppression results were quantitatively evaluated by using the Dice coefficient (DICE) and the contrast-to-noise ratio (CNR), and the lung nodule detection results were quantitatively evaluated by using the sensitivity under two conditions: “without” and “with vessel suppression”. The lung vessel suppression accurately removed vessels with a DICE of 0.943 and improved the CNR for nodules from 4.24 (6.27 dB) to 7.02 (8.46 dB), which subsequently improved the average sensitivity from 0.948 to 0.969 under 7 specified false positives for lung nodule detection.

We are developing quantitative image analysis methods for early diagnosis of lung cancer. We designed a hybrid deep learning (H-DL) method for volume segmentation of lung nodules with large variations in size, shape, margin and opacity. In our H-DL method, two UNet++ based DL models, one used a 19-layer VGG network and the other used a 201-layer DenseNet network as backbone, were trained separately and then combined to segment nodules with wide ranges of size, shape, margin, and opacity. A data set collected from LIDC-IDRI containing 430 cases with lung nodules manually segmented by at least two radiologists was split into 352 training and 78 independent test cases. The 50% consensus consolidation of radiologists’ annotation was used as the reference standard for each nodule. For the 78 test cases with 167 nodules, our H-DL model achieved an average 3D DICE coefficient of 0.732±0.158 for all nodules. For the nodules with size larger than 9.5 mm, nodules with margin described by LIDC-IDRI as sharp or spiculated, or nodules with structure described as lobulated or having solid opacity, the segmentation accuracy achieved by our H-DL model were not significantly different from the average of radiologists’ manual annotation in terms of the DICE coefficient. The results demonstrated that our hybrid deep learning scheme could achieve high segmentation accuracy comparable to radiologists’ average segmentations for the wide variety of nodules.

Lung Cancer screening trials have demonstrated significant mortality reduction. Low-Dose Computed Tomography (LDCT) screening can frequently discover many small nodules in at risk participants. However classification of these, sub-cm nodules as cancerous or benign is a challenging task even for expert clinicians. We use machine learning (ML) and deep learning (CNN) techniques to differentiate, sub-cm cancerous and benign nodules. Data for this study is drawn from a screening study (PanCan) from which we selected 612 distinct nodules (140 cancerous, and ~size matched 472 benign). Both methods demonstrated a ~80% accuracy, whereas currently used measures (size) had a 68% accuracy.

We present an interpretable end-to-end computer-aided detection and diagnosis tool for pulmonary nodules on computed tomography (CT) using deep learning-based methods. The proposed network consists of a nodule detector and a nodule malignancy classifier. We used RetinaNet to train a nodule detector using 7,607 slices containing 4,234 nodule annotations and validated it using 2,323 slices containing 1,454 nodule annotations drawn from the LIDC-IDRI dataset. The average precision for the nodule class in the validation set reached 0.24 at an intersection over union (IoU) of 0.5. The trained nodule detector was externally validated using a UCLA dataset. We then used a hierarchical semantic convolutional neural network (HSCNN) to classify whether a nodule was benign or malignant and generate semantic (radiologist-interpretable) features (e.g., mean diameter, consistency, margin), training the model on 149 cases with diagnostic CTs collected from the same UCLA dataset. A total of 149 nodule-centered patches from the UCLA dataset were used to train the HSCNN. Using 5-fold cross validation and data augmentation, the mean AUC and mean accuracy in the validation set for predicting nodule malignancy achieved 0.89 and 0.74, respectively. Meanwhile, the mean accuracy for predicting nodule mean diameter, consistency, and margin were 0.59, 0.74, and 0.75, respectively. We have developed an initial end-to-end pipeline that automatically detects nodules ≥ 5 mm on CT studies and labels identified nodules with radiologist-interpreted features automatically.

The term “Artificial Intelligence” has come into such common usage, that it is now used interchangeably with any software which automates routine tasks. In fact, AI software at its current usage is not intelligent and cannot think out of the box. In this lecture, we will explore the history of software and hardware development, of pattern recognition algorithms and lately deep learning, which are expected to allow healthcare providers to perform their tasks with more reliability, accuracy, and to avoid critical errors, known in medicine as sentinel events. While there is excitement about robots and androids also performing human tasks, as physicians and healthcare providers, we cannot lose the human touch, as that is a critical component in improving the experience and probably outcome of patients when faced with medical illness. Currently deployed AI and healthcare software often fails by distracting providers and decreases human interaction time with patients. The current state of software also often fails in catching mistakes and may even cause mistakes. We will discuss what is needed in order to truly improve the patient’s experience and outcomes, and which tasks would ideally be taken over by a future “AI”. Real life case examples will be used to make the points in this lecture.

Lesion segmentation on computed tomography (CT) scans is an important step for precisely monitoring changes in lesion/tumor growth. This task, however, is very challenging since manual segmentation is prohibitively timeconsuming, expensive, and requires professional knowledge. Current practices rely on an imprecise substitute called response evaluation criteria in solid tumors (RECIST). Although these markers lack detailed information about the lesion regions, they are commonly found in hospitals’ picture archiving and communication systems (PACS). Thus, these markers have the potential to serve as a powerful source of weak-supervision for 2D lesion segmentation. To approach this problem, this paper proposes a convolutional neural network (CNN) based weakly-supervised lesion segmentation method, which first generates the initial lesion masks from the RECIST measurements and then utilizes co-segmentation to leverage lesion similarities and refine the initial masks. In this work, an attention-based co-segmentation model is adopted due to its ability to learn more discriminative features from a pair of images. Experimental results on the NIH DeepLesion dataset demonstrate that the proposed co-segmentation approach significantly improves lesion segmentation performance, e.g the Dice score increases about 4.0% (from 85.8% to 89.8%).

Basic deep learning classifiers used for medical images often produce global labels. While annotation for localized disease detection might be costly, the knowledge of prevalence of conditions in different anatomical areas can help improve the accuracy by limiting the classifier to relevant areas. However, this improvement provided by context knowledge, is usually offset by the errors of the segmentation map used to isolate the area of interest. This paper proposes a framework for disease classification consisting of a segmentation network, a segmentation quality assessment network, and two separate classifiers on whole image and relevant segmented area. The quality assessment network controls the impact of the two disease classifiers on the final outcome, utilizing the masked image only when segmentation is acceptable. We show that in a very large dataset of chest X-ray images, this framework produces a 2% increase in the area under ROC curve for classification compared to a baseline.

Early detection of head and neck tumors is crucial for patient survival. Often, diagnoses are made based on endoscopic examination of the larynx followed by biopsy and histological analysis, leading to a high interobserver variability due to subjective assessment. In this regard, early non-invasive diagnostics independent of the clinician would be a valuable tool. A recent study has shown that hyperspectral imaging (HSI) can be used for non-invasive detection of head and neck tumors, as precancerous or cancerous lesions show specific spectral signatures that distinguish them from healthy tissue. However, HSI data processing is challenging due to high spectral variations, various image interferences, and the high dimensionality of the data. Therefore, performance of automatic HSI analysis has been limited and so far, mostly ex-vivo studies have been presented with deep learning. In this work, we analyze deep learning techniques for in-vivo hyperspectral laryngeal cancer detection. For this purpose we design and evaluate convolutional neural networks (CNNs) with 2D spatial or 3D spatio-spectral convolutions combined with a state-of-the-art Densenet architecture. For evaluation, we use an in-vivo data set with HSI of the oral cavity or oropharynx. Overall, we present multiple deep learning techniques for in-vivo laryngeal cancer detection based on HSI and we show that jointly learning from the spatial and spectral domain improves classification accuracy notably. Our 3D spatio-spectral Densenet achieves an average accuracy of 81%.

Ear diseases are frequently occurring conditions affecting the majority of the pediatric population, potentially resulting in hearing loss and communication disabilities. The current standard of care in diagnosing ear diseases includes a visual examination of the tympanic membrane (TM) by a medical expert with a range of available otoscopes. However, visual examination is subjective and depends on various factors, including the experience of the expert. This work proposes a decision fusion mechanism to combine predictions obtained from digital otoscopy images and biophysical measurements (obtained through tympanometry) for the detection of eardrum abnormalities. Our database consisted of 73 tympanometry records along with digital otoscopy videos. For the tympanometry aspect, we trained a random forest classifier (RF) using raw tympanometry attributes. Additionally, we mimicked a clinician’s decision on tympanometry findings using the normal range of the tympanogram values provided by a clinical guide. Moreover, we re-trained Inception-ResNet-v2 to classify TM images selected from each otoscopic video. After obtaining predictions from each of three different sources, we performed a majority voting-based decision fusion technique to reach the final decision. Experimental results show that the proposed decision fusion method improved the classification accuracy, positive predictive value, and negative predictive value in comparison to the single classifiers. The results revealed that the accuracies are 64.4% for the clinical evaluations of tympanometry, 76.7% for the computerized analysis of tympanometry data, and 74.0% for the TM image analysis while our decision fusion methodology increases the classification accuracy to 84.9%. To the best of our knowledge, this is the first study to fuse the data from digital otoscopy and tympanometry. Preliminary results suggest that fusing information from different sources of sensors may provide complementary information for accurate and computerized diagnosis of TM-related abnormalities.

Type 2 Diabetes (T2D) is a chronic metabolic disorder that can lead to blindness and cardiovascular disease. Information about early stage T2D might be present in retinal fundus images, but to what extent these images can be used for a screening setting is still unknown. In this study, deep neural networks were employed to differentiate between fundus images from individuals with and without T2D. We investigated three methods to achieve high classification performance, measured by the area under the receiver operating curve (ROC-AUC). A multi-target learning approach to simultaneously output retinal biomarkers as well as T2D works best (AUC = 0.746 [±0.001]). Furthermore, the classification performance can be improved when images with high prediction uncertainty are referred to a specialist. We also show that the combination of images of the left and right eye per individual can further improve the classification performance (AUC = 0.758 [±0.003]), using a simple averaging approach. The results are promising, suggesting the feasibility of screening for T2D from retinal fundus images.

The treatment of age-related macular degeneration (AMD) requires continuous eye exams using optical coherence tomography (OCT). The need for treatment is determined by the presence or change of disease-specific OCTbased biomarkers. Therefore, the monitoring frequency has a significant influence on the success of AMD therapy. However, the monitoring frequency of current treatment schemes is not individually adapted to the patient and therefore often insufficient. While a higher monitoring frequency would have a positive effect on the success of treatment, in practice it can only be achieved with a home monitoring solution. One of the key requirements of a home monitoring OCT system is a computer-aided diagnosis to automatically detect and quantify pathological changes using specific OCT-based biomarkers. In this paper, for the first time, retinal scans of a novel self-examination low-cost full-field OCT (SELF-OCT) are segmented using a deep learningbased approach. A convolutional neural network (CNN) is utilized to segment the total retina as well as pigment epithelial detachments (PED). It is shown that the CNN-based approach can segment the retina with high accuracy, whereas the segmentation of the PED proves to be challenging. In addition, a convolutional denoising autoencoder (CDAE) refines the CNN prediction, which has previously learned retinal shape information. It is shown that the CDAE refinement can correct segmentation errors caused by artifacts in the OCT image.

Lack of annotated data is a major challenge to machine learning algorithms, particularly in the field of radiology. Algorithms that can efficiently extract labels in a fast and precise manner are in high demand. Weak supervision is a compromise solution, particularly, when dealing with imaging modalities like Computed Tomography (CT), where the number of slices can reach 1000 per case. Radiology reports store crucial information about clinicians’ findings and observations in CT slices. Automatic generation of labels from CT reports is not a trivial task due to the complexity of sentences and diversity of expression in free-text narration. In this study, we focus on abnormality classification in lungs, liver and kidneys. Firstly, a rule-based model is used to extract weak labels at the case level. Afterwards, attention guided recurrent neural network (RNN) is trained to perform binary classification of radiology reports in terms of whether the organ is normal or abnormal. Additionally, a multi-label RNN with attention mechanism is trained to perform binary classification by aggregating its output for four representative diseases (lungs: emphysema, mass-nodule, effusion and atelectasis-pneumonia; liver: dilatation, fatty infiltration-steatosis, calcification-stone-gallstone, lesion-mass; kidneys: atrophy, cyst, stone-calculi, lesion) into a single abnormal class. Performance has been evaluated using the receiver operating characteristic (ROC) area under the curve (AUC) on 274, 306 and 278 reports for lungs, liver and kidneys correspondingly, manually annotated by radiology experts. The change in performance was evaluated for different sizes of training dataset for lungs. The AUCs of multi-label pretrained models: lungs - 0.929, liver - 0.840, kidney - 0.844; multi-label models: lungs - 0.903, liver - 0.848, kidney - 0.906; binary pretrained models: lungs - 0.922, liver - 0.826, kidneys - 0.928.

Malaria, caused by Plasmodium parasites, continues to be a major burden on global health. Plasmodium falciparum (P. falciparum) and Plasmodium vivax (P. vivax) pose the greatest health threat among the five malaria species. Microscopy examination is considered as the gold standard for malaria diagnosis, but it requires a significant amount of time and expertise. In particular, the automated and accurate detection of P. vivax is difficult due to the low parasitemia levels as compared to P. falciparum. In this work, we develop a rapid and robust diagnosis system for the automated detection of P. vivax parasites using a cascaded YOLO model. This system consists of a YOLOv2 model and a classifier for hardnegative mining. Results from 2567 thin blood smear images of 171 patients show the cascaded YOLO model improves the mean average precision about 8% compared to the conventional YOLOv2 model.

Segmentation of the uterine cavity and placenta in fetal magnetic resonance (MR) imaging is useful for the detection of abnormalities that affect maternal and fetal health. In this study, we used a fully convolutional neural network for 3D segmentation of the uterine cavity and placenta while a minimal operator interaction was incorporated for training and testing the network. The user interaction guided the network to localize the placenta more accurately. We trained the network with 70 training and 10 validation MRI cases and evaluated the algorithm segmentation performance using 20 cases. The average Dice similarity coefficient was 92% and 82% for the uterine cavity and placenta, respectively. The algorithm could estimate the volume of the uterine cavity and placenta with average errors of 2% and 9%, respectively. The results demonstrate that the deep learning-based segmentation and volume estimation is possible and can potentially be useful for clinical applications of human placental imaging.

Computer aided diagnosis (CADx) of polyps has shown great potential to advance the computed tomography colonography (CTC) technique with diagnostic capability. Facing the problem of numerous uncertainties such as polyp size, shape, and orientation in CTC, GLCM-CNN has been proved to be an effective deep learning based tumor classification method, where convolution neural network (CNN) makes decision based on the texture pattern encoded in gray level co-occurrence matrix (GLCM) containing 13 directions. The 13 directional GLCM, by sampling displacement, can be classified into 3 subgroups. Based on our evaluation on the information encoded in the three subgroups, we propose a multi-stage fusion CNN model, which makes the final decision based on two types of features, i.e. (1) a gate module selected group-specific features and (2) fused features learnt from all the features from three groups. On our polyp dataset, which contains 87 polyp masses, our proposed method outperforms both single sub-group based and 13 directional GLCM based CNN model by at least 1.3% in AUC by the average of 20 times 2 fold cross validation experiment results.

Stage T2 is the clinical threshold to administer neoadjuvant chemotherapy for bladder cancer. In this study a deep learning convolutional neural network (DL-CNN) was trained to aid clinicians in staging of bladder cancer in CT Urography (CTU). The DL-CNN utilized two datasets for training and testing. The primary training dataset included 84 bladder cancers from CTU scans of 76 clinically staged patients, 43 cancers were below stage T2, and 41 were stage T2 or above. The second dataset served as an independent test set containing 90 bladder cancers from CTU scans of 86 clinically staged patients, all bladder cancers were staged as T2 or above. Regions of interest (ROIs) were extracted from the lesions as input to the DL-CNN. The model structure and hyper-parameters were determined and asserted using the training dataset of 84 lesions split into two balanced partitions. Based on the lesion-based T2 likelihood score obtained by averaging the scores of all ROIs extracted from a given lesion, the decision threshold providing the highest classification accuracy was determined from the leave-one-out validation. The DL-CNN with the fixed decision threshold was then applied to the test ROIs. The classification accuracy for the independent test set of 90 cancers was 0.91. This performance is slightly higher than our previous radiomics approach based on SVM and BPNN models, which achieved 0.88 and 0.90 accuracy on the same test set, respectively, but the difference was not statistically significant. The results show the promise of using a DL-CNN in bladder cancer stage assessment.

Attention-Deficit/Hyperactivity Disorder (ADHD) is one of the most common childhood neuropsychiatric disorders, with characteristic symptoms of age-inappropriate levels of inattention, hyperactivity, and impulsivity that interfere with social and academic functioning. Recent studies have demonstrated that beyond purely neuroanatomical alterations, the disorder implies altered functional connectivity in several large-scale brain functional networks. In this study, we use the Graph Convolutional Networks (GCNs), which represent the population of patients and control subjects as a sparse graph. In this sparse graph, nodes are human subjects, which are associated with a brain connectivity-based feature vector, and edges are weighted using phenotypic information. We applied this framework on the large publicly available multi-institution ADHD-200 data repository of 921 resting-state functional MRI data sets. Using a 10k-fold cross-validation procedure, we obtain a mean accuracy of 76.95 and mean Area Under the receiver operating Curve (AUC) of 79.66 between typical controls and the ADHD Combined subtype. In addition, we performed classification between typical controls and all subtypes of ADHD patients, where we obtained a mean accuracy of 69.53 and mean AUC of 74.76, which outperforms the state-of-the-art methods in the literature. Our results suggest that resting-state functional MRI analysis with GCNs may provide contributions to developing biomarkers in ADHD and other neurological disorders.

Current techniques trying to predict Alzheimer's disease at an early-stage explore the structural information of T1-weighted MR Images. Among these techniques, deep convolutional neural network (CNN) is the most promising since it has been successfully used in a variety of medical imaging problems. However, the majority of works on Alzheimer's Disease tackle the binary classification problem only, i.e., to distinguish Normal Controls from Alzheimer's Disease patients. Only a few works deal with the multiclass problem, namely, patient classification into one of the three groups: Normal Control (NC), Alzheimer's Disease (AD) or Mild Cognitive Impairment (MCI). In this paper, our primary goal is to tackle the 3-class AD classification problem using T1-weighted MRI and a 2D CNN approach. We used the first two layers of ResNet34 as feature extractor and then trained a classifier using 64 × 64 sized patches from coronal 2D MRI slices. Our extended-2D CNN proposal explores the MRI volumetric information, by using non-consecutive 2D slices as input channels of the CNN, while maintaining the low computational costs associated with a 2D approach. The proposed model, trained and tested on images from ADNI dataset, achieved an accuracy of 68.6% for the multiclass problem, presenting the best performance when compared to state-of-the-art AD classification methods, even the 3D-CNN based ones.

Autism spectrum disorder (ASD) is a collection of neuro-developmental disorders with many symptoms, most prominently social impairment. It is known that there is a significant prevalence (4:1) of ASD in males compared to females. This suggests that there will likely be distinct neuro-anatomical structures across males and females that contribute to the distinct disease etiologies across the two sexes. Hence, in this work, we seek to develop “sex- specific" machine learning models that attempt to capture neuroanatomical differences in brain morphometry across ASD versus normal controls using structural MRI scans. Specifically, we train two different machine- learning models (one for male and one for female cohorts) consisting of hand-crafted" morphometric features such as shape and surface area of brain parcellations from T1w MRI, with deep" features learned from a Dense Convolutional Network (DenseNet). Our methodology consists of first computing morphometric hand-crafted features (i.e. volume and surface features of different brain parcellations) from the training cohort obtained from the ABIDE-II dataset consisting of 210 males and 98 females from structural T1w MRI scans. We then employ feed-forward feature selection within a linear discriminant analysis classifier trained separately for the male and female cohorts, to distinguish ASD from normal controls. Additionally, we train a DenseNet model with 4 dense blocks (with 2 layers each) to extract deep features from T1w MRI scans to distinguish ASD versus controls, separately for males and females. The deep features allow for capturing complementary data-driven feature differences in brain morphometry specific to male and female ASD cohort (versus normal controls). Finally, we combine the top morphometric and DenseNet features obtained from the training model and test them on the ABIDE I dataset for the male (n=85) and female (n=19) cohort separately, to distinguish ASD from normal controls. Our results demonstrated training and testing accuracies of 78% and 79% using hand-crafted features alone, 68% and 57% using DenseNet features alone, and 87% and 79% respectively using integrated hand-crafted and deep features for the male cohort. For the female cohort, we obtained accuracies of 81% and 84% with hand-crafted features alone, 71% and 62% for DenseNet features alone, and 81% and 84% with integrated hand- crafted and deep features, on training and testing sets respectively. With further optimization of deep features along with inclusion of a large multi-site cohort, our presented sex-specific ML approach could allow for improved diagnosis of ASD from controls, across males and females, using structural MRI scans.

Alzheimer’s Disease (AD) which causes declination of cognitive function is one of the most severe social issues in the world. It has already been known that AD cannot be cured and treatment can only delay its progression. Therefore, it is very important to detect AD in early stage and prevent it to be worse. Furthermore, sooner the progression is detected, better the prognosis will be. In this research, we developed a novel multi-modal deep learning method to predict conversion from Mild Cognitive Impairment (MCI), which is the stage between cognitively normal older people and AD. In our method, the multi-modal input data are defined as structural Magnetic Resonance Imaging (MRI) images and clinical data including several cognitive scores, APOE genotype, gender and age obtained from Alzheimer’s Disease Neuroimaging Initiative cohort (ADNI). Our criteria of selecting these input data are that they are mostly obtained by non-invasive examination. The proposed method integrates features obtained from MRI images and clinical data effectively by using bi-linear fusion. Bi-linear fusion computes the products of all elements between image and clinical features, where the correlation between them are included. That led to a big improvement of prediction accuracy in the experiment. The prediction model using bi-linear fusion achieved to predict conversion in one year with 0.86 accuracy, comparing with 0.76 accuracy using linear fusion. The proposed method is useful for screening examination for AD or deciding a stratification approach within clinical trials since it achieved a high accuracy while the input data is relatively easy to be obtained.

It has been shown in the literature that Autism Spectrum Disorder (ASD) is associated with changes in brain network connectivity. Therefore, we investigate, if it is possible to capture any significant difference between brain connections of healthy subjects and ASD patients using resting-state fMRI time-series. To this end, we have developed large-scale Extended Granger Causality (lsXGC), which combines dimension reduction with source time-series augmentation and uses predictive time-series modeling for estimating directed causal relationships among resting-state fMRI time-series. This method is a multivariate approach, since it is capable of identifying the influence of each time-series on any other time-series in the presence of all other time-series of the underlying dynamic system. Here, we investigate whether this model can serve as a biomarker for classifying ASD patients from typical controls using a subset of 59 subjects of the Autism Brain Imaging Data Exchange II (ABIDE II) data repository. In this study, we use brain connections as features for classification and estimate them by lsXGC. As a reference method, we compare our results with cross-correlation, which is typically used in the literature as a standard measure of functional connectivity. After feature extraction, we perform feature selection by Kendall’s Tau rank correlation coefficient followed by classification using a Support Vector Machine (SVM). In order to evaluate the diagnostic accuracy of lsXGC, we compare its classification performance with cross-correlation. Within a cross-validation scheme of 100 different training/test data splits, we obtain a mean accuracy range of [0.7,0.81] and a mean Area Under the Receiver Operator Characteristic Curve (AUC) range of [0.78,0.85] across all tested numbers of features for lsXGC, which is significantly better than results obtained with cross-correlation namely mean accuracy of [0.57,0.61] and mean AUC of [0.54,0.59], which clearly demonstrates the applicability of lsXGC as a potential biomarker for ASD.

In Amyotrophic Lateral Sclerosis (ALS) spinal cord (SC) showed a moderate increase in FDG uptake with respect to healthy subjects. The main aim of our study is to integrate the information concerning the divergent behavior of SC with skeletal muscle metabolism improving the informative potential of ¹⁸F-fluoro-2-deoxy-glucose (FDG) PET/CT imaging regarding specific pathophysiological mechanisms underlying ALS progression. We analyzed 50 ALS patients with spinal onset consecutively submitted to FDG PET/CT imaging. Obtained data were compared to the corresponding findings in 36 age and sex-matched controls. A computational method was used to extract psoas volume and attenuation coefficient from CT images. Psoas volume was normalized for patient ideal body weight (IBW). In co-registered PET images, FDG accumulation was defined by average normalized standardized uptake value (N-SUV). Average Hounsfield values (AVH) in the psoas were similar in patients and controls (39±8 AHV vs 39±11 AHV, respectively, p=ns). By contrast, ALS was associated with a significant reduction in psoas volume normalized for IBW (8.8±2.9 mL/Kg IBW vs 10.3±2.7 mL/Kg IBW, respectively, p<0.05). More interestingly, N-SUV was significantly higher in patients than in controls (0.44±0.19 vs 0.29±0.09; p<0.001). These SUV values predicted overall survival rate at Kaplan-Meyer analysis (p<0.05) with a predictive power that was confirmed by univariate as well as by multivariate Cox analysis (p<0.02). ALS is therefore associated with a psoas reduction in volume and increase in FDG uptake. The intensity of FDG uptake within this muscular district is related to disease aggressiveness.

Early detection has a major contribution to the curability of breast cancer, and using mammographic images, this can be achieved non-invasively. Supervised deep learning, the dominant computer-aided detection (CADe) tool currently, has played a great role in object detection in computer vision, but it suffers from a limiting property: the need of a large amount of labelled data. This becomes stricter when it comes to medical datasets which require high-cost and time-consuming annotations. Furthermore, medical datasets are usually imbalanced, a condition that often hinders classifiers performance. The aim of this paper is to learn the distribution of the minority class to synthesise new samples in order to improve lesion detection in mammography. Deep Convolutional Generative Adversarial Networks (DCGANs) can efficiently generate breast masses. They were trained on increasing-size subsets of a mammographic dataset and used to generate diverse and realistic breast masses. The effect of including the generated images and/or applying horizontal and vertical flipping was tested in an environment where an imbalanced dataset (ratio of 1:10) of masses and normal tissue patches was classified using a fully- convolutional network. A maximum of 0:09 improvement of F1 score was reported by using DCGANs along with flipping augmentation in contrast to using the original images. We show that DCGANs can be used for synthesising photo-realistic breast mass patches with a considerable diversity. It is demonstrated that appending synthetic images in this environment, along with flipping, outperforms the traditional augmentation method of flipping solely, offering faster improvements as a function of the training set size.

Computer-aided diagnosis and radiomics have shown potential in diagnosis and prognosis of breast cancer. The purpose of this study was to investigate repeatability of classifier output and its relationship to classification performance of breast lesions imaged with dynamic contrast-enhanced MRI. Images of 1,169 breast lesions (267 benign, 902 cancers) were retrospectively collected under HIPAA/IRB compliance. The lesions were segmented automatically using a fuzzy c-means method and thirty-eight radiomic features were extracted. Three classification tasks were investigated, with different proportions of cases in each class: (i) benign (23%) vs. malignant (77%), (ii) “pure” ductal carcinoma in situ (DCIS) (25%) vs. DCIS with invasive ductal carcinoma (IDC) (75%), and (iii) invasive cancers of molecular subtype luminal A or luminal B (66%) vs. other molecular subtypes (34%). For each task, support vector machine classifiers were trained and tested within 0.632+ bootstrap analyses (1000 iterations) and the 0.632+ bias-corrected area under the ROC curve (AUC) served as the classification performance metric. Repeatability of classifier output was evaluated at three levels: a) repeatability by case (performance metric: width of the 95% confidence interval of classifier-estimated posterior probabilities for each case), b) repeatability within the dataset (performance metric: median and 95% confidence interval of the by-case 95% confidence interval widths), and c) potential relationship between classification performance and repeatability. In classification performance assessment, median AUCs [95% confidence interval] for the three tasks were 0.85 [0.83, 0.87], 0.84 [0.80, 0.87], and 0.65 [0.60, 0.69], respectively. In repeatability assessment within the dataset, the median confidence interval widths [95% confidence interval] for the posterior probabilities were 0.25 [0.08, 0.72], 0.34 [0.14, 0.84], and 0.23 [0.14, 0.68]. In conclusion, the classifiers in the first two tasks demonstrated strong classification performance while in all three they showed similar repeatability in posterior probabilities.

Convolutional neural networks (CNN) are increasingly used for image classification tasks. In general, the architectures of these networks are set ad hoc with little justification for selecting various components, such as the number of layers, layer depth, and convolution settings. In this work, we develop a structured approach to explore and select architectures that provide optimal classification performance. This was developed with an IRB-approved data set with 9,216 2-D maximum intensity projection (MIP) MRI breast images, containing breast cancer malignant or benign classes. This data set was divided into 7,372 training, 922 validation, and 922 test images. The architecture search method employs a genetic algorithm to find optimal ResNet-based classification architectures through crossover and mutation. Each generation, new model architectures are created from the genetic algorithm and trained with supervised machine learning. This search method identifies the model with the highest area under the ROC curve (AUC). The genetic algorithm produced an optimal model architecture which classifies malignancy in images with 76% accuracy and achieves an AUC score of .83. This approach offers a rational framework for automatic architecture exploration, potentially leading to a set of more efficient and generalizable CNN-based classifiers.

Automation of systematic scoring of breast glandularity on CT thorax examinations performed for another clinical reason could aid in detecting postmenopausal women with increased breast cancer risk. We propose a novel method that combines automated deep learning based breast segmentation from CT thorax examinations with computation of breast glandularity based on radiodensity and volumetric breast density. Reasonable segmentation Dice scores were found as well as very strong correlation between the risk measures computed on the ground truth and with the proposed approach. Hence, the proposed method can offer reliable breast cancer risk measures with limited additional workload for the radiologist.

Breast cancer represents the most common type of cancer worldwide among women. One of the most important diagnostic methods of this disease are mammograms, however, the high prevalence of breast cancer has not been reduced due to the incorrect diagnosis of these images, since they can be complex to interpret. An approach that represents a fundamental process for the improvement of this diagnosis is digital image processing, since it can facilitate the interpretation of the images for the specialists. In this work is proposed the implementation of a new multilevel segmentation approach based on the minimum cross-entropy threshold - Harris Hawks Optimization (MCET-HHO) metaheuristic algorithm, identifying regions within the breast that have abnormal tissue. Then, these regions are subjected to an automatic classification system based on a bag-of-visual-words (BoVW) approach to identify healthy tissue, benign tumors, and malignant tumors. According to the results, the classifier reached an average accuracy of 0.86 in the training stage and 0.73 in the testing, proving to be statistically significant in the automatic classification of mammograms, presenting a preliminary tool for the support of specialists in the diagnosis of mammography images.

Deep-learning based application for digital mammography screening is limited due to lack of labeled data. Generating digital mammogram (DM) from existing labeled digitized screen-film mammogram (DFM) dataset is one approach that may alleviate the problem. Generating high resolution DMs from DFMs is a challenge due to the limitations of network capacity and lack of GPU memory. In this study, we developed a deep learning framework, Cycle-HDDM, with which high resolution DMs were generated from DFMs. Our Cycle-HDDM model first used a sliding window to crop DFMs and DMs into patches of 256 by 256 in size. Then, we divided the patches into three categories (breast, background and boundary) using breast masks. We paired patches from the DFM and DMdatasets for training with the constraint that these paired patches should be sampled from the same category of the two different image sets. We used U-Net as the generators and modified the discriminators so that the outputs of the discriminators were a two-channel image, one channel for distinguishing real and synthesized DMs, and the other for representing a probability map for breast mask. We designed a study to evaluate the usefulness of Cycle-HDDM in a segmentation task, the objective of which was to estimate the percentage of breast density (PD) on DMs using deep neural network (DNN). With IRB approval, 1651 DFMs and 813 DMs were collected. Both DFMs and DMs were normalized to a pixel size of 100μm × 100μm for the experiments. The results show that the synthesized DMs by Cycle-HDDM could significantly improve (p < 0.001) the DNN-based mammographic density segmentation.

Deep learning has achieved great success in image analysis and decision making in radiology. However, a large amount of annotated imaging data is needed to construct well-performing deep learning models. A particular challenge in the context of breast cancer is the number of available cases that contain cancer, given the very low prevalence of the disease in the screening population. The question arises whether normal cases, which in the context of breast cancer screening are available in abundance, can be used to train a deep learning model that identifies locations that are abnormal. In this study, we propose to achieve this goal through the generative adversarial network (GAN)-based image completion. Our hypothesis is that if a generative network has a difficulty to correctly complete a part of an image at a certain location, then such a location is likely to represent an abnormality. We test this hypothesis using a dataset of 4348 patients with digital breast tomosynthesis (DBT) imaging from our institution. We trained our model on normal only images, to be able to fill in parts of images that were artificially removed. Then, using an independent test set, at different locations in the images, we measured how difficult it was for the network to reconstruct an artificially removed patch of the image. The difficulty was measured by mean squared error (MSE) between the original removed patch and the reconstructed patch. On average, the MSE was 2.11 times higher (with standard deviation equal to 1.01) at the locations containing expert-annotated cancerous lesions than that at the locations outside those abnormal locations. Our generative approach demonstrates a great potential for using this model to aid breast cancer detection.

Low dose CT screening has been shown to significantly reduce mortality rates due to lung cancer. To assist radiologists, CAD systems continue to be developed for automatically detecting, segmenting, and categorizing potentially malignant lung nodules. Deep learning with the U-Net architecture has shown to be effective for automatic segmentation of 2-D images. The network consists of a down-sampling and up-sampling path, similar to an auto-encoder. However, the concept driving its success is skip-connections between down-sampling and up-sampling, allowing the network to preserve details and for easier backpropagation of error to deep layers. This concept has previously been brought into 3-D and successfully applied to image volumes such as MRI and CT scans. This paper applies concepts from these works (skip connections, batch normalization, Dice similarity coefficient loss, and stride convolution for down sampling) to a 3-D Convolutional Neural Network for segmenting nodule image patches in thoracic CT scans from the LIDC-IDRI database. This database contains scans from 1018 cases with annotations from up to 4 human experts. Within each scan, nodules are delineated by each of these experts. It is well known that manual delineation can be subjective between annotators. This paper proposes a model trained on a ground truth estimation from the four expert annotations using the STAPLE algorithm. Experiments in this paper show that when trained on STAPLE, automatic segmentation with a 3-D U-Net can result in improved similarity scores to human annotators than similarity scores between human annotators.

Computer-aided diagnosis system for diffuse lung diseases (DLDs) is necessary for the objective assessment of the lung diseases. In this paper, we develop semantic segmentation model for 5 kinds of DLDs. DLDs considered in this work are consolidation, ground glass opacity, honeycombing, emphysema, and normal. Convolutional neural network (CNN) is one of the most promising technique for semantic segmentation among machine learning algorithms. While creating annotated dataset for semantic segmentation is laborious and time consuming, creating partially annotated dataset, in which only one chosen class is annotated for each image, is easier since annotators only need to focus on one class at a time during the annotation task. In this paper, we propose a new weak supervision technique that effectively utilizes partially annotated dataset. The experiments using partially annotated dataset composed 372 CT images demonstrated that our proposed technique significantly improved segmentation accuracy.

Lung cancer stands as the deadliest cancer worldwide, and early detection of pulmonary nodules is the focus of many studies to enhance the survival rate. As with many diseases, deep learning is becoming a commonly used technique for computer-aided diagnosis (CAD) in detecting lung nodules. Most lung CAD systems rely on a detection module followed by a false positive (FP) reduction module (FPR); however, FPR removes FPs as well as true positives (TPs). Thus, as a tradeoff, in order to retain high sensitivity, a large number of FPs remain. In our experience, small pulmonary vessels have been the primary source of FPs. Hence, we propose an additional module cascaded on normal FPR module to specifically reduce the number of FPs due to pulmonary vessel. Utilizing a 3D deep learning architecture, we find that the inclusion of various fields of view (FOVs) improves the accuracy of the chosen model. We explore the impact of the selection of the FOVs, the method used to integrate the features from each FOV, and using the FOV as a data augmentation method. We show that this vessel specific FPR module significantly improves the CAD system’s FP rate while only sacrificing 5% of the previously achieved sensitivity.

Low-dose CT (LDCT) has been commonly used for lung cancer screening and it is much desirable to computerize the image analysis for risk evaluation to reduce healthcare disparities. While informative structural image features can be extracted from medical images using state-of-the-art deep neural networks, other quantitative clinical measurements can also contribute to the overall assessment but are often ignored by researchers and also difficult to obtain. This work introduces a multi-task learning framework, which can simultaneously extract image features from LDCT images and estimate the clinical measurements for all-cause mortality risk prediction. The proposed method is a hybrid neural network with multi-scale input and multi-task supervision labels. The presented work shows that the extracted feature vectors have improved mortality prediction as they are generated to include both abstracted image features and high-level clinical knowledge.

Chronic obstructive pulmonary disease (COPD) is predicted to become the third leading cause of death worldwide by 2030. Smoking is a well-known risk factor in the development of COPD. Association between COPD genes and smoking have been studied. This paper presents an association analysis of single nucleotide polymorphisms (SNPs) with a CT imagebased phenotype of emphysema progression in heavy smokers. The emphysema progression was quantitatively represented by the annual increment of low attenuation volume (LAV) on CT scans for five years. 10 candidate SNPs were selected from 316 SNPs in 125 papers of genetic studies of COPD and lung cancer. The genotypes were determined by real-time polymerase chain reaction (PCR) using deoxyribonucleic acid (DNA) extracted from saliva samples. The association analysis was performed by Fisher's exact test and logistic regression analysis. This method was applied to a dataset with 144 participants (71 smokers, 61 past smokers, and 12 non-smokers). The results showed that the genotypes of rs3923564 and rs13180 SNPs were candidate SNPs associated with the CT image based-emphysema progression.

Artificial intelligence (AI) has great potential in medical imaging to augment the clinician as a virtual radiology assistant (vRA) through enriching information and providing clinical decision support. Deep learning is a type of AI that has shown promise in performance for Computer Aided Diagnosis (CAD) tasks. A current barrier to implementing deep learning for clinical CAD tasks in radiology is that it requires a training set to be representative and as large as possible in order to generalize appropriately and achieve high accuracy predictions. There is a lack of available, reliable, discretized and annotated labels for computer vision research in radiology despite the abundance of diagnostic imaging examinations performed in routine clinical practice. Furthermore, the process to create reliable labels is tedious, time consuming and requires expertise in clinical radiology. We present an Active Semi-supervised Expectation Maximization (ASEM) learning model for training a Convolutional Neural Network (CNN) for lung cancer screening using Computed Tomography (CT) imaging examinations. Our learning model is novel since it combines Semi-supervised learning via the Expectation-Maximization (EM) algorithm with Active learning via Bayesian experimental design for use with 3D CNNs for lung cancer screening. ASEM simultaneously infers image labels as a latent variable, while predicting which images, if additionally labeled, are likely to improve classification accuracy. The performance of this model has been evaluated using three publicly available chest CT datasets: Kaggle2017, NLST, and LIDC-IDRI. Our experiments showed that ASEM-CAD can identify suspicious lung nodules and detect lung cancer cases with an accuracy of 92% (Kaggle17), 93% (NLST), and 73% (LIDC) and Area Under Curve (AUC) of 0.94 (Kaggle), 0.88 (NLST), and 0.81 (LIDC). These performance numbers are comparable to fully supervised training, but use only slightly more than 50% of the training data labels .

Convolutional neural networks are known to require large amounts of data to achieve optimal performance. In addition, data is commonly computationally augmented using a variety of geometric and intensity transformations to further extent the set of training samples. In medical imaging, annotated data is often scarce or costly to obtain, and there is considerable interest in methods to reduce the amount of data needed. In this work, we investigate the relative benefit of increasing the amount of original data, with respect to computationally augmenting the amount of training samples, for the case of false positive reduction of lung nodules candidates. To this end, we have implemented a previously published topology for classification, shown to achieve state of the art results on the publicly available Luna16 dataset. Numerous models were trained using different amounts of unique training samples and different degrees of data augmentation involving rotations and translations, and the performance was compared. Results indicate that in general, better performance is achieved when increasing the amount of data, or augmenting the data more extensively, as expected. Surprisingly however, we observed that after reaching a certain amount of unique training samples, data augmentation leads to significantly better performance compared to adding the same number of new samples to the training dataset. We hypothesize that the augmentation has aided in learning more general {rotation and translation invariant-features, leading to improved performance on unseen data. Future experiments include more detailed characterization of this behavior, and relating this to the topology and amount of parameters to be trained.

Background: Imaging biomarkers derived from quantitative computed tomography (QCT) enable to quantify lung diseases and to distinguish their phenotypes. However, variability in radiomic features can have an impact on their diagnosis and prognosis significance. We aim to assess the effect of CT image reconstruction parameters on radiomic features in the PROSPR lung cancer screening cohort (1); thereby identifying more robust imaging features across heterogeneous CT images. Methods: CT feature extraction analysis was performed using a lattice-based texture estimation for data (n = 330) collected from a single CT scanner (Siemens Healthineers, Erlangen, Germany) with two different sets of image reconstruction kernels (medium (I30f), sharp (I50f)). A total of 26 features from three major statistical approaches, graylevel histogram, co-occurrence, and run-length, were computed. Features were calculated and averaged within a range of window sizes (W) from 4mm to 20mm. Furthermore, an unsupervised hierarchal clustering was applied to the features to identify distinct phenotypic patterns for the two kernels. The difference across phenotypes by age, sex, and Lung-Rads was assessed. Results: The results showed two distinct subtypes for two kernels across different window sizes. The heat map generated by radiomic features of the sharper kernel provided more distinct patterns compared to the medium kernel. The extracted features across the two kernels and their corresponding clusters were compared based on different clinical features. Conclusions: Our results suggest a set of radiomic features across different kernels can distinguish distinct phenotypes and can also help to assess the sensitivity of texture analysis to CT variabilities; helping for a better characterization of CT heterogeneity.

Surgical procedures often require the use of catheters, tubes, and lines, collectively called lines. Misplaced lines can cause serious complications, such as pneumothorax, cardiac perforation, or thrombosis. To prevent these problems, radiologists examine chest radiographs after insertion and throughout intensive care to evaluate their placement. This process is time consuming, and incorrect interpretations occur with notable frequency. Fast and reliable automatic interpretations could potentially reduce the cost of these surgical operations, decrease the workload of radiologists, and improve the quality of care for patients. We develop a segmentation model which can highlight the medically relevant lines in pediatric chest radiographs using deep learning. We propose a two-stage segmentation network which first classifies whether images have medically relevant lines and then segments images with lines. For the segmentation stage, we use the popular U-Net architecture substituting the encoder path with multiple state-of-the-art CNN encoders. Our study compares the performance of different permutations of model architectures for the task of highlighting lines in pediatric chest radiographs and demonstrates the effectiveness of the two-stage architecture.

Lung cancer CT screening has been carried out. Unnecessary biopsy is performed in 20-55% of cancer candidate cases. Several malignant risk models have been published to reduce the false positive rate of lung cancer. In this study, we develop a high-performance malignant risk model. This risk model consists of Generalized Additive Model (GAM) using diameter, pleural attachment area rate, CT kurtosis, GLCM_Inertia, GLCM_IDM and GLCM_Energy_in_marginal_region. This model shows effectiveness by showing AUC 0.918 compared to the current Pancan model.

This study aims to combine differences in radiomic features between internal and peripheral portions of lungs diagnosed with idiopathic pulmonary fibrosis (IPF) and with TOLLIP and MUC5B genetic mutations to predict patient prognosis. A database of computed tomography (CT) scans from 169 IPF patients was selected from the INSPIRE study along with the corresponding genomic and demographic datasets. Three CT sections per patient were chosen to represent the superior, middle, and inferior portions of the lungs. Twelve regions of interest (ROIs) were placed in central and peripheral portions at each level of the lungs, and 142 radiomics features were calculated within each ROI. Based on feature reproducibility, 30 features were used with logistic regression and receiver operating characteristic (ROC) analysis to classify patients with various genetic mutations. Kaplan-Meier survival curve models quantified the ability of each feature to differentiate between survival curves based on a feature-specific threshold. Nine first-order features and one fractal feature were found to be predictive of TOLLIP-1 (rs4963062) mutation (AUC 0.54-0.74). Five Laws’ filter features were predictive of TOLLIP-2 (rs5743905) mutation (AUC 0.53-0.70), while no feature was found to be predictive for MUC5B mutations. First-order and fractal features reflected the greatest discrimination between Kaplan-Meier curves. A radiogenomic approach for predicting patient genetic mutations based on radiomics features extracted from thoracic CT images of patients with IPF has potential as a biomarker. These same features can also serve as predictors of patient prognosis using a survival curve modeling approach.

Volumetric lung tumor segmentation is essential for monitoring tumor response to treatment by tracking lung tumor changes. However, it is difficult to segment due to the diversity of size, shape, location as well as types such as solid, subsolid and necrosis of lung tumor and it is difficult to distinguish between the tumor and the nearby structures because of their low contrast in case of tumors attached to chest wall or mediastinum. In this study, we propose a coupling-net with shape-focused prior that focuses on segmentation of various types of lung tumor and prevent leakage into nearby structures. First, to extract shape information, 2D-Net is trained in each axial, coronal, and sagittal planes. Second, to generate the shape-focused prior including suspicious area of the lung tumor, the prediction maps are integrated with maximum voting, and shape-focused prior was made by applying the narrow-band distance propagation. Finally, to prevent leakage due to low contrast between lung tumor and adjacent structures and give the constraint using shape-focused prior, a 3D-Net is trained using shape-focused prior. To validate segmentation performance, we divide into four types according to the tumor location characteristics: non-attached tumor (Type 1), chest wall-attached tumor (Type 2), mediastinum-attached tumor (Type 3), and surrounded-tumor by chest wall, or liver in apex and base in the lung (Type 4). Our proposed network showed best segmentation performance without a leak to adjacent structures due to considering shape-focused prior.

Purpose: Knowledge of molecular subtypes of gliomas can provide valuable information for tailored therapies. This study aimed to investigate the use of deep convolutional neural networks (DCNNs) for noninvasive glioma subtyping with radiological imaging data according to the new taxonomy announced by the World Health Organization in 2016. Methods: A DCNN model was developed for the prediction of the five glioma subtypes based on a hierarchical classification paradigm. This model used three parallel, weight-sharing, deep residuallearning networks to process 2.5-dimensional input of trimodal MRI data, including T1-weighted, T1-weighted with contrast enhancement, and T2-weighted images. A data set comprising 1,016 real patients was collected for evaluation of the developed DCNN model. The predictive performance was evaluated via the area under the curve (AUC) from the receiver operating characteristic analysis. For comparison, the performance of a radiomicsbased approach was also evaluated. Results: The AUCs of the DCNN model for the four classification tasks in the hierarchical classification paradigm were 0.89, 0.89, 0.85, and 0.66, respectively, as compared to 0.85, 0.75, 0.67, and 0.59 of the radiomics approach. Conclusion: The results showed that the developed DCNN model can predict glioma subtypes with promising performance, given sufficient, non-ill-balanced training data.

Angiographic Parametric Imaging (API) is a tool based on the parametrization of Time-Density Curves (TDCs) from Digital Subtraction Angiography (DSA). Parameters derived from the TDCs correlate moderately with hemodynamics, yet underuse the hemodynamic information encoded in a TDC. To determine whether better diagnoses can be made through a more complete utilization of the information in the TDCs, we implemented an analysis using Recurrent Neural Networks (RNNs). These are a class of neural networks that analyze and make predictions using time sequences such as the TDCs. We investigated the feasibility of using RNNs to make treatment outcome predictions using TDCs obtained from angiograms of Intracranial Aneurysms (IAs) treated with Pipeline Embolization Devices (PED). Six-month follow-up angiograms were collected to create binary labels regarding treatment outcome (occluded/un-occluded). API parameters obtained were Mean Transit Time, Time to Peak, Time to Arrival, and Peak Height. Parameters were used to simulate TDCs which were normalized to account for variability between interventions. An RNN was trained and tested to predict IA treatment outcome. A 20-fold Monte Carlo Cross Validation was conducted to evaluate robustness of the RNN. The RNN predicted occlusion outcome of IAs with an average accuracy of 74.4% (95% CI, 72.6%-76.1%) and 65.6% (63.4%- 67.2%) and average area under the receiver operating characteristic curve of 0.73 (0.70-0.76) and 0.56 (0.51-0.61) for normalized and un-normalized sub-groups respectively. This study proves the feasibility of using RNNs to predict treatment outcome of IAs treated with a PED using TDCs simulated from temporal features obtained through API.

Several studies have found that different quantitative MRI (qMRI) measurements are associated with the presence of Alzheimer's disease. Cognitive Assessment scores, Apolipoprotein E4 (ApoE4) and cerebrospinal fluid (CSF) biomarkers are important factors associated with the risk of conversion from MCI to AD. Despite this awareness, the relationship of the qMRI measurements with the conversion rate and their effect in multivariate survival models that combine Radiomics, CSF, ApoE4 and Cognitive assessment is not known. The objective of this work was to evaluate the importance of each data source using several machine learning(ML) approaches that build Cox Survival models that combine cognitive assessments, CSF, ApoE4 and qMRI features. 321 features from 442 subjects from the ADNI study that converted from the MCI status to AD were used. ML methods were explored in a Cross-validation framework. Test results indicated that cognitive assessments plus qMRI data produce Cox survival models that are 92% concordant with the conversion time from MCI to AD, while CSF biomarkers did not have a mayor contribution on the final survival Model.

Parkinson's disease (PD) is a central nervous system disease that common occurs among older peoples. Automatic identification the degree of early Parkinson's disease is a meaningful work today. In this paper, the diagnostic experiment on Parkinson is executed on feature data sets which contain a variety data of handwriting and speech gathered from Parkinson's patients and ordinary people. Since most data in these datasets contains noise, right selection of feature from subsets plays a significant role on enhancing the performance of classification. To increase the accuracy of classification, this paper proposes an improved discrete artificial bee colony algorithm to determine the optimal subset of features. Our algorithm combines both advantages of filters and wrappers feature selection strategies to eliminate most of the uncorrelated or noisy features which could improve the efficiency of feature selection. The experimental results demonstrate our algorithm achieves better results on both precision and shortening the amounts of subsets among the comparison with other modern Evolutionary Computation algorithms.

The purpose of this work is to determine the strength of correlations between imaging data and local tumor grade using spatially specific tumor samples to validate against a histologic gold-standard. This improves our understanding of diagnostic imaging by correlating with underlying biology. Glioma patients were enrolled in an IRB approved prospective clinical imaging trial between 2013 and 2016. MR imaging was performed with anatomic (T1, T2, FLAIR, T1 post-contrast, and susceptibility), diffusion tensor, dynamic susceptibility and dynamic contrast sequences. During surgery stereotactic biopsy were collected prior to resection along with image space coordinates of the samples. A random forest were built to predict the grade of each sample using preoperative imaging data. The model was assessed based on classification accuracy, Cohen’s kappa, and sensitivity to higher grade disease Twenty-three patients with fifty-two total biopsy samples were analyzed. The Random Forest method predicted tumor grade at 94% accuracy using four inputs (T2, ADC, CBV and Ktrans). Using conventional imaging only, the overall accuracy decreased (89% overall, κ = 0.78) and 71% of high grade samples were misclassified as lower grade disease. We found that pathologic features can be predicted to high accuracy using clinical imaging data. Advanced imaging data contributed significantly to this accuracy, adding value over accuracies obtained using conventional imaging only. Confirmatory imaging trials are justified.

In brain cancer surgery, maximal tumor resection improves overall survival and quality of life survival in low-grade and high-grade glioma. Different technologies such as intraoperative magnetic resonance imaging and computed tomography have made major contributions; however, these technologies do not provide quantitative, real-time and three-dimensional continuous guidance. Optical Coherence Tomography (OCT) is a non-invasive, label-free, real-time, high-resolution imaging modality that has been explored for glioma infiltration detection. Here we report a novel Artificial Neural Network (ANN)-based computer-aided diagnosis (CAD) method for automated, real-time, in situ detection of glioma-infiltrated tumor margins. Near 500 volumetric OCT samples were intraoperatively obtained from resected brain tissue specimens of 21 patients with glioma tumors of different stages and labeled as either non-cancerous or glioma-infiltrated based on histopathology evaluation (gold standard). Labeled OCT images from 12 patients were used as training dataset to develop the artificial neural network. Unlabeled OCT images from the other 9 patients were used as a validation dataset to quantify the method detection performance. The CAD system achieved excellent levels of both sensitivity and specificity (~90%) for detecting glioma-infiltrated tissue with high spatial resolution (~16 μm laterally). Previous methods for OCT-based detection of glioma-infiltrated brain tissue rely on underlying optical properties such as attenuation coefficient from the OCT signal requiring sacrificing spatial resolution and cumbersome calibration procedures. By overcoming these major challenges, our novel ANN-assisted CAD system will enable implementing practical OCT-guided surgical tools for continuous, real-time and accurate intra-operative detection of glioma-infiltrated brain tissue, facilitating maximal glioma resection and superior surgical outcomes for glioma patients.

Psychotic disorders such as schizophrenia and bipolar disorder are difficult to classify because they share overlapping symptoms. Deriving biomarkers of illness using structural MRI dataset are essential because they may lead to improved diagnosis. Previous studies typically predict the diagnosis labels using supervised classifiers that rely on truly labeled dataset. Mislabeled subjects may increase the complexity of the predictive model and may impact its performance. In this work, we address the problem of inaccurate diagnosis labeling of psychotic disorders using a data-driven approach. We performed dimension reduction using PCA on the vectorized images and then k-mean clustering on the components. We evaluate our method on a structural MRI dataset, with over 900 subjects labeled using DSM-IV and biotypes. An ANOVA statistical significance test was performed after clustering based on each labelling approach and after clustering. Subjects were grouped into 5 clusters using our method, and each cluster includes all types of patients. However, we found statistically significant group differences in brain regions across 5 clusters, while for DSM and biotype, there were no significant differences. Our results also show the performance of the predictive model improved significantly using datadriven labels. Our method shows underlying biological changes associated with mental illness may be identified by studying and considering features of the brain imaging data, and annotating brain imaging data using a data-driven approach may eventually lead to improved diagnosis and advanced drug discovery and help patients.

Background: Vascular segmentation of cerebral vascular imaging is tedious and manual, hindering translation of imagebased computational tools for neurovascular disease (such as intracranial aneurysm) management. Current cerebrovascular segmentation techniques use classic model-based algorithms, but such algorithms are incapable of distinguishing vasculature from artifacts. Deep Learning, specifically the widely accepted U-Net architecture, could be an effective alternative to conventional approaches for cerebrovascular segmentation, but has been shown to perform poorly in segmentation of smaller yet critical vessels. Methods: In this study, we present a methodology using a specialized convolutional neural network (CNN) architecture— DeepMedic—which uses multi-resolution inputs to enhance the field of view of the architecture, thereby enhancing the accuracy of segmentation of smaller vessels. To show the capability of this architecture, we collected and segmented a total of 100 digital subtraction angiography (DSA) images of cerebral vessels for training, internal validation, and testing (n=80, n=10, and n=10, respectively). Results: The DeepMedic architecture yielded high performance with a Connectivity-Area-Length (CAL) of 0.84±0.07 and a dice similarity coefficient (DSC) of 0.94±0.02 in the independent testing cohort. This was better than U-Net optimized for the patch-size and %-overlap in predictions, which performed with a CAL of 0.79±0.06 and a DSC of 0.92±0.02. Notably, our work demonstrated that DeepMedic (CAL: 0.45±0.12) outperformed U-Net (CAL: 0.59±0.11) for segmentation of smaller vessels. Conclusions: Our work showed DeepMedic performs better than the current state-of-the-art method for cerebrovascular segmentation. We hope this study begins to bring a high fidelity deep-learning based approach closer to clinical translation.

Brain metastases are one of the most common neurologic complications of cancers, occurring in about 30% of all patients with cancer. Moreover, about 40% of brain metastases patients have more than three metastases. Stereotactic radiosurgery (SRS) is a well-established treatment for brain metastases, which requires accurate detection and delineation of the brain metastases. However, manually detecting and locating all the brain metastases can be very time-consuming and laborintensive, which is a big efficiency bottleneck in this typical one-day outpatient SRS procedure. Developing a fast automatic detection tool of brain metastases is highly desirable, but is very challenging given the large number of brain metastases that a patient can have and the small size that a brain metastasis can be. In this work, we propose to use a 3D Mask R-CNN method to automatically and quickly detect the brain metastases on magnetic resonance (MR) images for SRS. At the training stage, coarse feature maps were extracted from 3D MR image patches using pretrained ResNet. Then, a region proposal network (RPN) was used to predict the locations and sizes of the rough candidate tumor ROIs from the coarse feature maps. By using a uniformed fully convolution network (FCN), the metastases within ROI was segmented. The segmentation loss, classification loss (metastases or non-metastases), as well as ROI location and size regression loss were used to supervise the proposed networks. For a new query patient, candidate ROIs and predicted probability maps within ROIs were obtained from our trained model. By aggregating ROIs and the tumor probability maps and performing a consolidation via weighted cluster scoring, the final ROIs of the brain metastases was obtained. We have tested our method on 20 patients’ brain contrast T1-weighted MR images, and achieved 86.5%±3.2% sensitivity and 89.7%±4.8% specificity. For each patient, it took our trained model a few seconds to detect the brain metastases on the 3D MR images. The results of our preliminary study have demonstrated its efficacy and clinical feasibility. This auto-detection method could be a useful tool to significantly improve the efficiency of SRS treatment planning and hence ultimately improve the clinical outcome.

Stereotactic radiosurgery (SRS) is widely used to obliterate arteriovenous malformations (AVMs). Its performance relies on the accuracy of delineating the target AVM. Manual segmentation during a framed SRS procedure is timeconsuming and subject to inter- and intra-observer variation. Therefore, it is important to develop an automatic segmentation method to delineate the AVM target from CT images. In this study, we retrospectively investigated 80 patients who were treated with SRS. Ground truth was manually generated by an experienced physician using both DSA and CT images. A fast region proposal network was first trained to propose a bounding box that contains the AVM lesion for detection. The bounding box was then used to guide image patch sampling process for V-Net training. In the testing stage, possible AVM locations were first proposed by the region proposal network. Subsequently, V-Net was used for the final label prediction. Both the region proposal network and V-Net were trained using 60 patients and tested using 20 patients. The mean Dice similarity coefficient (DSC) was calculated to evaluate the accuracy of the proposed method. The automatic contours were in very good agreement to the ground truth contours with an average DSC < 0.85.

Sagittal craniosynostosis is the most common craniosynostosis and the premature closure of the sagittal suture makes the skull shape biparietal narrow and elongated. Surgery in infants with sagittal craniosynostosis is a common treatment for correcting the shape of the deformed skull, but the evaluation of the morphological improvement of the skull before and after operation is made by the surgeon’s subjective judgment. Therefore, we propose an efficient method to quantify the shape of the skull before and after surgery based on the mean normal skull model to assess the surgical outcome. In the preprocessing step, to construct the skull model from the pre- and post-operative CT images, each skull model is constructed consisting of the outer surface of the skull. To distinguish individual cranial bones separated by sutures, mean normal skull model is composed of five cranial bones. In the skull model deformation step, to distinguish from the whole bone of the preoperative skull model to the regional bone, the mean normal skull model is deformed into a preoperative skull model, and the deformed mean normal skull model is again deformed into a postoperative skull model. In the regional skull shape index calculation step, to evaluate the degree of expansion and reduction of postoperative skull relative to the preoperative skull, the regional skull shape index is calculated. Experimental results showed that our regional skull shape index can quantify the degree of expansion and reduction of the postoperative skull relative to the preoperative skull of each cranial bone.

About a third of all cancer patients develop brain metastases. Stereotactic radiosurgery (SRS) is a common treatment for patients with brain metastases and offers excellent tumor control rates. However, the most significant clinical challenge for patients after SRS is reliably differentiating tumor progression from radiation necrosis, a side effect of SRS, which occurs in about 20% of all metastatic brain tumor patients. The current diagnostic accuracy in distinguishing radiation necrosis from tumor recurrence via visual inspection by expert radiologists on routine MRI scans is 50-60%. Invasive surgical intervention provides the only definitive means of diagnosis. Therefore, finding non-invasive markers to reliably differentiate between these diagnoses are urgently needed. In this work, we present a radiomic-pipeline that involves capturing texture features (Laws, Haralick, Gabor) from not just the lesion but also the peri-lesion environment (defined as “lesion habitat”) to distinguish radiation necrosis and tumor recurrence. We obtained a discovery dataset (n=37 studies) from Cleveland Clinic with post-Gd T1w, T2w, and FLAIR volumes. Patients were confirmed via histopathology (biopsy or resection) as having a recurrent tumor (n=17) or radiation necrosis (n=20). A separate set of 12 patients was kept blinded as the hold-out" set. Each patient's T2 and FLAIR volumes were co-registered onto Gd-T1w. Lesion habitat sub-compartments were annotated by an expert as enhancing tumor, necrosis, and perilesional edema. Following pre-processing, a total of 4,740 texture features (including Haralick, Gabor, Laws, CoLlAGe) were extracted from each sub-compartment of the lesion habitat across all three available MRI sequences. To identify the most discriminatory features, we used Wilcoxon rank-sum test to remove correlated features (correlation 0.75). This was followed by a 3-fold cross-validation scheme, within a random forest classifier using the training cohort. The most discriminatory features were then evaluated on the hold out test set. Within our discovery set, we obtained an average AUC of .9781 with a sensitivity of .91 and a specificity of .92. On the blinded hold out set, we obtained an accuracy of 91% in distinguishing radiation necrosis from tumor recurrence. Laws features (capturing edges, waves, ripple patterns) from the enhancing tumor region on Gd-T1w MRI were identified as being the most discriminatory in distinguishing radiation necrosis and tumor recurrence in METs patients.

Routine surveillance endoscopies are currently used to detect dysplasia in patient with Barrett's Esophagus (BE). However, most of these procedures are performed by non-expert endoscopists in community hospitals. Leading to many missed dysplastic lesions, which can progress into advanced esophageal adenocarcinoma if left untreated.1 In recent years, several successful algorithms have been proposed for the detection of cancer in BE using high-quality overview images. This work addresses the first steps towards clinical application on endoscopic surveillance videos. Several challenges are identified that occur when moving from image-based to video-based analysis. (1) It is shown that algorithms trained on high-quality overview images do not naively transfer to endoscopic videos due to e.g. non-informative frames. (2) Video quality is shown to be an important factor in algorithm performance. Specifically, temporal location performance is highly correlated with video quality. (3) When moving to real-time algorithms, the additional compute necessary to address the challenges in videos will become a burden on the computational budget. However, in addition to challenges, videos also bring new opportunities not available in the current image-based methods such as the inclusion of temporal information. This work shows that a multi-frame approach increases performance compared to a naive single-image method when the above challenges are addressed.

In this work, we present a computer-aided diagnosis system that uses deep learning and decision fusion to classify patients into one of three classes: “Likely Prostate Cancer," “Equivocal" and “Likely not Prostate Cancer." We impose the group “Equivocal" to reduce misclassifications by allowing for uncertainty, akin to prostate imaging reporting systems used by radiologists. We trained 3D convolutional neural networks to perform two binary patient-level classification tasks: classification of patients with/without prostate cancer and classification of patients with/without clinically significant prostate cancer. Networks were trained separately using volumetric T2-weighted images and apparent diffusion coefficient maps for both tasks. The probabilistic outputs of the resulting four trained networks were combined using majority voting followed by the max operator to classify patients into one of the three classes mentioned above. All networks were trained using patient-level labels only, which is a key advantage of our system since voxel-level tumour annotation is often unavailable due to the time and effort required of a radiologist. Our system was evaluated by retrospective analysis on a previously collected trial dataset. At a higher sensitivity setting, our system achieved 0.97 sensitivity and 0.31 specificity compared to an experienced radiologist who achieved 0.99 sensitivity and 0.12 specificity. At a lower sensitivity setting, our system achieved 0.78 sensitivity and 0.77 specificity compared to 0.76 sensitivity and 0.77 specificity for the experienced radiologist. We envision our system acting as a second reader in pre-biopsy screening applications.

This paper presents a visualization method of intestine (the small and large intestine) regions and their stenosed parts caused by ileus from CT volumes. Since it is difficult for non-expert clinicians to find stenosed parts, the intestine and its stenosed parts should be visualized intuitively. Furthermore, the intestine regions of ileus cases are quite hard to be segmented. The proposed method segments intestine regions by 3D FCN (3D U-Net). Intestine regions are quite difficult to be segmented in ileus cases since the inside the intestine is filled with liquids. These liquids have similar intensities with intestinal wall on 3D CT volumes. We segment the intestine regions by using 3D U-Net trained by a weak annotation approach. Weak-annotation makes possible to train the 3D U-Net with small manually-traced label images of the intestine. This avoids us to prepare many annotation labels of the intestine that has long and winding shape. Each intestine segment is volume-rendered and colored based on the distance from its endpoint in volume rendering. Stenosed parts (disjoint points of an intestine segment) can be easily identified on such visualization. In the experiments, we showed that stenosed parts were intuitively visualized as endpoints of segmented regions, which are colored by red or blue.

Three-dimensional (3D) convolutional neural networks (CNNs) can process volumetric medical imaging data in their native volumetric input form. However, there is little information about the comparative performance of such models in medical imaging in general and in CT colonography (CTC) in particular. We compared the performance of a 3D densely connected CNN (3D-DenseNet) with those of the popular 3D residual CNN (3D-ResNet) and 3D Visual Geometry Group CNN (3D-VGG) in the reduction of false-positive detections (FPs) in computer-aided detection (CADe) of polyps in CTC. VGG is the earliest CNN design of these three models. ResNet has been used widely as a de-facto standard model for constructing deep CNNs for image classification in medical imaging. DenseNet is the most recent of these models and improves the flow of information and reduces the number of network parameters as compared to those of ResNet and VGG. For the evaluation, we used 403 CTC datasets from 203 patients. The classification performance of the CNNs was evaluated by use of 5-fold cross-validation, where the area under the receiver operating characteristic curve (AUC) was used as the figure of merit. Each training fold was balanced by use of data augmentation of the samples of real polyps. Our preliminary results showed that the AUC value of the 3D-DenseNet (0.951) was statistically significantly higher than those of the reference models (P < 0.005), indicating that the 3D-DenseNet has the potential of substantially outperforming the other models in reducing FPs in CADe for CTC. This improvement was highest for the smallest polyps.

The diagnosis of Crohn's disease (CD) can be challenging given variation in anatomic disease distribution, morphology, and proportion of intestine affected. Subsequently, the appearance and presentation of disease on cross-sectional imaging are a heterogeneous combination of shapes and image features, making differentiation of normal vs. diseased small intestine prone to inter-observer variation. Applying machine learning methods to cross-sectional, imaging interpretation may improve the accuracy of CD diagnosis and distinguish normal from diseased intestine by automated approaches. Using a set of 207 CT-enterography (CTE) scans, two independent radiologists labeled the presence of disease vs. non-disease at 7.5mm intervals along the length of the bowel (mini-segments), generating a dataset of 10,552 observations for model training and testing. We introduce two types of classifiers to quantitatively assess CD related intestinal damage for each mini-segment. The sensitivity, specificity and AUC for the best performing ensemble and CNN models are 84.9%, 84.7%, 0.93, and 90.9%, 78.6%, 0.92 respectively. The accuracy for classifying full segments as diseased vs. normal using ensemble and CNN models are 96.3% and 90.7% respectively.

Over the past few decades, primarily developed countries witnessed an increased incidence of esophageal adenocarcinoma (EAC). Screening and surveillance of Barrett’s esophagus (BE), which is known to augment the probability of developing EAC, can significantly improve survival rates. This is because early-stage dysplasia in BE can be treated effectively, while each subsequent stage complicates successful treatment and seriously reduces survival rates. This study proposes a convolutional neural network-based algorithm, which classifies images of BE visualized with White Light Endoscopy (WLE) as either dysplastic or non-dysplastic. To this end, we use merely pixels surrounding the dysplastic region, while excluding the pixels covering the dysplastic region itself. The phenomenon where the diagnosis of a patient can be determined from tissue other than the clearly observable diseased area, is termed the field effect. With its potential to identify missed lesions, it may prove to be a helpful innovation in the screening and surveillance process of BE. A statistical significant difference test indicates the presence of the field effect in WLE, when comparing the distribution of the algorithm classifications of unseen data and the distribution obtained by a random classification.

Purpose of this paper is to present a method for visualising decision-reasoning regions in computer-aided pathological pattern diagnosis of endocytoscopic images. Endocytoscope enables us to perform direct observation of cells and their nuclei on the colon wall at maximum 500-times ultramagnification. For this new modality, computer-aided pathological diagnosis system is strongly required for the support of non-expert physicians. To develop a CAD system, we adopt convolutional neural network (CNN) as the classifier of endocytoscopic images. In addition to this classification function, based on CNN weights analysis, we develop a filter function that visualises decision-reasoning regions on classified images. This visualisation function helps novice endocytoscopists to develop their understanding of pathological pattern on endocytoscopic images for accurate endocytoscopic diagnosis. In numerical experiment, our CNN model achieved 90 % classification accuracy. Furthermore, experimental results show that decision-reasoning regions suggested by our filter function contain characteristic pit patterns in real endocytoscopic diagnosis.

Gastric cancer is one of the most common malignant tumors with high mortality rate worldwide. In order to optimally treating gastric cacner patients and reduce cancer mortality, it requires to accurately predict Tumor, Node, Metastasis (TNM) staging of the tumor, which will determine whether the patients need neoadjuvant chemotherapy before surgery. However, subjectively reading CT images is difficult to predict TNM with large intra- and inter-reader variability. To address this challenge, we developed and tested a new CAD approach that uses radiomics features computed from the segmented tumor regions depicting on CT images to build a machine learning classifier to predict TNM and divide patients into two groups of whether need neoadjuvant chemotherapy. A CT image dataset involving 219 gastric cancer patients was retrospectively assembled and used in this study. In addition, 3 clinicopathological markers were also acquired and used. From an initial pool of 367 radiomics features, an optimal set of 11 features was selected. Then, 3 machine learning classifiers using (1) 11 CT image features, (2) 3 clinicopathological markers, and (3) fusion of both 11 CT image features and 3 clinicopathological markers, were trained and tested using a leave-one-case-out validation methods. Areas under ROC curves of three classifies are 0.74, 0.71, and 0.79, respectively. The results indicated that (1) radiomics image features computed from CT images carry higher discriminatory power to predict TNM than using clinicopathological markers acquired from surgically resected specimen and (2) fusion of CT image features and clinicopathological markers can further increase performance to predict TNM of gastric cancer patients.

A computer-aided detection (CAD) second reader of colorectal polyps can decrease the rate of missed polyps in actual colonoscopy procedures. Currently, regular screening of colorectal cancer (CRC) demands a colonoscopy procedure during which polyps are located and removed. Unfortunately, different investigations have reported 22%-28% of polyps and 20%-24% of adenomatous polyps are missed. The adenoma detection rate (ADR) is a colonoscopy quality indicator highly dependent on expert training, spent time, device withdrawal technique, colon preparation and procedure-dependent factors. Several approaches have improved ADR, namely image enhancement, advancements in endoscope design and developments of accessories. Recently, artificial intelligence (AI) has shown potential to aid the task of polyp detection. This paper introduces an automatic detection of polyps that localize hyperplastic and adenomatous colorectal polyps in colonoscopy images and full video sequences. The proposed pipeline is achieved by two sequentially encoder-decoder Convolutional Neural Networks: The first detects frames with high probability of having polyps and the second estimates the actual location of the polyp. Detection of polyps showed an Annotated Area Covered AAC = 0.889 and IoU = 0.816 in actual colonoscopy images containing at least a polyp. In addition, in colonoscopy videos achieved a 0.63, 0.85, 0.65 of precision, specificity, and F1-score respectively for the ASU-Mayo database.

Computer-aided diagnosis (CADx) of polyps is essential for advancing computed tomography colonography (CTC) with diagnostic capability. In this paper, we present a study of investigating the performance between deep learning and Random Forest (RF) classifier for polyp differentiation in CTC. First, we conducted feature extraction via an extended Haralick model (eHM) to build a total of 30 texture features. The gray level co-occurrence matrix (GLCM) is generated to encode 3D CT image information into a 2D matrix as input to the convolutional neural network (CNN). Then, we split the polyp classification into two state-of-the-art frameworks: the eHM texture features/RF and the GLCM texture matrices/CNN. We evaluated their performances by the merit of area under the curve of receiver operating characteristic using 1,278 polyps (confirmed by pathology). Results demonstrated that by balancing the data, both CNN model and RF classifier can learn or analyze features effectively, and achieve high performance. RF classifier in general outperformed CNN model with a gain of 6.4% (balanced datasets) and 5.4% (unbalanced datasets), showing its effective in feature extraction and analysis for polyp differentiation. However, the performance of CNN got improved through the addition of new data with a gain of 3.6% (balanced datasets) and 3.4% (unbalanced datasets), whereas RF classifier showed no gain when we enlarged datasets. This demonstrated that CNN model have the potential to improve the classification task performance when dealing with larger dataset. This study provided valuable information on how to design experiments to improve CADx of polyps.

Colorectal cancer is the second leading cause of cancer deaths worldwide. Computed tomographic colonography (CTC) can detect large colorectal polyps and cancers at a high sensitivity, whereas it can miss some of the smaller but still clinically significant 6 – 9 mm polyps. Dual-energy CTC (DE-CTC) can be used to provide more detailed information about scanned materials than does conventional single-energy CTC. We compared the classification performance of a 3D convolutional neural network (DenseNet) with those of four traditional 3D machine-learning models (AdaBoost, support vector machine, random forest, Bayesian neural network) and their cascade and ensemble classifier variants in the detection of small polyps in DE-CTC. Twenty patients with colonoscopy-confirmed polyps were examined by DE-CTC with a reduced one-day bowel preparation. The traditional machine-learning models were designed to identify polyps based on native radiomic dual-energy features of the DE-CTC image volumes. The performance of the machine-learning models was evaluated by use of the leave-one-patient-out method. The DenseNet was trained with a large independent external dataset of single-energy CTC cases and tested on blended image volumes of the DE-CTC cases. Although the DenseNet yielded the highest detection accuracy for typical polyps, AdaBoost and its cascade classifier variant yielded the highest overall polyp detection performance.

Deep learning such as Convolutional Neural Network (CNN) has demonstrated its superior in the field of image analysis. However, in the medical imaging field, deep learning faces more challenges for tumor classification in computer-aided diagnosis due to uncertainties of lesions including their size, scaling factor, rotation, shapes, etc. Thus, instead of feeding raw images, texture-based CNN model has been designed to classify the objects with their good attributes. For example, gray level co-occurrence matrix (GLCM) can be chosen as the descriptor of the texture pattern for many good properties such as uniform size, shape invariance, scaling invariance. However, there are many different texture metrics to measure the different texture patterns. Thus, an effective and efficient integration model is essential to further improve the classification performance from different texture patterns. In this paper, we proposed a multi-channel texture-based CNN model to effectively integrate intensity, gradient and curvature texture patterns together for differentiating the malignant from benign polyps. Performance was evaluated by the merit of area under the curve of receiver operating characteristics (AUC). Around 0.3~4.8% improvement has been observed by combining different texture patterns together. Finally, classification performance of AUC=86.7% has been achieved for a polyp mass dataset of 87 samples, which obtains 1.8% improvement compared with a state-of-the-art method. The results indicate that texture information from different metrics could be fused and classified with a better classification performance. It also sheds lights that data integration is important and indispensable to pursuit improvement in classification task.

In this article, we introduce a deformation independent model to solve the shape and posture changing issue for polyp characterization in computer-aided diagnosis (CADx) via CT colonography. After volumetric data parameterization in a four-dimensional space, the first fundamental form (FFF) is employed to construct the polyp model which contains several excellent properties such as locality, symmetry, orientation robustness, shift and isometric invariance. In consideration of the scaling effects, gray level co-occurrence matrix (GLCM) is utilized to remove the scaling factor and extract texture descriptors. As a symmetrical square tensor, however, it is difficult to put the FFF into GLCM directly. To solve this problem, we perform matrix decomposition on FFF to extract its eigenvalues and eigenvectors which are used to construct three metric images as the input of GLCM. Then Haralick measures extracted from GLCM are applied to construct texture descriptors which are fed to a random forest classifier to perform polyp classification. Experiments show that the proposed method obtains an encouraging classification performance with area under the curve of receiver operating characteristics (AUC score) of 95.3% which is a significant improvement comparing with five existing methods.

In-vivo imaging of the prostate has shown to be useful for prostate cancer (PCa) localization especially during biopsy procedures. Multi-parametric MRI (mp-MRI) is gaining rapid popularity amongst clinicians but is complex and difficult to interpret by even expert radiologists. Prostate specific membrane antigen positron emission tomography (PSMA PET) is emerging as a new tool for PCa detection and has shown promising results towards lesion identification. Both imaging procedures suffer from intra- and inter- observer variability in PCa detection. Computer-aided diagnosis (CAD) systems have been developed as a solution to mitigate observer variability and have shown to boost diagnostic accuracy. There are currently no studies published that assessed the benefit of incorporating PSMA PET imaging and mp-MRI into a CAD system for PCa detection. We compared the accuracy of CAD models trained and tested on features from mp-MRI+PSMA PET, mp-MRI and PSMA PET by training on 1-10 features chosen from three feature selection methods for 10 different classifiers for each of the three experiments. We found that models trained on mp-MRI provided lower overall error and greater specificity, and models trained on mp-MRI+PSMA PET and PSMA PET provided greater sensitivity to lesions in the central gland, which is a known area of difficulty for mp-MRI. Further validation using a larger dataset is required to prove the added benefit of PSMA PET imaging as a second modality to PCa CAD systems. Once fully validated, these results will demonstrate the added benefit of incorporating PSMA PET imaging into CAD models towards PCa detection.

Periprostatic fat composition on T2-weighted (T2w) MRI has been shown to be associated with aggressive prostate cancer and may influence extraprostatic extension (EPE). In this study, we interrogate the periprostatic fat (PPF) region adjacent to cancer lesion on prostate T2w MRI. Patients with pathologic stage ≥ pT3a are considered to experience EPE (EPE+) and those with stage ≤ T2c are without EPE (EPE-) post radical prostatectomy (RP). We use a cohort of N = 45 prostate cancer patients retrospectively acquired from a single institution who underwent 3T multi-parametric MRI prior to RP. Radiomic features including 1st and 2nd order statistics, Haralick, Gabor, CoLlAGe features are extracted from a region of interest (ROI) in the PPF on pre-surgical T2w MRI delineated by an experienced radiologist. Haralick, gradient and CoLlAGe features were observed to be significantly different (p<0.05) in PPF ROIs between EPE+ and EPE- and were significantly over expressed in EPE+ patients compared to EPE- patients, suggesting a higher heterogeneity within the PPF region for EPE+ patients. These features were used to train machine learning classifiers using a 3-fold cross validation approach in conjunction with feature selection methods to predict EPE. The best classification performance was obtained with Support Vector Machine (SVM) classifiers resulting in an AUC = 0.88 (±0.04). On univariable and multivariable analysis, we observed that radiomic classifier predictions resulted in significant separation between EPE+ and EPE- while none of the routinely used clinical parameters including prostate specific antigen (PSA), Gleason Grade Groups (GGG), age, race and prostate imaging reporting and data system (PI-RADS v2) scores showed significant differences. Our results suggest that radiomic features may quantify the underlying heterogeneity in periprostatic fat and predict patients who are likely to experience extraprostatic extension of disease post RP.

Liver segmentation is a prerequisite for measuring hepatic volume in liver transplantation, modeling of the liver anatomy in hepatic surgery planning, and contouring in radiotherapy treatment planning. The main challenges of liver segmentation are the appearance similarity of liver and surrounding stomach, heart, and spleen in 2D images and are the large shape variations of liver in 3D volume. Therefore, we propose a deep learning-based liver segmentation method by using global context of three orthogonal planes to localize the liver in whole abdomen and by using local context of targeted liver bounding volume and high-score shape prior to delineate the liver without leakage to the surrounding structures. To localize the liver within the whole abdomen and exclude outliers through the global context, three 2D segmentation networks are learned on each axial, coronal, and sagittal planes. To consider the shape information obtained from the 2D segmentation network in the next 3D segmentation network, the high-score shape prior is generated by a weighted fusion of three score maps. To correct the fine details of the liver in the targeted liver bounding volume and to be less affected by shape variation, the 3D segmentation network is learned based on 3D U-Net with highscore shape prior. Experimental results show that the DSC of the proposed segmentation network with high-score shape prior (LiverNet-WS) was 94.3%, which is 5.4% higher than LiverNet without high-score shape prior. The proposed method accurately localized the liver within the whole abdomen by using global contexts of three orthogonal planes. Moreover, segmentation accuracy improved fine details considering local context and avoided over-segmentation considering high-score shape prior.

The use of quantitative radiomic features of MRI to predict the aggressiveness of prostate cancer has attracted increasing amounts of attention due to its potential as a non-invasive biomarker for prostate cancer. Although clinical studies have shown that apparent diffusion coefficient (ADC) values correlate with the aggressiveness of prostate cancer, most studies on radiomic features have been performed only with T2-weighted MR (T2wMR). Therefore, we investigate the usefulness of radiomic features of T2wMR and ADC to predict prostate cancer aggressiveness. To define the prostate cancer region of T2wMR based on ground truth pathology, a radiologist manually segmented prostate cancer referring to a fusion result of registration of histopathology image and T2wMR. The prostate cancer region of the ADC is then defined as the same region as the T2wMR through registration of the ADC on the T2wMR. To extract radiomic features to predict prostate cancer aggressiveness, total 68 features are calculated for each region of T2wMR and ADC. To predict the aggressiveness of prostate cancer, a random forest classifier is trained for each region in T2wMR and ADC. The prostate cancer regions were categorized as G1 (GS <= 3+4) and G2 (GS <= 4+3). As results, the prediction model of ADC was provided high performance than that of T2wMR, and the area under the curves of the receiver operating characteristic (ROC) were 0.70 and 0.74 in T2wMR and ADC. Experiment results showed that the possibility of determining the aggressiveness of prostate cancer through the quantitative radiomic features of T2wMR and ADC.

Segmentation of the renal parenchyma which consists of renal cortex and medulla responsible for the renal function is necessary to evaluate contralateral renal hypertrophy and to predict renal function after renal partial nephrectomy (RPN). However, segmentation of the renal parenchyma is difficult due to the large variations in the shape of the kidney among patients and similar intensities with nearby organs such as the liver, spleen, vessels and the collecting system. Therefore, we propose an automatic renal parenchyma segmentation based on 2D and 3D deep convolutional neural networks with similar atlas selection and transformation in abdominal CT images. First, kidney is localized using 2D segmentation networks based on U-net on the axial, coronal, and sagittal planes and combining through a majority voting. Second, similar atlases to test volume in the training set are selected by calculating mutual information between the kidney test volume and the training volume, and then transformed to the test volume using volume-based affine registration. Finally, renal parenchyma is segmented using 3D segmentation network based on U-net. The average dice similarity coefficient of renal parenchyma was 94.59%, showed better results of 10.41% and 0.80% compared to the segmentation method using fusion of three 2D segmentation networks results and combined 2D and 3D segmentation networks, respectively. Our method can be used to assess the contralateral renal hypertrophy and to predict the renal function by measuring the volume change of the renal parenchyma, and can establish the basis for treatment after RPN.

We are developing a deep learning based U-Net (U-DL) model for bladder wall segmentation in CT urography (CTU) as a component of a computer-assisted pipeline for bladder cancer detection and treatment response assessment. This task is challenging due to variations in size and shape of the wall among cases, low contrast between the bladder wall and surrounding structures, and some walls being extremely thin and occasionally invisible compared to the overall size of the bladder. Our previous method used a deep-learning convolution neural network and level sets (DCNN-LS) within a user-input bounding box. In the current study, we propose two new methods for bladder wall segmentation: 1) the outer and inner bladder wall contour masks are generated to train two different U-DLs and the segmented bladder regions are subtracted to obtain the final bladder wall; 2) a combined wall mask for the bladder wall is generated by subtracting the hand-outlined bladder inner and outer contour masks, and a single U-DL is trained to segment the bladder wall. The new methods use only U-Net without level-set post-processing. Hand-segmented contours from 67 training and 14 validation cases were used for this study. The combined wall mask training method in particular shows promise in improving both accuracy and reducing pipeline complexity.

Prediction of survival period for patients with hepatocellular carcinoma (HCC) provides important information for treatment planning such as radiotherapy. However, the task is known to be challenging due to the similarity of tumor imaging characteristics from patients with different survival periods. In this paper, we propose a survival prediction method using deep learning and radiomic features from CT images with support vector machine (SVM) regression. First, to extract the deep features, the convolutional neural network (CNN) is trained for the task of classifying the patients for 24-month survival. Second, the radiomic features including texture and shape are extracted from the patient images. After concatenating the radiomic features and the deep features, the SVM regressor is trained to predict the survival period of the patients. The experiment was performed on the CT scans of 171 HCC patients with 5-fold cross validation. In the experiments, the proposed method showed an accuracy of 86.5%, a root-mean-squared-error (RMSE) of 11.6, and a Spearman rank coefficient of 0.11. In comparisons with the deep feature-only- and radiomic feature-only regression results, the proposed method showed improved accuracy and RMSE than both, but lower rank coefficient than the radiomic feature-only regression. It can be observed that (1) the deep learning of CT images has a promising potential for predicting the survival period of HCC patients, and (2) the radiomic feature analysis provides useful information to strengthen the power of deep learning-based survival prediction.

The goal of this study was to assess the performance of a deep convolutional neural network trained with a limited number of cases for the detection of distal tibial fractures in children.We identified 516 ankle and leg radiographic exams in children (6.4±4.4 (mean±s.d) years) containing 2118 individual images. Radiographs with implants, casts, advanced healing, and including other pathology such as intra-osseous bone lesions were excluded. After these exclusions, 490 positive distal tibial fracture radiographs were identified and a matching number of normal radiographs ware selected, creating a dataset of 980 radiographs. These were sequentially partitioned, in 10 permutations, into a training set (784 radiographs), validation set (98 radiographs) and a test set (98 radiographs), with equal numbers of fracture and normal radiographs in each set. A modified transfer learning network based on the Xception-V3 architecture with additional fully convoluted reasoning layers was trained against each of the subsets. The best performing trained network successfully recognized 47 of 49 fractures and 47 of 49 normal exams (95.9% accuracy). The best three performing networks were all very similar, with accuracy of 95.6 ± 0.6%. In no instances were normal physes or normal developmental epiphyseal or medial malleolus fragmentation misclassified as a fracture. Using a pre-trained deep convolutional neural network adapted to identifying or excluding distal tibial fractures in children using only a small number of radiographs is feasible and highly accurate without the need for the large training sets typically needed for network training.

Evaluating the severity of knee osteoarthritis (OA) accounts for significant plain film workload and is a crucial component of knee radiograph interpretation, which informs surgical decision-making for costly and invasive procedures such as knee replacement. The Kellgren-Lawrence (KL) grading scale systematically and quantitatively assesses the severity of knee OA but is associated with notable inter-reader variability. In this study, we propose a deep learning method for the assessment of joint space narrowing (JSN) in the knee, which is an essential part of determining the KL grade. To determine the extent of JSN, we analyzed 99 knee radiographs to calculate the distance between the femur and tibia. Our algorithm's measurements of JSN and KL grade correlated well other radiologists' assessments. The average distance (in pixels) between the femur and tibia bones as measured by our algorithm was 9.60 for KL=0, 7.60 for KL=1, 6.89 for KL=2, 3.75 for KL=3, 1.25 for KL=4. Additionally, we used 100 manually annotated knee radiographs to train the algorithm to segment the femur and tibia bones. When evaluated on an independent set of 20 knee radiographs, the algorithm demonstrated a Dice coefficient of 96.59%. An algorithm for measurement of JSN and KL grades may play a significant role in automatically, reliably, and passively evaluating knee OA severity and influence and surgical decision-making and treatment pathways.

Purpose: Detecting knee orientation automatically from scout scans with high speed and accuracy is essential to a successful workflow of MR knee imaging. Although traditional methods of image processing such as rigid image registration and object detection are potentially available solutions, they are sensitive to image noise such as missing features due to the metal implants and anatomical variability in knee size and tissue composition. Method: In this study, a segmentation-based approach was proposed to calculate a 3-D transformation matrix that defined 3-D knee orientation using low-res MR scout scans. Specifically, 3-D U-net was used to segment a plane that was parallel to the knee meniscus plane and reconstruct the plane norm as one of the vectors (v₁) needed for a 3d transformation matrix. A separate model of 3-D U-net was then trained to segment another plane that was perpendicular to the meniscus and reconstruct the plane norm as v₂. A linear 3-D transformation matrix was then obtained for each patient case in 14 testing subjects that were initially manually rotated in small (group S) and large (group L) degrees for training. Angle corrected images were also visually compared against their corresponding ground truth. Results: The average of v₁ and v₂ error in group S were 5.62° and 5.12° , respectively, whereas the error average of these two vectors were 6.65° and 8.25° , respectively for group L. The standard deviation for v₁ and v₂ in group S and L were 2.51° , 2.84° , 5.65° , and 7.65° , respectively. The Dice similarity coefficient (DSC) of reconstructed v₁ and v₂ planes were 0.78, 0.70, 0.71, and 0.65 for group S and L. The qualitative assessment further showed consistent knee representation after correction for knees with heavy distortion and fatty tissue. Conclusion: Initial results suggest that our approach has the potential to automatically correct for small knee rotations commonly seen in clinical setting and is robust even under stress test for knees with anatomical structures (e.g. fatty tissue) that were even absent in the training data set and that appear heavily distorted.

Lesion-specific myocardial ischemia is a common heart disorder and a significant cause of cardiovascular morbidity and mortality. It alters left ventricular myocardial thickness progressively. Clinical decision-making is based on Fractional flow reserve (FFR), which is invasive and may prolong the surgery time and with extra radiation exposure. Although coronary computed tomography angiogram (CCTA) has high accuracy and negative predictive value (NPV) in the evaluation of coronary artery disease (CAD), it has low specificity in the diagnosis of lesion-specific myocardial ischemia. We propose a learning method for the assessment of lesion-specific myocardial ischemia using noninvasive CCTA and radiomics study. Sixty patients with suspected or known to have CAD were enrolled. The left ventricular myocardial on the CCTA was manually segmented. One hundred radiomic features of left ventricular myocardial were extracted. The most informative and non-redundant features were selected to train a Support Vector Machine (SVM) is to differentiate lesion-specific myocardial ischemia and without lesion-specific myocardial ischemia (normal). Analysis of the predictions showed that the reported method consistently predicted lesion-specific myocardial ischemia with the accuracy of 0.8550 ± 0.0333 and area under the receiver operating characteristic curve (AUC) 0.8952 ± 0.0370. This study shows that LVMradiomic features derived from CCTA data can be used to classify lesion specific myocardial ischemia. The radiomic features of left ventricular myocardial from CCTA could be a useful tool for determining lesion specific myocardial ischemia.

This study demonstrates that a variant of a Siamese neural network architecture is more effective at classifying highdimensional radiomic features (extracted from T2 MRI images) than traditional models, such as a Support Vector Machine or Discriminant Analysis. Ninety-nine female patients, between the ages of 20 and 48, were imaged with T2 MRI. Using biopsy pathology, the patients were separated into two groups: those with breast cancer (N=55) and those with GLM (N=44). Lesions were segmented by a trained radiologist and the ROIs were used for radiomic feature extraction. The radiomic features include 536 published features from Aerts et al., along with 20 features recurrent quantification analysis features. A Student T-Test was used to select features found to be statistically significant between the two patient groups. These features were then used to train a Siamese neural network. The label given to test features was the label of whichever class the test features with the highest percentile similarity within the training group. Within the two highest-dimensional feature sets, the Siamese network produced an AUC of 0.853 and 0.894, respectively. This is compared to best non-Siamese model, Discriminant Analysis, which produced an AUC of 0.823 and 0.836 for the two respective feature sets. However, when it came to the lower-dimensional recurrent features and the top-20 most significant features from Aerts et al., the Siamese network performed on-par or worse than the competing models. The proposed Siamese neural network architecture can outperform competing other models in high-dimensional, low-sample size spaces with regards to tabular data.

Medical imaging devices, such as X-ray machines, inherently produce images that suffer from visual noise. Our objectives were to (i.) determine the effect of image denoising on a medical image classification task, and (ii.) determine if there exists a correlation between image denoising performance and medical image classification performance. We performed the medical image classification task on chest X-rays using the DenseNet-121 convolutional neural network (CNN) and used the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) metrics as the image denoising performance measures. We first found that different denoising methods can make a statistically significant difference in classification performance for select labels. We also found that denoising methods affect fine-tuned models more than randomly-initialized models and that fine-tuned models have significantly higher and more uniform performance than randomly-initialized models. Lastly, we found that there is no significant correlation between PSNR and SSIM values and classification performance for our task.

Computer aided diagnosis systems are used to assist radiologists in their decision making. The sensitivity of these systems is hindered by the complexity of the structures inside the lungs. Several systems and methods have been proposed to detect and classify lung nodules, but all of them have their strengths and weaknesses. One way to overcome the weaknesses is to combine multiple systems. Systems based on handcrafted features capture a limited set of characteristics from the image, while deep learning based classifiers can deal with a wider range of structures. In this work, several ways to combine a handcrafted feature based classifier with four convolutional neural network are explored. The systems were combined merging the probabilities assigned to the detections in several ways. Support-vector machine, multilayer perceptron and random forest classifiers were used to combine the selected classifiers. The LUNA16 Challenge was used to evaluate the performance of the resulting hybrid systems. In all cases, the hybrid systems outperformed the individual systems. Although the average of sensitivities are similar for most of the combinations, the best hybrid system achieves a gain of 35 extra nodules at 4 FP per scan.

We propose a new standardization method to mitigate the undesired effects of factors such as body weight, duration of radiotracer uptake period, partial volume effects, etc., that impede accurate quantitative analysis of PET images. We introduce new metrics for evaluation and comparison among different standardization methods. The proposed standardization method can improve disease quantification from PET images, as PET is one of the major clinical imaging tools used to evaluate patients with cancer and other disorders. It is based on the MRI standardization method proposed by Nyul et al., but modified for application to PET images. A new approach is proposed to remove the unwanted effect of the tail of image histograms by finding an optimal maximum percentile from the coefficient variations of metabolic activity of reference healthy organs. We show that after applying our standardization method, the coefficient of variation among the mean metabolic activities of healthy organs decreases compared to that obtained from non-standardized images.

This paper presents segmentation of multiple organ regions from non-contrast CT volume based on deep learning. Also, we report usefulness of fine-tuning using a small number of training data for multi-organ regions segmentation. In medical image analysis system, it is vital to recognize patient specific anatomical structures in medical images such as CT volumes. We have studied on a multi-organ regions segmentation method from contrast-enhanced abdominal CT volume using 3D U-Net. Since non-contrast CT volumes are also usually used in the medical field, segmentation of multi-organ regions from non-contrast CT volume is also important for the medical image analysis system. In this study, we extract multi-organ regions from non-contrast CT volume using 3D U-Net and a small number of training data. We perform fine-tuning from a pre-trained model obtained from the previous studies. The pre-trained 3D U-Net model is trained by a large number of contrast enhanced CT volumes. Then, fine-tuning is performed using a small number of non-contrast CT volumes. The experimental results showed that the fine-tuned 3D U-Net model could extract multi-organ regions from non-contrast CT volume. The proposed training scheme using fine-tuning is useful for segmenting multi-organ regions using a small number of training data.

In current clinical practice, paired computed tomography (CT) providing electron density information for dose calculation and magnetic resonance imaging (MRI) providing molecular information for GTV delineation are acquired during radiation therapy planning of head cancer. Aimed to reduce repeatedly scanning procedures, we developed a patch-based deep learning approach to generate synthetic CT from paired MRI of the same patient. In this approach, 2D slices of MRI and CT would be divided into several overlap patches and sent to cycle-consistent generative adversarial network (CycleGAN) for training with a combination of multiple loss functions. For comparison, we also applied CycleGAN and pix2pix model using whole 2D slices as input. With IRB approval, a total number of 2542 paired MRI and CT images were collected in the experiment. Mean absolute error (MAE) and peak signal to noise ratio (PSNR) were used as evaluation metrics. The result showed that our proposed model performed best on both whole brain areas. We also provided the difference map between synthetic and real CT to give a visual evaluation of our proposed model.

Heterogeneous multi-modal medical imaging data need to be properly handled in classification. Currently, generating models using multi-modal imaging data has become a common practice and greatly benefits the brain disorder diagnosis, which also holds considerable clinical potential. Although the majority of classification studies focus on using features from single modality, there is substantial evidence suggesting that classification based on multi-modal features is on upward trend. Hence, effective integration of heterogeneous data is in urgent demand. Here, we proposed a multi-kernel SVM for schizophrenia classification with nested 10-fold cross validation, which could integrate multi-modal data using the subspace similarity of the decomposed components in each MRI modality. To validate the effectiveness of the proposed method, we performed experiments on two independent datasets with three different modalities to classify schizophrenia patients and healthy controls. Specifically, multi-modal fusion method was first applied on preprocessed fMRI, DTI and sMRI data to generate components that could be used for classification. Then multi-kernel SVM models were trained on the selected component features using subspace similarity measures, and were tested on independent validation data across sites. The results on both datasets demonstrated that our method achieved accuracies of 87.6% and 79.9% separately on two datasets when combining all three modalities, which outperformed alternative methods and might provide potential biomarkers for cross-site classification and co-varying components among different modalities.

Computer-Aided Diagnosis systems are required to extract suitable information about retina and its changes. In particular, identifying objects of interest such as lesions and anatomical structures from the retinal images is a challenging and iterative process that is doable by image processing approaches. Microaneurysm (MAs) are one set of these changes that caused by diabetic retinopathy (DR). In fact, MAs detection is the main step for identification of DR in the retinal images analysis. The objective of this study is to apply an automated method for detection of MAs and compare the results of detection with and without vessel segmentation and masking either in the normal or abnormal image. The steps for the detection and segmentation are as follows. At the first step, we did preprocessing, by using top-hat transformation. Our main processing was included applying Radon transform, to segment the vessels and masking them. At last, we did MAs detection step using combination of Laplacian-of-Gaussian and Convolutional Neural Networks. To evaluate the accuracy of our proposed method, we compare the output of our proposed method with the ground truth that collected by ophthalmologists. With vessel segmentation, our algorithm found sensitivity of more than 85% in detection of MAs with 11 false positive rate per image for 100 color images in a local retinal database and 20 images of a public dataset (DRIVE). Also without vessel segmentation, our automated algorithm finds sensitivity of about 90% in detection of MAs with 73 false positives per image for all 120 images of two databases. In conclusion, with vessel segmentation we have acceptable sensitivity and specificity, as a necessary step in some diagnostic algorithm for retinal pathology.

Early detection of glaucoma is important for slowing disease progression and preventing total vision loss. The diagnosis of glaucoma is closely related to the shape of the optic disc and cup (cup-disc) and whether there is a defect in the retinal nerve fiber layer (RNFL). In previous studies, it was common to predict glaucoma by analyzing changes in cup-to-disc ratio, or to directly classify fundus images for glaucoma using a deep learning classification model. This paper proposes a method for diagnosing glaucoma by combining the cup-disc shape information and retinal nerve fiber layer defect (RNFLD) information. We use a fully convolutional neural network that based on a multi-scale attention mechanism (AM-CNN） to identify cup-disc morphology and RNFLD regions, further use previous methods and image processing methods to extract features in these two spaces. Finally, we use the SVM method in machine learning to classify the sample for glaucoma based on the features fusion of the two spaces. Specifically, we first establish a small database with both the cup-disc annotation, retinal nerve fiber layer defect annotation and glaucoma diagnosis results, which includes 735 fundus images labeled with either positive glaucoma (356) or negative glaucoma (379). Then, a semantic segmentation model based on attention is designed. By adding attention to the context information of the model, a more accurate segmentation image is obtained, not only has a good effect on the segmentation of the cup-disc, but also has a significant effect on the recognition of RNFLDs. Finally, the four-dimensional features were extracted from the cup-disc segmentation map by the previous method, and the four-dimensional features such as distance and area were extracted from the retinal nerve fiber layer segmentation map. Combine the two kinds of features using SVM algorithm to establish a classification model for glaucoma classification. The experiment results show that adding the attention module to the decoder can improve the effect of segmentation tasks for more complex problems and the classification model fusion cup-disc shape and RNFLD information significantly advances glaucoma detection.

Thyroid cancer (TC) is a prevalent malignancy with a high predicated new case number and estimated death in 2019. Although four to seven percent of the adult population has a palpable thyroid nodule, however only one of twenty clinically identified TNs is malignant. Imaging modalities, including US, CT, and magnetic resonance (MRI), have been widely used for thyroid nodule evaluation, but the reliability is low. We propose a learning method for the classification of thyroid using thyroid non-enhanced thyroid computed tomography and radiomics study. Ninety-two patients with suspected or known to have abnormal thyroid nodules in their thyroid were enrolled. The thyroid on the non-enhanced thyroid CT was manually segmented. One hundred radiomic features of the thyroid were extracted. The most informative and nonredundant features were selected to train a Support Vector Machine (SVM) to differentiate benign thyroid and malignant thyroid (with malignant TNs). Analysis of the predictions showed that the reported method has accuracy 0.8185 ± 0.0366 and area under the receiver operating characteristic curve (AUC) 0.8376 ± 0.0343. This study shows that thyroidradiomic features derived from non-enhanced thyroid CT data can be used to classify benign vs. malignant thyroid. The radiomic features of thyroid from non-enhanced thyroid CT could be a useful tool for determining benign or malignant thyroid.

Ultrasound imaging has become a routine means of diagnosing atherosclerosis. The classification of carotid ultrasound images and detection for the plaques automatically are critical for the diagnosis of atherosclerosis, which has important clinical significance for further analysis of plaque vulnerability and risk assessment of cardiovascular and cerebrovascular events. At present, manual measurement is used for the classification, which has obvious disadvantages such as inaccurate measurement and operator variability. In this paper, we proposed an automatic classification method based on convolutional neural network (CNN) for the carotid ultrasound images from different research institutions and ultrasound machines. 820 and 830 carotid ultrasound images from Zhongnan Hospital of Wuhan University and Robarts Research Institute of Canada were used for the classification of normal, thickened vessel wall and plaque images. To solve the problem of uneven image quality and size, we used six different image normalization schemes. Furthermore, we designed five CNNs with slightly different structures and compared them with texture-based features classifications. The CNN results showed significant superiority in classification performance with total accuracy of 90.30% and recall rate of 89.70%, indicating the automatic classification of carotid ultrasound images based on CNN is potentially useful for clinical application in the diagnosis of carotid atherosclerosis.

Background: Endodontic maladies incur significant costs to the military and are difficult to diagnose. Early identification and treatment of endodontic conditions are critical. Our research focused on designing a novel, vitality-based photoplethysmography imaging (PPGI) system capable of determining the pulse frequency within the dental pulp, allowing for direct, accurate, real-time visualization of tooth vitality. Objective: The objective of this study was to verify the accuracy of a developed video stabilization and digital signal processing algorithm – Pulp Assessment by Local Observation (PABLO). Methodology: MATLAB was used to create the PABLO video analysis algorithm to determine the pulse frequency of the dental pulp from PPGI signals. In order to verify the accuracy of this algorithm, four trials were conducted to test the algorithm under various parameters. The control group was simply a measurement of the algorithm’s ability to detect a pulse within an ex vivo tooth model. Following this, separate trials were conducted to test the effects of a simulated gumline, adjacent teeth, and a video stabilization protocol on the algorithm’s accuracy. Results: Video recordings from the ex vivo model were analyzed using the PABLO algorithm to determine its accuracy in detecting the pulse frequency of the pulp. Results of the analysis showed that the algorithm had a pulse detection sensitivity above 90% and a percent error less than 11% in all trials. In the control case, results showed a sensitivity of 93% and a pulse detection error of 7.3%, indicating that this algorithm has promise as a diagnostic tool for clinicians.

Radiation treatment for head-and-neck (HN) cancers requires accurate treatment planning based on 3D patient models derived from CT images. In clinical practice, the treatment volumes and organs-at-risk (OARs) are manually contoured by experienced physicians. This tedious and time-consuming procedure limits clinical workflow and resources. In this work, we propose to use a 3D Faster R-CNN to automatically detect the location of head and neck organs, then apply a U-Net to segment the multi-organ contours, called U-RCNN. The mean Dice similarity coefficient (DSC) of esophagus, larynx, mandible, oral cavity, left parotid, right parotid, pharynx and spinal cord were ranging from 79% to 89%, which demonstrated the segmentation accuracy of the proposed U-RCNN method. This segmentation technique could be a useful tool to facilitate routine clinical workflow in H&N radiotherapy.

The thyroid gland is a butterfly-shaped organ and belongs to the endocrine system. The abnormality in shape and volume of thyroid can reveal the occurrence of various diseases. Ultrasound (US) imaging is currently the most popular diagnostic tool for diagnosing thyroid diseases. However, most physicians would still make decisions depending on computed tomography (CT) because of its excellent resolution to show more details of the thyroid and its surroundings. The thyroid CT imaging before surgery is important because it can assist in determining the anatomical distribution of a lesion and its involvement in adjacent organs or tissues. However, precise segmentation of the thyroid relies heavily on the experience of the physician and is very time-consuming. In this work, we propose to use a 3D deep attention U-Net method to segment the thyroid from CT image automatically. The quantitative evaluation of the segmentation performance of the proposed method, we calculated the Dice similarity coefficient (DSC), sensitivity, specificity, and mean surface distance (MSD) indices between the ground truth and automatic segmentation We demonstrated high accuracy and robustness of the proposed deep-learning-based segmentation method visually and quantitatively. The resultant DSC, precision, and recall were 85% ± 6%, 86% ± 5% and 90% ± 5%, respectively.