This PDF file contains the front matter associated with SPIE Proceedings Volume 11511, including the Title Page, Copyright information, and Table of Contents.

Conventional microscopy focusing methods perform a time consuming sweep through the Z-axis in order to estimate the focal plane. As an alternative, we developed a deep learning model that predicts in one shot the distance offset to the focal plane from any initial position using an input of only two images taken a set distance apart. The difference of these two images is processed through a regression CNN model, which was trained to learn a direct mapping between the amount of defocus aberration and the distance from the focal plane. A training dataset was acquired from a semiconductor sample at different surface locations on the sample and at different distances from focus. The ground truth focal plane was determined using a parabolic autofocus algorithm with the Tenengrad scoring metric. The CNN model was tested on bare semiconductor sample using the projected shape of the F-stop. The model was able to determine the in-focus position with high reliability, and was also significantly faster than conventional methods that rely on classical computer vision. Furthermore, the rare cases where our algorithm does not find the focal plane can be detected, and a fine-focus algorithm can be applied to correct the result. With the collection of sufficient training data, our deep learning focusing model provides a significantly faster alternative to conventional focusing methods.

The video captioning problem consists of describing a short video clip with natural language. Existing solutions tend to rely on extracting features from frames or sets of frames with pretrained and fixed Convolutional Neural Networks (CNNs). Traditionally, the CNNs are pretrained on the ImageNet-1K (IN1K) classification task. The features are then fed into a sequence-to-sequence model to produce the text description output. In this paper, we propose using Facebook's ResNeXt Weakly Supervised Learning (WSL) CNNs as fixed feature extractors for video captioning. These CNNs are trained on billion-scale weakly supervised datasets constructed from Instagram image-hashtag pairs and then fine-tuned on IN1K. Whereas previous works use complicated architectures or multimodal features, we demonstrate state-of-the-art performance on the Microsoft Video Description (MSVD) dataset and competitive results on the Microsoft Research-Video to Text (MSR-VTT) dataset using only the frame-level features from the new CNNs and a basic Transformer as a sequence-to-sequence model. Moreover, our results validate that CNNs pretrained with weak supervision can effectively transfer to tasks other than classification. Finally, we present results for a number of IN1K feature extractors and discuss the relationship between IN1K accuracy and video captioning performance. Code will be made available at https://github. com/flauted/OpenNMT-py.

Low signal to noise ratio (SNR) conditions degrade microscopy imaging quality, which complicates downstream post-processing and analysis. One conventional method to improve SNR by reducing noise is to average a large number of sequentially acquired images. However, this results in increased data acquisition time and reduced throughput. Longer exposures are also problematic for light-sensitive samples. We developed an alternative method, using a deep learning model based on the U-Net architecture that significantly reduces the number of images required to obtain exceptionally high SNR. Our model takes 5 noisy grayscale images as an input to generates a denoised image as an output. The model is trained on synthetically generated examples with added noise and fine tuned on real data. We demonstrate fast and robust denoising for images of fluorescent samples. Our method is capable of enhancing features while minimizing sample degradation from prolonged light exposure.

Object detection is a central theme for many Artificial Intelligence (AI) applications such as autonomous vehicles, surveillance etc. The algorithms providing this capability rely on training data being available in corresponding sensor mode. Having co-located data from multiple sensor modes enhances the detection confidence, but the availability of training data in desired sensor mode is not always readily available, which slows down progress. In this paper, we investigate the ability to translate images from one sensor mode to another, on a single fixed camera dataset, using conditional Generative Adversarial Network (cGAN). Specifically, images are transferred from Electro-Optical (EO) to Infra-Red (IR) images and vice-versa using cGAN models, which are generative models that learn the data distribution in a minimax game setting. To investigate the usability of such transferred images, we apply object detection algorithm on ground truth and transferred images and compare their performance. The results indicate that transferred images match closely to real images and object detection has good performance on transferred images, especially when the object size is large.

We developed a deep learning system for inferring detailed retinal blood flow in structural optical coherence tomography (OCT) images. Motivations include enhanced diagnosis of retinal diseases and reducing time and cost of acquiring OCT angiography (OCTA) images. Using OCTA images as ground truth, we trained a conditional generative adversarial network (cGAN) to predict capillaries from OCT cross-sections. The inferred cross-sections and resulting en-face blood flow map images show comparable detail of small capillaries to the target images. The results demonstrate the potential of cGANs in inferring blood flow maps from new and existing retinal OCT datasets.

Optical coherence tomography (OCT) and retinal fundus images are widely used for detecting retinal pathology. In particular, these images are used by deep learning methods for classification of retinal disease. The main hurdle for widespread deployment of AI-based decision making in healthcare is a lack of interpretability of the cutting-edge deep learning-based methods. Conventionally, decision making by deep learning methods is considered to be a black box. Recently, there is a focus on developing techniques for explaining the decisions taken by deep neural networks, i.e. Explainable AI (XAI) to improve their acceptability for medical applications. In this study, a framework for interpreting the decision making of a deep learning network for retinal OCT image classification is proposed. An Inception-v3 based model was trained to detect choroidal neovascularization (CNV), diabetic macular edema (DME) and drusen from a dataset of over 80,000 OCT images. We visualized and compared various interpretability methods for the three disease classes. The attributions from various approaches are compared and discussed with respect to clinical significance. Results showed a successful attribution of the specific pathological regions of the OCT that are responsible for a given condition in the absence of any pixel-level annotations.

Deep Learning (DL) shows the state-of-art performance in detection, classification and segmentation tasks compared to existing methods. As computational pathology becomes a very promising area of research, it is important to determine which color space provides better results for pathological image segmentation tasks. In this paper, we have considered six different color spaces, namely RGB, LAB, CIE, YCrCb, HSV and HSL for nuclei segmentation tasks where the Recurrent Residual based U-Net (R2U-Net) model is applied. The ISMI-2017 publicly available dataset is used for evaluating the model in this implementation. The Lab color space shows an F1-score of 0.9365, which is the highest segmentation performance when compared to the other color spaces. The Lab color space model shows around 0.38% better performance compared to the RGB color space for nuclei segmentation tasks. This investigation will provide a clear guidance in advance of pathological image segmentation and analysis tasks.

At the present time, medical endoscopy is the main procedure for exploring the internal cavities of the human body. The development of methods for integrating image processing and endoscopic visualization allows improving image quality and more accurately identify cancerous abnormalities. This paper proposed a new approach for the automatic detection of gastric polyp in an endoscopic video using a texture descriptor and machine learning approach. The proposed approach uses a Gabor feature and constructing a texture descriptor of dense micro-block difference (DMD). The approach improves the performance of the automatic detection of polyp in comparison with state-of-the-art methods.

State-of-the-art deep learning models have demonstrated success in classifying facial expressions of adults by relying on large datasets of labeled images. Unfortunately, there is a scarcity of labeled images of child expressions. Deep learning models trained on adult data do not generalize well on child data due to the domain shift caused by morphological differences in their faces. Recent deep domain adaptation approaches align the data distribution of a target domain with the source domain using a few target domain samples. We propose that the domain adaptation may be improved by incorporating steps of deep transfer learning, such as initialization with pre-trained source weights and freezing early layers of the model. The knowledge of a few labeled examples from the child data (target domain) is incorporated into the adult data distribution (source domain) using a contrastive semantic alignment (CSA) loss. This work combines deep transfer learning and domain adaptation approaches to generate seven expression labels (‘happy’, ‘sad’, ‘anger’, ‘fear’, ‘surprise’, ‘disgust’, plus ‘neutral’) for facial images of children in reference to the source domain, adult facial expressions, using 10 or fewer samples per expression. Our hybrid approach outperforms the transfer learning model by 12% on mean accuracy using only 10 samples per expression class.

Reducing Magnetic Resonance (MR) image acquisition time can potentially help to reduce cost and make the MRI based assessment more accessible for the patients care. The GRAPPA and Compressive Sensing (CS) have been applied for image reconstruction from the data where original sampled at a rate inferior to the Nyquist-Shannon sample theory. However, most of the existing techniques are iterative and time-consuming, as a result these methods are not suitable for the real-time applications for MRI reconstruction problem. Nowadays, the Deep Learning (DL) based methods have been showing state-of-the-art performance for different image acquisition and reconstruction tasks including MRI reconstruction. The DL based solution is more robust, generic, transferable, and faster. Hence, the deep learning-based solution can be used to deploy a real-time and faster MRI reconstruction system. In this work, we applied the NABLA-N deep learning model for the MRI reconstruction in both image-domain and frequency-domain (k-space). For evaluating the model, the experiments have conducted on publicly available MRI reconstruction challenge dataset. The results demonstrate that the NABLA-N Net (∇^N-Net) model is capable to reconstruct high-quality MR images and show that kspace reconstruction model is better and more capable compared to image-space based reconstruction model.

Automated cell counting in in-vivo specular microscopy images is challenging, especially in situations where single-cell segmentation methods fail due to pathological conditions. This work aims to obtain reliable cell segmentation from specular microscopy images of both healthy and pathological corneas. We cast the problem of cell segmentation as a supervised multi-class segmentation problem. The goal is to learn a mapping relation between an input specular microscopy image and its labeled counterpart, indicating healthy (cells) and pathological regions (e.g., guttae). We trained a U-net model by extracting 96×96 pixel patches from corneal endothelial cell images and the corresponding manual segmentation by a physician. Encouraging results show that the proposed method can deliver reliable feature segmentation enabling more accurate cell density estimations for assessing the state of the cornea.

Blood vessel segmentation is an important step in the automated diagnosis of ophthalmic disease from retinal fundus images. The UNet is a popular encoder-decoder architecture widely used in biomedical pixel-wise seg- mentation problems. In this paper, we analyze how the UNet can be used in a more computationally efficient way. Pre-trained weights are used to initialize the network and 3 different architectures are used to compare and analyze the efficacy of the models in terms of both computational cost and performance. Three different deep architectures (VGG16, ResNet34, DenseNet121) are discussed and their efficiencies are compared for the blood vessel segmentation task. Resnet34 architecture achieved highest sensitivity of 0.849 and accuracy and specificity of 0.961, 0.9843 with number of parameters as low as 510178 compared to normal UNet with 34525168 parameters and a sensitivity of 0.756.

Neuro-degenerative diseases can break brain's common output pathways of peripheral nerves and muscles in an individual, inhibiting his ability to perform daily tasks. Brain Computer Interfaces BCI make decoding-encoding of brain signals into control instructions for external devices. This work proposes the use of stacked autoencoders and a softmax layer for classification of visual stimuli from Electrocorticographic (ECoG) signals as an input to the BCI control system. Experimental results show that the proposed method has a good classification performance (average accuracy across subjects 0.95 +/- 0.05), compared to state-of-the-art approaches as Support Vector Machines SVM. Furthermore, the proposed network architecture allows analysis of the weights learned by the classifier making it possible to obtain insights of what signal features the classifier uses to discriminate the visual stimulus.

Many datasets in important fields like healthcare and finance are often in a tabular format, where each observation is expressed as a vector of various feature values. While there exist several competitive algorithms such as random forests and gradient boosting, convolutional neural networks (CNNs) are making tremendous strides in terms of new research and applications. In order to exploit the power of convolution neural networks for these tabular datasets, we propose two vector-to-image transformations. One is a direct transformation, while the other is an indirect mechanism to first modulate the latent space of a trained generative adversarial network (GAN) with the observation vectors and then generate the images using the generator. On both simulated and real datasets, we show that CNNs trained on images based on our proposed transforms lead to better predictive performance compared to random forests and neural networks that are trained on the raw tabular datasets.

Monte Carlo based thermal radiative transfer (TRT) codes provide a flexible framework for large-scale high-energy-density simulations. Compared to their deterministic counterparts, Monte Carlo methods are easier to implement in complex geometries and higher dimensions. However, the probabilistic nature of the Monte Carlo approach and the complexity of the underlying TRT equations describing nonlinear interactions between radiation and matter imply that Monte Carlo radiation transport codes involve massive amounts of particle data and have enormous memory footprints. This study is focused on building a statistical learning framework that can be employed to `compress' IMC data into a few probability density functions characterized by a relatively small number of distribution parameters. In order to learn complex distributions describing Monte Carlo data, we employ the Expectation Maximization (EM) method and finite mixture models adapted to learn particle distributions in the energy and angular domains targeting Implicit Monte Carlo (IMC) simulations in 1-d. Specifically, we couple EM iteration to the weighted hyper-Erlang finite mixture model in order to learn parameters of the photon distributions. We couple the EM iteration to the von Mises (Circular Normal) finite mixture model in order to learn angular distributions. The learning approach proposed in this study promises accurate resampling and significant memory savings at checkpointing and restarts of the IMC and other Monte Carlo radiation transport codes.

We demonstrate using a convolutional neural network (CNN) architecture such as Resnet20 how to perform complete Laguerre-Gauss (LG) decomposition of an unknown, incoming laser beam using only intensity images. For our proof-ofconcept simulations, we use random super-positions of first 10 azimuthal LG modes with l-values from 0 to 9. For these random super-positions, both the amplitudes and phase values are random. Random white noise is added to the intensity images of these simulated fields. 80,000 such training images are used to train our Resnet20 CNN while another 20,000 images are in the test set. Prediction results on the test set show a correlation value of 98.75% showing the efficacy of using a CNN to perform spatial mode decomposition.

Accurately measuring optical aberrations is an important process for several eyecare tasks. Managing the individual variations found in human eyes plays a large role in properly defining these aberrations. A common method to measure optical aberrations uses sensors to locally capture the gradients across a grid of points. This data is used to reconstruct the wavefront resulting from light passing through the eye. In using this method of measurement, typical individual variations such as scarring or eyelashes can lead to shortcomings in the data. These shortcomings can manifest as noise or even areas of entirely missing data. The use of ANN (artificial neural networks) is one way to minimize the effects of these unpredictable deviations. In this work, ANN’s were used to determine higher order Zernike coefficients based on coordinates and gradients of the wavefront at those locations. To accomplish this, different ANN architectures were evaluated using sets of ideal inputs to establish the best performing baseline model. This baseline model was then compared with models that were trained under varying conditions created by incorporating deviations from ideal samples. By training neural networks using variable quality data, models can be created to reconstruct wavefronts and account for these conditions. This in turn, can lead to useful applications when measuring for aberrations resulting from the eye.

Automatic detection of defects in as-built parts is a challenging task due to the large number of potential manufacturing flaws that can occur. X-Ray computed tomography (CT) can produce high-quality images of the parts in a non-destructive manner. The images, however, are grayscale valued, often have artifacts and noise, and require expert interpretation to spot flaws. In order for anomaly detection to be reproducible and cost effective, an automated method is needed to find potential defects. Traditional supervised machine learning techniques fail in the high reliability parts regime due to large class imbalance: there are often many more examples of well-built parts than there are defective parts. This, coupled with the time expense of obtaining labeled data, motivates research into unsupervised techniques. In particular, we build upon the AnoGAN and f-AnoGAN work by T. Schlegl et al. and created a new architecture called PandaNet. PandaNet learns an encoding function to a latent space of defect-free components and a decoding function to reconstruct the original image. We restrict the training data to defect-free components so that the encode-decode operation cannot learn to reproduce defects well. The difference between the reconstruction and the original image highlights anomalies that can be used for defect detection. In our work with CT images, PandaNet successfully identifies cracks, voids, and high z inclusions. Beyond CT, we demonstrate PandaNet working successfully with little to no modifications on a variety of common 2-D defect datasets both in color and grayscale.

In-situ Laser Powder Bed Fusion (LPBF) sensor packages seek to enable both the commercial and Department of Defense (DoD) supply chains via process monitoring for qualification and machine feedback. An automated material identification and geometric segmentation would be valuable for LPBF process monitoring. In this paper, various segmentation approaches are presented and discussed to determine the best approach. Later, deep learning method(s) to classify the materials as either AlSi10Mg or IN718 are presented. Diverse videos (in terms of shape, size, structure, and camera angle) of both materials are captured and labeled as either AlSi10Mg (24357 frames) or IN718 (9222 frames). A given frame can contain single or multiple parts of a material. The segmentation approach is applied to extract each part and 121,036 images are obtained. The dataset is randomly split into groups of 72%, 8% and 20% for training, validation, and testing respectively. Classification performance(s) using the proposed Convolutional Neural Network (CNN) in addition to transfer learning approaches using established networks such as AlexNet, ResNet, and SqueezeNet are studied. An overall accuracy of 99.6% is obtained on a set of 24,214 test images. In addition, efficacy of the proposed classification model is demonstrated by testing the algorithm on a completely different variant (in terms of shape, size, structure, or camera angle) of either material. The class activation mapping results of these networks are presented, yielding an insight into the network’s decision, and assisting the manufacturers in their decision-making process.

We present a study that uses machine learning to solve the forward and inverse scattering problems for synthetic aperture radar (SAR). Using a training set of known reflectivities as inputs and the resulting SAR measurements as outputs, the machine learning method produces an approximation for the sensing matrix of the forward scattering problem. Conversely, employing that same training set but with the SAR measurements used as inputs and the reflectivities as outputs, the machine learning method produces an approximate inverse of the sensing matrix. This learned approximate inverse mapping allows us to solve the inverse scattering problem as it maps SAR measurements to an estimate of the reflectivity. To interpret these results, we restrict our attention to a neural network arranged as a single, fully-connected layer. By doing so, we are able to interpret and evaluate the mappings produced by machine learning in addition to the results of those mappings. Employing a training set made up of 50,000 of the CIFAR-10 dataset as the reflectivities, we simulate SAR measurements using a physical model for the sensing matrix. With this training set of reflectivities and corresponding SAR measurements, we find that the results of machine learning accurately approximate the sensing matrix and provide a better answer to the inverse scattering problem than the standard SAR inversion formula. We also test the performance of the proposed methodology on a dataset with high resolution images while training with a lower resolution data set. The results are very promising showing again a superior performance for the learned approximate inverse mapping.

This paper addresses large-scale multi-look dataset generation of multi-domain vehicles taken into account contributions of backscattered radiations from scatterers in the environment, operating contexts constraints, and atmospheric conditions. In this study, we use IRIS electromagnetic modeling and simulation system for virtualization of such test scenarios signifying the requirements. Through this approach, we initially construct and employ the physics-based CAD models of different category of vehicles in their inherent operating environments. Next, we apply physics-based remote sensing to generate realistic multimodality synthetic imagery of the test vehicles and systematically annotate them. To evaluate and verify the effectiveness of this approach, we compare the generated synthetic vehicle imagery with those images of the corresponding SAR remote sensors. In this paper, we discuss the technical aspects of this modeling and simulation and the obtained results.

Deep Learning classifiers, particularly, Convolutional Neural Networks (CNNs), have been demonstrated to be very effective in the area of SAR Automatic Target Recognition (ATR). Despite of this achievement, still there is challenges with the effective multi-feature classification of spackled SAR imagery suffering from low signal-to-noise ratio. In this paper, we address technical challenges of implementing a Multi-output Convolutional Neural Network (M-CNN) as an effective multi-feature classifier model. Primarily, we employed the IRIS Electromagnetic (IRIS EM) modeling and simulation software to generate systematic simulated SAR images from an array of physics-based CAD models of 350 target vehicles scanned from different azimuth and elevation angles. For denoising and classification of image, we proposed a step-wise retraining of a CNN via a transfer learning technique. The proposed classifier achieves higher levels of model generalization on unseen data when tested against IRIS-SAR datasets. Further, we performed multi-feature classifications of target vehicles using the M-CNN model. The proposed M-CNN shows both efficiency and effectiveness in performing multi-feature classification of test target vehicles simultaneously. In this paper, we discuss our classification results while comparing its performance against those from different comparable CNN classifier models.

We are interested in ATR on various military target types such as military tanks and mobile rocket/missile launching vehicles from as high of UAV flying altitudes as possible. However, many of our current high-flying UAV datasets do not contain these types of military targets. Therefore, a high-fidelity target insertion (HFTI) tool has been developed for testing Advanced ATR on military targets. In this paper, we present results on development of Advanced ATR using the state-of-the-art Transfer Learning and Deep Learning (DL) Convolutional Neural Networks (CNN) target detection and recognition models for the military target detection and recognition. Large labelled training datasets have been generated by the newly developed HFTI tool to train the CNN ATR. We have developed and tested two different CNN ATRs: (1) detection-based YOLOv2 model and (2) segmentation-based U-Net model. Both ATRs have achieved promising performance. A Multi-Target Tracker (MTT) has also been developed to track military vehicles that were detected and recognized by the CNN U-Net ATR. In the presentation, we will show live videos for the ATR performance with accurate multiple (moving and static) targets recognition and tracking.

Miniaturized Fourier transform infrared spectrometers serve important market needs in many applications such as gas analysis. The miniaturization comes at the cost of lower performance than benchtop instrumentation especially for the spectral resolution. Higher spectral resolution is needed for better identification of materials. This article presents a convolutional neural network (CNN) for enhancing the resolution of infra-red gas spectra for 3X resolution enhancement. The proposed network extracts a set of high-dimensional features from the input spectra and constructs high-resolution outputs by nonlinear mapping. The network was trained using synthetic noisy spectra of different resolutions of mixtures of a set of gases that are relevant to the gas industry. Results are presented for both synthetic and experimentally measured spectra.

We present an automatic classification algorithm for retinal optical coherence tomography (OCT) images based on convolution neural network (CNN). This algorithm inherently contains feature extraction and classification, thus avoiding the design feature extractor manually. Firstly, we processed the OCT images to focus on and determine the pathological area of the retinal OCT images, and to speed up the training of the network. Then we input the original images to crop them, which can effectively prevent the noise introduced in the processes of image processing and changing the pixels in the original image. Secondly, we augmented the OCT images in the source data set to obtain sufficient images, and to alleviate the impact of a relatively small number of target classification images on the model accuracy and generalization ability. Our method was introduced the random translation in image cropping and horizontal flipped to augment the OCT images. Then we applied two methods to build two data sets used to train the network, and we divided each of the data sets into a training set and a validation set. Next, we designed an efficient classification network and trained it with the two training sets respectively, to acquire the two models that can classify OCT images. The results indicate that the network trained by the augmented data can classify images more effectively. In our classification algorithm, the accuracy, the sensitivity and the specificity are 93.43%, 91.38%, and 95.88%, respectively.

The fovea is one of the most significant regions of the human eye. Accurate localization of the fovea is very helpful to downstream computer-aided diagnosis analysis in retinal imaging. Most previous researches have focused on fovea localization for the color fundus imaging. In this paper, we propose a three-step process to localize the foveal center and the foveal region for three retinal modalities: color fundus, Infrared (IR), and optical coherence tomography (OCT). From the experimental results, the proposed method achieves accurate fovea localization for multimodal imaging.

By the time of the abstract, the coronavirus COVID-19 is affecting 213 countries with more than 9.4 million cases, and more than 480 thousand deaths. In the US it has been confirmed that there are more than 2.4 million COVID-19 cases and approximately 123 thousand related deaths, according to Worldometer. The Wuhan originated COVID-19 has become a global challenge since late December 2019. As control measures lift, life and businesses start going to open, the coronavirus pandemic continues to grow. In this research, we propose a Deep Learning model that recognizes COVID-19 cases using X-ray images (DeepCOVID-19). The model would help physicians to be more confident by having a second opinion in assessing patients. An implementation of the model would help countries where there is shortage of test kits, while X-ray devices are widely available.

A typical issue with development of 3D image recognition models and systems is that 3D images often come with large volumes of data and require expensive computational cost for image reconstruction and model training. Transfer learning is a machine learning technique where a model trained on one task is reused on another related task. In this research, we propose a Transfer Learning method that allows training a 3D object recognition model with very limited training data, so requires much fewer reconstructed image slices, and help reduce the computational cost of both image reconstruction and training such models. To the best of our knowledge, this is the first report regarding Transfer Learning for 3D object recognition using integral imaging.