This guest editorial introduces the Special Section on Quality Control by Artificial Vision VI.

Burns are one of the obnoxious injuries subjecting thousands to loss of life and physical defacement each year. Both high income and Third World countries face major evaluation challenges including but not limited to inadequate workforce, poor diagnostic facilities, inefficient diagnosis and high operational cost. As such, there is need to develop an automatic machine learning algorithm to noninvasively identify skin burns. This will operate with little or no human intervention, thereby acting as an affordable substitute to human expertise. We leverage the weights of pretrained deep neural networks for image description and, subsequently, the extracted image features are fed into the support vector machine for classification. To the best of our knowledge, this is the first study that investigates black African skins. Interestingly, the proposed algorithm achieves state-of-the-art classification accuracy on both Caucasian and African datasets.

Psoriasis is an inflammatory cutaneous disease of unknown origin, characterized by the appearance of red, itchy, and scaly plaques of abnormal skin. One of the most important issues that guides treatment of psoriasis is to evaluate the degree of the illness determining the psoriasis and area severity index (PASI). Dermatologists usually use visual and tactile senses to assess lesion severity, involving a subjective judgment depending on the medical practitioner. For this purpose, we propose an image processing system based on photometric stereo acquisition technique and adapted analysis criteria to evaluate skin parameters. This diagnosis aided system gives an objective and accurate evaluation of skin erythema (redness), skin thickness, and scaling, which are the three clinical parameters, with the area of lesioned skin, that determine PASI score. Thus, we estimate the intensity of erythema from the albedo map and thickness from the three-dimensional estimation of the skin surface. Finally, the combination of color and geometry results allows identifying and quantifying the skin scaling. A validation protocol with an image database that covers all parameter variation ranges is proposed, and the results show a high correlation with dermatologist scores.

Reflectance transformation imaging (RTI) is a multilight-based imaging technique that can provide relevant information on both local microgeometry and visual appearance of a studied surface. The pixelwise angular reflectance is modeled to allow the relighting of the surface appearance under any arbitrary light direction. The primary methods used to model this local reflectance in each pixel are polynomial texture mapping, hemispherical harmonics, and, more recently, discrete modal decomposition. For all these methods, a uniform distribution of the light positions over the hemisphere is an implicit hypothesis. However, it is not always possible to satisfy this condition in practice. As a result of this nonuniform distribution, several artifacts can affect the reconstruction and alter the quality of the visual appearance assessment. To address this issue, we propose a methodology consisting of the estimation of the spatial distribution of the lighting directions used during RTI acquisitions based on a local density estimation. These local density values are used afterward to weight the least square regression and thus correct the contributions of each image to the RTI acquisition. This methodology is applied on three surfaces with visual singularities, which present different reflectance responses. From the presented results, it can be concluded that it is necessary to take into account this nonuniformity in order not to alter the quality of reconstruction/relighting from RTI data and the subsequent inspection task.

Solutions for the quality control of metallic surfaces are proposed. Specifically, we study a deflectometric apparatus based on coaxial structured light and the related algorithmic procedure, which is able to detect the faulty surface of a sample captured by a video sequence. First, by considering the metallic surface a dynamic scene illuminated under different light conditions, we develop the descriptor residuals of linear evolution of light (RLEL) that extracts the defectiveness information starting from the movement of the object without explicitly considering the physical characteristics of the light structure. Then, leveraging on RLEL, we present a hybrid learning (HL) technique capable of overcoming the data-driven approach used in classic deep learning (DL). By exploiting a multisteps training process, we combine the model-based descriptor RLEL and a classical data-driven convolutional neural network (CNN) to obtain an unconventional gray-box CNN, which exceeds the performance of popular DL solutions such as 3-D-inception and 3-D-residual DL networks. Remarkably, HL also shows its validity in comparing the performance of the same network structures trained not in a hybrid way, namely without the injection of the model-based information given by RLEL.

We propose a method to estimate the surface normal of concave objects. The target object of our method has a specular surface without diffuse reflection. We solve the problem by analyzing the polarization state of the reflected light. The polarization analysis gives a constraint to the surface normal. However, polarization data from a single view has an ambiguity and cannot uniquely determine the surface normal. To solve this problem, the target object should be observed from two or more views. However, the polarization of the light should be analyzed at the same surface point through the different views. This means that both the camera parameters and the surface shape should be known. The camera parameters can be estimated a priori using known corresponding points. However, it is a contradiction that the shape should be known in order to estimate the shape. To resolve this problem, we assume that the target object is almost planar. Under this assumption, the surface normal of the object is uniquely determined. We show that the surface normal of the nonplanar part can also be estimated using the proposed method.

The purpose of this paper is to explore the use of fully convolutional neural networks (FCN) to perform a semantic segmentation of deflectometric recordings for quality control of reflective surfaces. The proposed method relies on a U-net network to identify the location and boundaries of the object and the possible defective areas present on it by performing a pixel-wise classification based on local curvatures and data modulation. Experiments were performed on a real industrial problem using four variations of the architecture. The results demonstrate that the method combining geometric and photometric information enables the identification of a wider variety of shape and texture imperfections, with the resulting segmentations closely correlated with the visual impact of the defects. In addition, several suggestions are presented for near-term industrial utilization.

Usage of a three-dimensional (3-D) sensor and point clouds provides various benefits over the usage of a traditional camera for industrial inspection. We focus on the development of a classification solution for industrial inspection purposes using point clouds as an input. The developed approach employs deep learning to classify point clouds, acquired via a 3-D sensor, the final goal being to verify the presence of certain industrial elements in the scene. We possess the computer-aided design model of the whole mechanical assembly and an in-house developed localization module provides initial pose estimation from which 3-D point clouds of the elements are inferred. The accuracy of this approach is proved to be acceptable for industrial usage. Robustness of the classification module in relation to the accuracy of the localization algorithm is also estimated.

In recent years, driven by new standards and image brand, product quality has become highly critical in the food industry. Particularly concerning the beverage industry, the potential presence of very small foreign objects, especially glass fragments, has to be checked. The visible optical domain offers an advantageous alternative solution to the expensive and intrusive x-ray systems. However, such a solution requires the development of robust processing algorithms satisfying real-time industrial constraints, which is a very challenging task. A detection method based on kurtosis and local information that emphasizes connected components is proposed, as well as two robust criteria that give a greater robustness to the detector. A specific kurtosis is calculated using an inclined frame definition that is based on both time and spatial dimensions. Such kurtosis increases the detection capacity of the moving objects and allows the estimation of their direction without adding more costs. The kurtosis being very sensitive to outliers, noisy objects that are very small are filtered, allowing efficient detection of foreign objects, such as glass fragments, as long as they are bigger than the noisy objects. In case of big foreign objects, the low sensitivity of the kurtosis is compensated for by the large detected surface. The proposed detection method can be directly applied on video sequences acquired by a standard RGB camera in industrial environments. The experimental results show the effectiveness of the method in detecting real random foreign objects, regardless of their size or their transparency, in various semiopaque bottles.

To ease visual inspections of exterior aircraft fuselage, new technical approaches have been recently deployed. Automated unmanned aerial vehicles (UAVs) are now acquiring high-quality images of aircraft in order to perform offline analysis. At first, some acquisitions are annotated by human operators in order to provide a large dataset required to train machine learning methods, especially for critical defect detection. An intrinsic problem of this dataset is its extreme imbalance (i.e., there is an unequal distribution between classes). The rarest and most valuable samples represent few elements among thousands of annotated objects. Deep learning (DL)-only based approaches have proven to be very effective when a sufficient amount of data are available for each desired class, whereas few-shot learning (FSL)-dedicated methods (matching network, prototypical network, etc.) can learn from only few samples. In a previous work, those approaches were compared on our applicative case and it was demonstrated that combining DL model and prototypical neural network in a hybrid architecture improves the results. We extend this work by questioning the interface between models in such a hybrid architecture. We show that by carefully selecting the data from the well-represented class when using FSL techniques, it is possible to enhance the previously proposed solution.

A polarization filter array (PFA) camera is an imaging device capable of analyzing the polarization state of light in a snapshot manner. These cameras exhibit spatial variations, i.e., nonuniformity, in their response due to optical imperfections introduced during the nanofabrication process. Calibration is done by computational imaging algorithms to correct the data for radiometric and polarimetric errors. We reviewed existing calibration methods and applied them using a practical optical acquisition setup and a commercially available PFA camera. The goal of the evaluation is first to compare which algorithm performs better with regard to polarization error and then to investigate both the influence of the dynamic range and number of polarization angle stimuli of the training data. To our knowledge, this has not been done in previous work.

We present a robust approach for detecting defects on an aircraft electrical wiring interconnection system in order to comply with the safety regulations such as the forbidden interference and allowed bend radius of cables and/or harness in mechanical assemblies. For this purpose, we exploit 3-D point clouds acquired with a 3-D scanner and the 3-D computer-aided design (CAD) model of the assembly being inspected. Our method mainly consists of two processes: an offline automatic selection of informative viewpoints and an online automatic treatment of the acquired 3-D point cloud from said viewpoints. The viewpoint selection is based on the 3-D CAD model of the assembly and the calculation of a scoring function, which evaluates a set of candidate viewpoints. After the offline viewpoint selection is completed, the robotic inspection system is ready for operation. During the online inspection phase, a 3-D point cloud is analyzed for measuring the bend radius of each cable and its minimum distance to the other elements in the assembly. For this, we developed a 3-D segmentation algorithm to find the cables in the point cloud, by modeling a cable as a collection of cylinders. Using the segmented cable, we carried out a quantitative analysis of the interference and bend radius of each cable. The performance of the inspection system is validated on synthetic and real data, the latter being acquired by our precalibrated robotic system. Our dataset is acquired by scanning different zones of an aircraft engine. The experimental results show that our proposed approach is accurate and promising for industrial applications.

The inspection of solder joints on printed circuit boards is a difficult task because defects inside the joints cannot be observed directly. In addition, because anomalous samples are rarely obtained in a general anomaly detection situation, many methods use only normal samples in the learning phase. However, sometimes a small number of anomalous samples are available for learning. We propose a method to improve performance using a small number of anomalous samples for training in such situations. Specifically, our proposal is an anomaly detection method using an adversarial autoencoder (AAE) and Hotelling’s T-squared distribution. First, the AAE learns features of the solder joint following the standard Gaussian distribution from a large number of normal samples and a small number of anomalous samples. Then, the anomaly score of a solder joint is calculated by Hotelling’s T-squared method from the features learned by the AAE. Finally, anomaly detection is performed by thresholding using this anomaly score. In experiments, we show that our method performs anomaly detection with few false positives in such situations. Moreover, we confirmed that our method outperforms the conventional method using handcrafted features and a one-class support vector machine.

Quality control is a key requirement of the Fourth Industrial Revolution, especially in manufacturing. For high-value technical parts, the geometric and appearance quality need to be guaranteed. The control of the part, directly on the manufacturing line, can be achieved with noncontact sensors. Visible spectrum imaging is commonly used. However, more information could be obtained from broader spectrum imaging and from the degree of linear polarization of the light from the part. We investigate polarimetry. We have developed a simple polarimetric imaging system to verify whether linear polarization is capable of amplifying geometric and appearance defects. We evaluate the use of this polarimetric imager given the industrial constraints of injection molding. We compare the performances of different machine learning pipelines with hand-crafted feature extraction and deep transfer learning. With its industrial robustness, polarimetry could be a valuable addition to noncontact imagers for geometric and appearance quality control of plastic molded parts.

The coloring efficiency of most colorization algorithms is suboptimal. Therefore, the dual feature extractor generative adversarial network was designed for colorization. The U-Net-like network was used as the trunk network in the generator. The encoder was used to extract the local features of a grayscale image. The branch extractor used the ResNeXt network that was added to the SE module as a high-level feature extractor to extract the global features of the grayscale image. The two features were fused to predict the chrominance. This strategy prevented color leakage and detail loss in colorization. Moreover, the adversarial loss was added to minimize the tendency toward acquiring an unsaturated tone. The effectiveness of the proposed method was evaluated quantitatively and qualitatively.

Traditional face alignment based on machine learning usually tracks the localizations of facial landmarks employing a static model trained offline where all of the training data are available in advance. When new training samples arrive, the static model must be retrained from scratch, which is excessively time-consuming and memory-consuming. It results in the limitation of its performance on sequential images with extensive variations. Therefore, the most critical and challenging aspect in this field is how to enhance the predictive capability of pretrained model incrementally. To that end, a fast and accurate online learning algorithm for face alignment is proposed. Particularly, extreme learning machine (ELM) is incorporated into a parallel cascaded regression (CR) framework, which we coin parallel cascade regression based on extreme learning machine (CRELM). The proposed model can be fast updated in an incremental way. It has a stronger prediction capability than conventional CR methods. The experimental results demonstrate that the proposed model is more accurate and efficient on still images or videos compared with the recent state-of-the-art approaches.

Plenoptic cameras attract extensive attention these days for their unique information acquisition and postcapture processing capability. In prior work, the decoding pipeline of plenoptic cameras mainly consisted of calibration, aligning, slicing, and resampling. Besides, slicing and resampling are performed on a generated grid of projected centers. Such methods consist of a series of processing steps, and the errors from different steps are accumulated. We propose a simplified decoding pipeline for plenoptic cameras. We first propose a coarse-to-fine strategy to calibrate the microlens array accurately and automatically. Then, unlike prior work, we perform slicing and resampling using a nonregular grid of projected centers. Such a strategy avoids using aligning and excludes the dark regions among different microlenses from decoding. Experiments on published datasets and real-world scenes demonstrate the validation of the proposed method.

Defect detection is critical for product quality assessment. Currently, machine vision technology has gradually replaced inefficient manual inspections. Due to the intricate textures and assorted defects on the product surface, conventional defect detection technology still requires ameliorating. We propose an improved deep learning network for defect detection of magnetic sheets, which contains the major contributions at two aspects. (a) Image preprocessing is utilized to enhance the defect features of the dataset. (b) With the superiorities of the core structures of various efficient convolutional neural networks, several perception modules are formulated for multiscale feature extraction and are stacked to construct our inspection network. The parameters are effectively reduced while pursuing the detection accuracy, which is more in line with industrial computing requirements. Experiments show that our defect detection on magnetic sheets has achieved great results, and the computational resources are saved. Moreover, a case extended to the defect detection of hot rolled steel indicates that the proposed network is scalable and has great application potential.

A deep learning algorithm for Gaussian noise removal from both grayscale and color images is developed. As opposed to most existing discriminative methods that train a specific model for each noise level, the proposed method can handle a wide range of noise levels using only two trained models, one for low noise levels and the other for high noise levels. In the proposed algorithm, the training process consists of three successive steps. In the first step, a classifier is trained to classify the noisy and clean images. In the second step, a denoiser network aims to remove the noise in the image features that are extracted by the trained classifier. Finally, a decoder is utilized to map back the denoised images features into images pixels. To evaluate the performance of the model, the Berkeley segmentation dataset of 68 images (BSDS68) and 12 widely used images are used, and the denoising performance for additive white Gaussian noise is compared with several state-of-the-art methods in terms of peak signal-to-noise ratio (PSNR) and visual quality. For grayscale image denoising of BSDS68, our method gives the highest PSNR on all noise levels (significant mean improvement of 0.99). For color image denoising of BSDS68, except for one low noise level, the proposed method gives the highest PSNR on all other noise levels (mean improvement of 0.3).

When light propagates through water, there will be absorption and scattering. Due to this reason, underwater images suffer from color distortion, contrast loss, and detail blurriness. In order to improve the quality of underwater images, an underwater image restoration method based on adaptive color correction and adaptive transmission fusion strategy is proposed. First, an adaptive color correction algorithm is proposed to compensate the color attenuation. Second, based on the image blurriness, the underwater red channel prior (RCP) and the maximum RCP, an adaptive transmission fusion strategy is designed to estimate the transmission map, which can benefit from each other and collaboratively transmit the transmission accurately. Experimental results show that the proposed method can effectively remove color cast and improve contrast and visibility, especially for underwater images with high turbidity or artificial light sources.

To secure the secret communication, a robust video steganography algorithm is proposed. The major objectives of the proposed method (PM) are: (1) perceiving the region of interest (ROI) keypoints’ locations in video frames for concealing the secret data and (2) determining the appropriate amount of data to be embedded into the perceived ROI keypoints’ region. In the PM, keypoints are initially extracted from each video frame using scale-invariant feature transform (SIFT) and a speeded-up robust features (SURF) descriptor against a set of predefined geometrical and signal processing attacks. Next, ROI keypoints are generated by comparing the SIFT and SURF keypoint descriptors of the original frame and attacked versions of the frame. Then ROI keypoints are divided into four least significant bit (LSB) groups to determine the embedding capacity of each ROI keypoint. Subsequently, the secret data are encrypted using a symmetric key-based shift cipher to provide an additional security layer for secret communication. Finally, the encrypted secret data have been embedded into the ROI keypoints using the LSB substitution method based on the four LSB groups’ values. The PM is tested on 22 standard benchmark videos. The efficiency of the PM is evaluated in terms of the perceptual invisibility, robustness, and concealing capacity. From experimental results, it is observed that the PM outperforms contemporary methods by attaining significant outcomes.

It is well known that more than three or four spectral measurements are required for accurate measurement of color under arbitrary illumination. Previous work has shown 6 to 10 measurements can yield accurate results on average, but with significant numbers of errors above the threshold of obvious visual detection. We demonstrate an imaging system (LED box) consisting of a monochrome camera, multiple LED illumination in a diffuse reflecting chamber that produces excellent color capture in perceptual quality measured in ΔE₀₀. We evaluate this system with respect to practical applications with emphasis on textile quality assessment. Our system of 12 LEDs produces average and maximum training ΔE₀₀ errors around 0.7 and 2.4, respectively; and average and maximum testing ΔE₀₀ errors around 1.5 and 5.0, respectively. This performance exceeds that of reported colorimetric camera systems, except highly specialized configurations. We find that the stability of the system to LED variability under operating conditions is good. We show that pseudoinverse estimate produces better results than the MMSE estimate under actual operating conditions.

In vehicle monitoring, recognizing graphic vehicle identification number (VIN) on the car frame is a particularly important step. While text recognition methods have made great progress, automatic graphic vehicle VIN recognition is still challenging. In VIN images, the VIN text is engraved on the car frame, with complex background and arbitrary orientation, which make it extremely difficult for recognition. We propose an efficient framework for recognizing rotational VIN. First, combining lightweight convolutional neural network and per-pixel segmentation in the output layer, we achieve fast and excellent VIN detection. Second, we take the VIN recognition task as a sequential position-dependent classification problem. By attaching sequential classifiers, we predict VIN text without character segmentation. Finally, we introduce a VIN dataset, which contains 2000 raw rotational VIN images and 90,000 horizontal VIN images for validating our framework. Experiments results show that the framework we proposed achieves good performance in VIN detection and recognition. By automatically identifying the VIN, we can quickly confirm the vehicle’s identity and help vehicle monitoring and tracking.

Most existing dehazing approaches employ image priors such as a dark channel prior, which may be violated in the nighttime. In addition, these approaches perform dehazing directly on the original hazy image, which usually results in detail blurring and noise amplifying. To address both issues, an effective single nighttime image haze removal approach is presented. We first decompose the hazy image into a structure layer and a texture layer. Specifically, the dehazing operation is performed only on the structure layer to avoid detail loss and noise amplification. Due to the low frequency characteristics of the haze and the glow of artificial lights, the atmospheric veil is estimated on structure layer dehazing. Furthermore, to eliminate the nonuniform illumination effects, we present an ambient light estimation approach based on multiscale fusion. At the same time, the denoising is performed in the texture layer. Extensive experiments demonstrate that the proposed approach recovers a high-quality, haze-free, and noise-free image with vivid details.

Fusing the pertinence of natural scene statistics-based methods and the ubiquity of convolutional neural network-based methods, a no-reference image quality assessment (IQA) method fusing deep learning and statistical visual features for no-reference image quality assessment (FDSVIQA) is proposed. For the statistical visual features, a local normalized luminance map and a local normalized local binary pattern (LBP) map of the image are constructed, and the local normalized luminance features and the gradient-weighted local normalized LBP features are extracted on the two maps, respectively. These two kinds of features are concatenated to build the image statistical visual features. For deep learning, the local normalized luminance block and the localized normalized LBP block are input into a double-path deep learning network, and the statistical visual features are input into the deep learning network to be integrated with the depth features. After learning and training, IQA is achieved. The performance of the proposed FDSVIQA algorithm is tested on the Laboratory for Image and Video Engineering (LIVE), LIVE Multiply Distorted Image Quality Database, and Multiply Distortion Optics Remote Sensing Image databases. Experimental results show that the FDSVIQA algorithm has excellent subjective and objective consistency and good robustness for both distorted natural images and distorted remote sensing images. In addition, the FDSVIQA has database independence.

Image correspondence is established by “matching” the feature descriptors of the interest points in the target image to that of the reference image. By acceptance testing, we refer to a postmatching hypothesis test used to screen out potential false matches—conventionally using descriptor test statistics. We propose a new acceptance testing strategy that does not rely on the descriptor test statistics exclusively. The contribution we bring is to demonstrate that, unlike feature matching, acceptance testing may incorporate additional photometric values of the scene to improve the recall rate. We show experimentally that the acceptance testing strategy that incorporates image feature detection statistics we refer to as detector response-ratio thresholding that are usually excluded from the feature descriptor vectors has a superior recall–precision performance compared to the state-of-the-art feature extraction techniques.

The fast detection and classification of underwater targets is a key issue in the operation of intelligent underwater robots. In order to improve the detection speed of underwater targets and reduce the missed detection rate of small targets, an improved YOLOv3 algorithm named YOLOv3-Marine is proposed. The network parameters were reduced and the detection speed was increased due to improving the YOLOv3 network structure. The residual module was optimized to improve the feature extraction capabilities of the network, which greatly reduced the rate of missed detection in the case of densely distributed targets. Finally, the prediction scale module and the loss function were improved to increase the detection accuracy of small underwater targets. The final experimental results showed that the proposed YOLOv3-Marine algorithm has a higher detection speed and detection accuracy than the YOLOv3 algorithm.

Person reidentification (ReID) is an important issue in the field of image processing and computer vision. Because pedestrian images are often affected by various interference factors, such as occlusion, illumination changes, posture changes, and background changes, extracting discriminative features is an important method to improve the accuracy of ReID. Based on the two existing methods of pose-sensitive embedding and batch feature erasing, a new feature extraction model for person ReID tasks is proposed. The model uses the view information as global features and uses the batch feature erasure method to extract fine-grained features. The mutual complementarity of the two features improves the accuracy of person ReID. In addition, by introducing the attention module, the structure of the complex network becomes concise and the amount of calculation becomes smaller. Through a large number of experiments on three public datasets, it can be seen that the proposed model can effectively deal with the occlusion environment, and it can also obtain competitive results when compared with other state-of-the-art models.

In the field of pattern recognition and computer vision, applying the symmetric positive definite (SPD) matrix to represent an image set is widely studied, and the remarkable performance that it yields demonstrates its effectiveness. However, the computational burden of the original SPD matrices is usually high, which may restrict the applicability of existing methods. To address this problem, we proposed an SPD manifold dimensionality reduction (DR) algorithm. Specifically, we map the original SPD manifold into a more discriminative lower-dimensional one via a learned mapping. We first construct a graph model using the mechanism of collaborative representation to characterize the local structure of the original manifold data. Then, we formulate the SPD manifold DR problem into an elaborately designed objective function introduced by the graph-embedding framework, aiming to learn the mapping. Finally, the trace optimization method is chosen to solve this optimization problem. The experimental results on some benchmark datasets demonstrate the superiority of our method over the state-of-the-art image set classification methods.

An image detail enhancement algorithm is proposed based on in-place residual homogeneity (IP). Residual homogeneity is a physical law, which mainly explains the texture similarity between the same image residual at slightly different resolutions. As we all know, a single image can be divided into a base layer and a detail layer, and the effective estimation of the detail layer is the key in a detail enhancement algorithm. In the experiment, we find that the residual layer of an image obtained by bilinear interpolation is closely related to its detail layer, hence it can be used as the initial estimation of the detail layer, then residual homogeneity is applied to update the residual layer until the accurate detail layer is acquired. In the process of updating residuals, a searching method called fast in-place searching (FIPS) is used. FIPS only takes advantage of the residual homogeneity within the in-place region, which accelerates the project about 93%. Different from the local-based and global-based methods, our IP gets the detail layer directly and amplifies it. It has many good properties, such as being fast, edge-aware, robust, and parameter-free. Good performance has been demonstrated on several widely used datasets by both subjective and objective evaluations.

Image registration is an essential prerequisite for multisource image fusion. To further improve the registration accuracy for long-wave infrared and visible images, a feature point descriptor construction method—edge-oriented log-Gabor descriptor is proposed. Initially, grayscale edge images are obtained by a structured random forests edge detector, which is more suitable for extracting features using multiscale and multioriented log-Gabor filters. Then, edge features are used for selecting orientation-stabilized feature points from speeded up robust feature points, and descriptors are constructed with a two-stage Gaussian weighting scheme. Finally, feature point pairs are matched by computing the difference of descriptors. Experimental results show that the proposed approach achieves better performance compared with state-of-the-art methods.

Riemannian manifold has attracted an increasing amount of attention for visual classification tasks, especially for video or image set classification. Covariance matrices are the natural second-order statistics of image sets. However, nonsingular covariance matrices, known as symmetric positive defined (SPD) matrices, lie on the non-Euclidean Riemannian manifold (SPD manifold). Covariance discriminative learning (CDL) is an effective discriminative learning method that employs the Riemannian manifold in the SPD kernel space. However, in practice, the discriminative learning of CDL often suffers from the problems of poor generalization and overfitting caused by a finite number of training samples and noise corruption. Hence, we propose to address these problems by importing eigenspectrum regularization and graph-embedded frameworks. Discriminative learning with SPD manifold is generalized by the graph-embedded framework, which combines with eigenspectrum regularization in the SPD kernel space. Three local Laplacian graphs of graph-embedded framework and two eigenspectrum regularized models are incorporated to the proposed method. Comprehensive mathematical deduction of the proposed method is depicted with the “kernel tricks.” Experimental results on set-based face recognition and object categorization tasks reveal the effectiveness of the proposed method.

Images collected in fog can lose contrast and fidelity due to the scattering and absorption of light by the fog particles. In particular, unlike common white fog, colored fog exhibits varying degrees of light attenuation at different wavelengths, resulting in the simultaneous deviation of contrast and color. We present the failure of the traditional algorithm in the colored haze removal process. In order to eliminate the negative influence of colored fog on images, we introduce a hazy image restoration algorithm based on the modified color balance and variant of the dark-channel dehazing process. Our proposed technique corrects the color deviation, such that the transmission of each color channel is consistent, thus allowing for defogging via the dark-channel principle to improve the image quality. Moreover, we use the L₀ gradient minimization algorithm to improve the dark-channel algorithm, thus optimizing the output of underwater image processing. Our proposed method is implemented for the restoration of several hazy underwater images, proving its ability to effectively eliminate the influence of different types of colored fog, overcoming the key limitation of the traditional dehazing approach.

Deep convolutional neural networks (CNNs) have achieved considerable success with image denoising. However, they still lack consistent performance across different noise types and levels. We extend noise scenarios to four categories: Gaussian, random-impulse, salt-and-pepper, and Poisson. We also propose a multinoise-type blind denoising network (MBDNet) that solves the blind denoising task using a uniform deep CNN architecture. The network can be divided into two stages where a concise CNN is first used to estimate auxiliary noise-type and noise-level information. Estimation results are then integrated as additional channels of the noisy image and are fed to the subsequent denoising stage. A unique two-branch structure is further adopted in the residual denoising CNN, wherein a shallow branch predicts the filter-flow mask and adaptively adjusts the feature extraction of the parallel deep branch. Extensive experiments on synthetic noisy images validate the effectiveness of the noise-estimation and denoising subnetworks and show that MBDNet is highly competitive as compared to state-of-the-art methods in both denoising performance and model runtime.

Local binary patterns (LBP) are well documented in the literature as descriptors of local image texture, and their histograms have been shown to be well-performing texture features. A method for texture description that is based on the α-cutting approach is presented. The presented approach combines basic definitions from the fuzzy set theory with the main concept of LBP descriptors, which resulted in powerful texture features. The general method is introduced and defined and its binary, ternary, and quinary versions evaluated in tests produced excellent results in texture classification. The performance of our method is presented by an extensive evaluation on four datasets—KTH-TIPS2b, UIUC, Virus, and Brodatz. The introduced descriptors are compared with some of the classical approaches—LBP, improved LBP, local ternary pattern, including one very promising LBP variant—median robust extended LBP (MRELBP), as well as with three non-LBP methods, based on deep convolutional neural networks approaches—ScatNet, FV-AlexNet, and fisher vector based very deep VGG. Our method effectively deals with many classification challenges and exceeds most of the other approaches. It outperforms the classical approaches on all datasets, even in its simplest binary version. It outperforms the MRELBP descriptor on the UIUC, KTH-TIPS2b, and Brodatz datasets and reaches a better classification performance than two out of the three deep learning approaches on the KTH-TIPS2b dataset.

Road detection is a vital task for autonomous vehicles, as it has a direct link to passengers’ safety. Given its importance, researchers aimed to improve its accuracy and robustness. We look at the task from a holistic point of view, where we aim to balance computation and accuracy. A multimodal road detection pipeline is proposed, which fuses the camera image with the preprocessed LIDAR input. First, the LIDAR input is preprocessed using three-dimensional models inspired from computer graphics to generate image-like representations. Then, the preprocessed LIDAR input is combined with the camera image using a fusion module named inputs cross-fusion module, to reduce the computation amount required by other fusion strategies. To prevent the accuracy loss caused by the computation gain, we introduce the surface normal information to add distinctiveness. Furthermore, we propose a cost/benefit metric to evaluate the trade-off between computation cost and accuracy of road detection approaches. Several tests were conducted using the KITTI road detection benchmark based on deep convolutional neural networks, the obtained results were considered very satisfactory. In particular, the robustness of the proposed approach resulted in accuracies higher than 95% on different road types, comparable to those of the state-of-the-art techniques. In addition to marginally reducing the inference time of the used DCNN on images with a resolution of 1248  ×  352  pixels to 130 ms using an NVIDIA GTX-1080TI.

Current iris recognition technology faces practical difficulties. For example, the unsteady morphology of a heterogeneous iris is generated by a variety of different devices and environments. The existing iris data sets have difficulty meeting the requirements of learning methods under a single deep learning framework. Research on iris recognition security has limited response to results stealing. Therefore, we propose a one-to-one heterogeneous iris certification method with universal sensors and safety output based on multisource fusion features and entropy labels of lightweight samples. This method is based on the classic convolutional neural network structure, and a lightweight neural network certification structure is designed. At first, convert the constrained multistate iris image into the digital features based on statistical learning and multisource feature fusion mechanism. The information entropy of the iris feature label is used to set the iris entropy feature category label and design a certification function that meets the requirements of different acquisition sensors according to the category label to obtain the certification result. Through the result encryption output module, the security output is achieved between the user and the certification result, and measures can be taken in time to confirm the stealing attack to improve the security of the output of certification result. As the requirement for the number and quality of irises changes, the category labels in the certification function are dynamically adjusted using a feedback learning mechanism. Three different acquisition sensors in the JLU iris library are used to do the experiments. The results prove that, for lightweight constrained multistate irises, the above mentioned problems are ameliorated to a certain extent by this method.

Multiple region proposal networks (RPNs) have been recently combined with the Siamese network with deeper backbone networks for tracking and shown excellent accuracy with high efficiency. Although the destruction of the strict translation invariance caused by network padding in the original ResNet-50 is solved by a custom sampling strategy, its impact is not eliminated from the network structure itself, and the multilayer feature fusion is insufficient. To this end, we propose an object tracking framework based on SiamRPN with the deeper backbone networks and cascaded RPN (D-CRPN). First, we exploit the cropping-inside residual units for reforming ResNet-50 to break the spatial invariance restriction and train the robust backbone networks for visual tracking. Then, the feature transfer blocks are proposed to achieve the effective integration of the outputs of multiple blocks in a specific network stage. Finally, to improve the robustness of our tracker, we present a quality measure for the synthetic response maps of RPN modules and then use it to calculate the adaptive weights for the linear weighting method. The extensive evaluation performed on OTB100, VOT2016, and VOT2018 benchmark datasets demonstrates that the proposed D-CRPN tracker outperforms most of the state-of-the-art approaches while maintaining real-time tracking speed.

With recent advances in human pose estimation and human skeleton capture systems, pose-based action recognition has drawn lots of attention among researchers. Although most existing action recognition methods are based on convolutional neural network and long short-term memory, which present outstanding performance, one of the shortcomings of these methods is that they lack the ability to explicitly exploit the rich spatial-temporal information between the skeletons in the behavior, so they are not conducive to improving the accuracy of action recognition. To better address this issue, the two-stream spatial-temporal neural networks for pose-based action recognition is introduced. First, the pose features that are extracted from the raw video are processed by an action modeling module. Then, the temporal information and the spatial information, in the form of relative speed and relative distance, are fed into the temporal neural network and the spatial neural network, respectively. Afterward, the outputs of two-stream networks are fused for better action recognition. Finally, we perform comprehensive experiments on the SUB-JHMDB, SYSU, MPII-Cooking, and NTU RGB+D datasets, the results of which demonstrate the effectiveness of the proposed model.

Diffusion-based inpainting can be used to repair some damaged parts or remove the undesirable regions in an image. Generally, good visual effects can be achieved after inpainting. However, some traces, such as the differences of local variances and noise pattern, are left in the inpainted image, making it easy for the forensic algorithms to locate the inpainted regions. To eliminate this drawback and achieve an anti-forensics capability, we propose an approach that can remove the traces of the diffusion-based inpainting. Since the pixel values of the inpainted regions are diffused inward by the surrounding pixels, we first analyze the noise pattern of the pixels neighboring the inpainted regions and select the nearby pixels that are directly used for inpainting. After that, a statistical probability model is constructed for each channel in the image, which is used to generate the noise pattern and fill the inpainted regions. Experimental results show that the proposed approach has a good capability for anti-forensics.

We propose a method for hyperspectral image (HSI) super-resolution by designing a tensor singular value decomposition (t-SVD) and three-dimensional total variation (3D-TV) regularization terms. The super-resolution method is designed as an optimization problem whose cost function consists of a data-fidelity term, the low-rank representation term by t-SVD, and the 3D-TV regularization term. The sparse representation term is used to enhance the low-rank quality to unify the spectrum and space of HSI. Furthermore, the 3D-TV regularization term exploits the spectral and spatial similarity between adjacent pixels of HSI. Then we develop an effective algorithm for solving the resulting optimization by the alternative direction method of multipliers. The results on the simulated and the real data demonstrate that the proposed method is competitive with other state-of-the-art methods.

Given a natural language description, description-based person re-identification aims to retrieve images of the matched person from a large-scale visual database. Due to the existing modality heterogeneity, it is challenging to measure the cross-modal similarity between images and text descriptions. Many of the existing approaches usually utilize a deep-learning model to encode local and global fine-grained features with a strict uniform partition strategy. This breaks the part coherence, making it difficult to capture meaningful information from the within-part and semantic information among body parts. To address this issue, we proposed an inner-cross-modal attentional multigranular network (IMG-Net) to incorporate inner-modal self-attention and cross-modal hard-region attention with the fine-grained model for extracting the multigranular semantic information. Specifically, the inner-modal self-attention module is proposed to address the within-part consistency broken problem using both spatial-wise and channel-wise information. Following it is a multigranular feature extraction module, which is used to extract rich local and global visual and textual features with the help of group normalization (GN). Then a cross-modal hard-region attention module is proposed to obtain the local visual representation and phrase representation. Furthermore, a GN is used instead of batch normalization for the accurate batch statistics estimation. Comprehensive experiments with ablation analysis demonstrate that IMG-Net achieves the state-of-the-art performance on the CUHK-PEDES dataset and outperforms other previous methods significantly.

The ear has emerged as a new biometric trait to recognize humans from their profile faces. Stability over the years, noninvasive capturing process, expressionless images, and significant variation in shape among individuals make the ear a suitable choice when compared with other biometrics. Convolutional neural network (CNN)’s capability to learn and discriminate specific features irrespective of image variation makes it the most suitable choice for ear recognition in comparison to local geometrical feature extraction methods. The recognition performance affecting factors such as occlusion, image resolution, and head rotation during ear acquisition in an uncontrolled environment makes it a challenging task, and the use of spatial features only in CNN limits the recognition rate. We explored the various CNN configurations and propose the spectral–spatial feature-based CNN. An embedding algorithm is proposed to fuse multilevel spectral information from the image domain with spatial features extracted at each deep layer of the CNN network. The poor convergence problem of CNN is tackled by introducing batch normalization after every convolution. The analysis of the result shows that the proposed framework is superior among all CNN achieving 98.15% and 78.88% accuracy on the Annotated Web Ears with 1000 images and earVN1.0 with 28,412 images, respectively. In addition to fine-tuning and configuring the layers of CNN, the fusion of missing spectral features and spatial features can strengthen the CNN.

Head pose estimation is a typical computer vision task that has been applied to many helpful applications. It is still challenging due to occlusions, low resolution, and extreme pose changes. A method is proposed for landmark-free head pose estimation in a fusion inception deep neural network. The fusion inception network is developed to predict head pose angles without landmarks in RGB images only, which avoids the error caused by landmark location faults. A three-channel prediction module in the developed network is designed to perform classification and regression jointly. A data augmentation method is proposed for sample expansion to alleviate overfitting. A balance loss strategy is proposed to take place of cross-entropy for pose angle classification. The developed balance loss can be applied to deal with class imbalance. The proposed method has excellent performance in head pose estimation by comparison with state-of-the-art methods on some challenging datasets.