Sample specificity learning aims to treat every single sample as a separate class and mine the underlying class-to-class visual similarity relationship, thus learning discriminative feature embeddings without using category labels. We introduce a correlational instance feature embedding approach to improve the representation ability of deep neural networks. It exploits the self-correlation and cross-correlation of instances in each training batch by learning a feature embedding with intrainstance variation and interinstance interpolation, resulting in stronger discriminability and better generalizability. The exhaustive experiments on several benchmarks show the performance advantages of our proposed method over the existing methods.

Lane detection is challenging under varied light conditions (e.g., night, shadow, and dazzling light) because a lane becomes blurred and extracting features becomes more difficult. Some researchers have proposed methods based on multitask learning and contextual information to solve this problem; however, these methods result in additional computing. A data enhancement method based on retinex theory is proposed. This method improves the adaptability of a lane model under varied light conditions. In particular, we design an image enhancement network for calculating the reflectivity of images, modifying their exposure, and then generating images with consistent exposure. These images are fed to the lane detection model for training and detection. Our network consists of two parts: exposure-consistent image generation and lane detection. We validate our method on the CULane dataset, and results show that it can improve lane detection performance, particularly on light-related datasets.

Arithmetic operations on images are important primary transformations that can be applied to many complicated image operations. We mainly study the intensity-based arithmetic operations on the QIRHSI quantum image representation model to facilitate the development of quantum image processing. The quantum circuits and unitary matrices for four arithmetic operations including reverse, complement, addition, and subtraction are proposed. Finally, the complexity analysis of quantum networks is illustrated to show the effectiveness of the proposed arithmetic operations.

The detection of surface defects in hot-rolled strip steel is an important step in the production process of hot-rolled strip steel, and it is closely related to the final quality. At present, many researchers have conducted many in-depth studies on the classification of strip steel surface defects and achieved fruitful results. However, current research has always focused on modifying the model; this approach has limited improvement in classification accuracy and often brings a reduction in model efficiency. A data augmentation method named SDDA, which can perform multiscale transformations on the images input to the deep learning models, is proposed and validated on X-SDD dataset by MixNet model. The test results show that after the SDDA data augmentation, the accuracy of the model reached 96.81%, which is 1.71% higher than the previous best result; meanwhile, since the improvement occurs on the data side, it only leads to an increase in training time and does not increase the inference time of the model. Through comparison experiments, it can be found that our data augmentation approach provides significant improvements in multiple metrics such as accuracy and F1 for a variety of deep learning models.

Anomaly detection is a key issue in public security. Its accuracy is essential to identify abnormalities and take corresponding actions to ensure the safety of relevant objects, which have a broad application space. The traditional anomaly detection method based on deep learning has too strong generalization ability. At the same time, it lacks recognition ability because it only uses normal data for training. To this end, we propose an anomaly detection model based on few-shot learning, guided by memory modules and trained by a large number of normal samples combined with a small number of observed abnormalities. We introduce memory modules to record normal features, which has the function of updating and reading. When training modules, feature compactness, and separateness loss are utilized, it successfully weakens the strong generalization of CNN and improves the memory module to identify normal samples. Then, based on the few-shot learning approach, we learn a more compact normal data distribution and expand the margin between normal and anomalous events to improve the discriminant ability. Many experimental results demonstrate that our method is practical and feasible, and its performance is better than the existing detection methods.

With the increasing demand for authentication using iris biometrics, iris recognition systems are gradually being deployed on low-power devices. Iris segmentation is a critical step for iris recognition systems. However, existing iris segmentation methods often require a more complex network to obtain higher accuracy, which makes the segmentation model have a greater number of parameters and a higher computation cost. Therefore, these algorithms are not suitable for deployment on low-power devices. To address this problem, we propose a lightweight iris segmentation network, referred to as ATTention Linknet. First, we redesign the feature extraction part of the Linknet, which combines traditional convolution based on an attention mechanism with depthwise separable convolution to reduce computation cost while maintaining accuracy. Then, a proposed loss function that better segments the iris images taken under different cross-sensors is used to train the network. Experimental results on the Chinese Academy of Sciences (CASIA-V4), Indian Institute of Technology Delhi (IITD), and the University of Beira Interior (UBIRIS.V2) iris databases show that the proposed method better balances iris segmentation precision and efficiency compared with state-of-the-art iris segmentation methods. Specifically, our model obtains an error rate of 1.47% on the CASIA-V4 database with 1.69 M parameters.

Hand detection plays an important role in human–computer interaction. Because of the convenient and natural advantages of hands, hand detection is increasingly used in virtual reality, remote control, and other fields. However, since the complex background and the diversity of hand postures, the YOLOv4 algorithm for hand detection suffers from low accuracy and robustness. Therefore, A YOLOv4-HAND network, improved from YOLOv4, is proposed to solve the problem. We first use the dilation convolution to build the feature enhancement pyramid that enables the network to expand semantic information. Second, for better detection of different hand scales, we design a multiscale attention module to capture the correlation of channel information within different scales. Third, we design a head that incorporates a spatial attention module to compensate for the network’s lack of spatial contextual location information correlation. Finally, we use soft nonmaximum suppression to reduce the impact of occlusion. The results show that the YOLOv4-HAND detection network can achieve 83.22% and 93.95% mAP on the publicly available datasets Oxford hand and Egohands datasets. Compared with the most recent method, the YOLOv4-HAND network effectively improves the accuracy of hand detection for practical applications.

Motion detection (MD) is a fundamental step in many advanced computer vision applications, but the various complex challenges in real surveillance videos lead to some false positives and false negatives in the detection results of traditional MD algorithms. Therefore, joint instancewise and instance-union fusion for improving MD algorithms, in which an instance segmentation model is combined with a traditional MD algorithm. are proposed to address this problem. First, for each input frame (indexed by t), the MD algorithm produces a binary mask M_t, and the instance segmentation model produces the specific categories of binary instance masks (BIMs). Second, according to the instance confidence, BIMs are divided into high-quality binary instance masks (HBIMs) and low-quality binary instance masks (LBIMs). Then instancewise fusion of HBIMs with M_t and instance-union fusion of LBIMs with M_t are used to generate a high-quality foreground segmentation mask DtH and a low-quality foreground segmentation mask DtL, respectively. Finally, the bitwise logic addition operation of DtH and DtL produces a more accurate foreground segmentation result than M_t, called D_t. The experimental results show that our proposed method with visual background extractor and YOLACT++ processes at a resolution of 320  ×  240 videos at 30 frames per second. For the Changedetection.net -2014, SBM-RGBD, and labeled and annotated sequences for integral evaluation of segmentation algorithms datasets, the highest overall F-measure of our experimental results with our proposed method are 0.8454, 0.8094, and 0.8939, respectively, surpassing state-of-the-art unsupervised MD methods.

Correlation filter (CF)-based trackers have demonstrated promising performance in the field of unmanned aerial vehicle (UAV) tracking, benefiting from their high computational efficiency. However, model degradation and the boundary effect remain two major obstacles for CF-based trackers. We present an innovative temporal regularized and weighted surrounding-aware CF for UAV object tracking. First, we considered the consistency of response between the current frame and the previous frame of an object in a time series, which effectively enhances the discriminability and generalization capability of CF-based trackers to abrupt appearance variations. Second, we developed an innovative weighted context learning strategy that is able to enlarge the visual field of trackers to cope with the boundary effect. Specifically, we extract weighted surrounding patches with the same size and shape of the object to empower the discriminating ability of the CF. Finally, to validate the effectiveness of our proposed method, substantial comparison experiments with other state-of-the-art algorithms were conducted on three UAV standard data sets. The experimental results demonstrate that the algorithm proposed has good robustness for occlusion, deformation, fast motion, and other challenging disturbances in complex scenes and effectively tracks the target on the platform of a UAV.

Single-image super-resolution (SISR) in unconstrained environments is challenging due to various illuminations, occlusions, and complex environments. Recent research has achieved remarkable progress on super-resolution tasks with the development of deep learning in computer vision. We aim to improve the performance of deep-learning-based super-resolution tasks in different scenarios. Existing works are inherently limited to defective transfer structures such as the pooling method and the one-way information transfer between convolutional layers. To address this limitation, we propose a dense U-Net with the shuffle pooling method, a scalable method that can perform a down-sampling method for extracting features from shallow layers. The modified U-Net with dense blocks for SISR introduces a multi-way information transfer between high-to-low and low-to-high convolutional paths. Meanwhile, an innovative shuffle pooling module is designed to retain all information in the feature map of the previous layer, which is effective and easy-to-port. To minimize structural similarity and gradient error, a hybrid loss function was developed that combined mean square error, structural similarity index, and mean gradient error. The proposed method achieves superior accuracy on previous state-of-the-arts on the three benchmark datasets: SET14, Berkeley Segmentation Dataset 300, and International Conference on Document Analysis and Recognition Dataset 2003.

Human detection still suffers from occlusion, complex backgrounds, and scale-variant problems. Projecting three-dimensional (3D) points onto the ground to generate an orthographic top view (OTV) image for detection can effectively alleviate these problems. However, depth sensors may be placed arbitrarily, making it difficult to create OTV images by the dense point cloud converted from a depth image. We focus on the generation of OTV images and human detection via the constructed OTV image. First, we propose a ground plane extraction method that is well suitable for various camera positions and orientations in complex scenes. Next, points are converted to a uniform coordinate system by ground parameters and encoded to generate a three-channel OTV image. Then, the mainstream two-dimensional (2D) network is employed to detect the human directly on OTV images and further obtain the 3D bounding box by computing the mapping from the OTV image. Besides, we propose a semiautomated annotation method to solve the problem of few OTV image annotations. The proposed method is evaluated on the EPFL dataset, including two subsets, and achieves state-of-the-art performance compared with the existing approaches. Moreover, our 2D and 3D human detection method can run more than 26FPS on the CPU.

With the development of artificial intelligence, deep learning has been widely used in image super-resolution reconstruction. To solve the problems of feature extraction insufficiency, detail loss, and gradient disappearance in super-resolution reconstruction based on traditional deep learning, we propose a lightweight multihierarchical feature fusion network for single-image super-resolution. An important part of our network is dual residual block. To better extract features and reduce the amount of parameters as much as possible, the dual residual block we designed is an excite-and-squeeze structure. To transmit feature information, webadd autocorrelation weight unit into dual-residual block, which can weight each channel according to the image feature information. Extensive experiments show that our method is significantly better than LapSRN, MSRN, and other representative methods. The PSNR on SET14, URBAN100, and MANGA109 datasets are improved by 5 dB and SSIM is improved by 4% compared with the baseline method.

At present, generative adversarial network (GAN) has made outstanding progress for image super-resolution (SR), solving the problem of edge smoothing in SR images. Nevertheless, there are also problems with a lack of texture details, noise, and artifacts. The high-frequency (HF) information of an image is significant for keeping the texture part of the reconstructed image, and the introduction of HF information can increase the learning ability of super-resolution generative adversarial network (SRGAN) for the image features, so we propose a super-resolution GAN based on high-frequency feature fusion (HFF-SRGAN), which can increase the texture detail of the reconstructed image and reduce the noise and artifacts with less computational cost. Experimental results show that our algorithm evaluates significantly better than SRGAN and has significant advantages in texture details and noise compared with other SR algorithms.

Security is becoming an essential field of research as network technology is changing at a rapid pace. Digital images are transmitted in many modes of communication. This transmission must be secure, particularly in applications that demand a high level of security, such as military applications, surveillance, radars, and biometrics. Two hardware solutions of the lightweight block cipher LBlock are provided to secure image data. These proposed architectures are implemented on a Xilinx field programmable gate arrays (FPGA) platform, and the implementation results are compared with existing state-of-the-art designs. The proposed designs took 56 and 64 slices operating at a frequency of 492.441 and 681.663 MHz, respectively, on a Virtex-5 FPGA platform. In addition, the proposed round-based and pipelined designs showed improvement in hardware efficiency over state-of-the-art designs. The proposed hardware architectures are capable of securing different sizes of gray-scale and RGB color images. These hardware solutions were verified by different analyses, including correlation coefficient, mean square error (MSE), peak signal-to-noise ratio (PSNR), histogram, entropy, number of pixel changing rate (NPCR), and unified average changing intensity (UACI). The existing solutions were compared with correlation coefficients, NPCR, UACI, MSE, PSNR, and entropy values and provide better solutions than the state-of-the-art designs.

Specular highlight in images is detrimental to accuracy in object recognition tasks. The prior model-based methods for single image highlight removal (SHIR) are limited in images with large highlight regions or achromatic regions, and recent learning-based methods do not perform well due to lack of proper datasets for training either. A network for SHIR is proposed, which is trained with losses that utilize image intrinsic features and can reconstruct a smooth and natural specular-free image from a single input highlight image. Dichromatic reflection model is used to compute the pseudo specular-free image for providing complementary information for the network. A real-world dataset with highlight images and the corresponding ground-truth specular-free images is collected for network training and quantitative evaluation. The proposed network is validated by comprehensive quantitative experiments and outperforms state-of-the-art highlight removal approaches in structural similarity and peak signal-to-noise ratio. Experimental results also show that the network could improve the recognition performance in applications of computer vision. Our source code is available at https://github.com/coach-wang/SIHRNet.

The purpose of person re-identification (ReID) is to retrieve a person of interest from a set of images taken by multiple cameras. In the person ReID task, the use of global-local features and attention mechanisms to construct robust pedestrian features has been shown to be effective. However, in these methods, the models only focus on extracting pedestrian features with strong discrimination while ignoring the potential features, which are equally valuable and can play an important role in person ReID tasks. To extract these potential features, which are hidden by salient features, we propose a person ReID network based on weight-driven saliency hierarchical utilization. Three improvements are exploited to extract more comprehensive and diverse information of pedestrians. First, we use the non-local module to enhance the feature extraction ability of the model. To mine potential features, we use a saliency enhancement and suppression operation in the non-local module. Second, we employ a new multi-stage global feature fusion module to increase the diversity of features. Third, we use the multi-branch attention module to extract more fine-grained part features to improve the model performance. Extensive experiments show that our model achieves excellent performance on the Market-1501, duke multi-tracking multi-camera re-identification, and multi-scene multi-time person ReID datasets.

Noise level estimation is a long-standing problem in image processing. The challenge arises from the estimation being easily affected by texture information. We propose an innovative noise level estimation method via the kurtosis test, which is a normalized fourth-order moment. The proposed method consists of two stages: the first one is to determine the image patches with normality using the kurtosis test; the noise level is then estimated from these selected normal patches in the second stage, in which the average of the standard deviations is used as the final estimation. Experimental results show that the proposed method outperforms the state-of-the-art estimation techniques in terms of noise level estimation and guided denoising.

Few-shot object detection (FSOD) aims at training an object detector that can rapidly adapt to detect novel classes with a few annotation examples. Existing meta-learning-based FSOD networks have achieved substantial progress, however, they still suffer from several drawbacks: they neglect the fact that more discriminative support features can boost the performance of few-shot learning; the commonly used channel-wise features interaction lacks the spatial information, which may lead to a typical problem that the object is well-localized but given a misclassified label. We study how to leverage the powerful attention mechanism and margin-based softmax loss to tackle the FSOD task. Specifically, we select the cosine margin loss that allows learned features with minimum within-class variance and maximum between-class variance to optimize the lightweight convolutional neural networks of the independent support set branch, which endows the extracted support features with better discrimination. In addition, we design an affinity relation reasoning module (ARRM) to promote the interaction of the support features and the region of interest (ROI) features. The ARRM fully explores the element-wise spatial attention to integrate distinct features via the affinity matrix that measures the relationship between the support features and ROI features. The ARRM also introduces holistic channel attention as a supplement to spatial attention. The holistic channel attention provides global semantic context about support features, which can alleviate the misclassification problem. We empirically evaluate the proposed network on Pascal visual object classes and Microsoft common objects in context benchmarks, and the experimental results demonstrate that our network achieves state-of-the-art performance.

In the last few years, person re-identification (re-id) has made significant progress in supervised learning for matching pedestrians across disjoint camera views in surveillance. However, it is infeasible in many new scenes without sufficient labeled images when extending a re-id system. Therefore, unsupervised methods in person re-id tasks are vital for saving labeling costs. However, cross-camera scene variation is a crucial challenge for unsupervised person re-id, such as the occlusion problem. It results in uneven pairwise similarity distributions, which degrade matching performance. To solve this issue, we propose a local manifold consistency learning (LMCL) framework that consists of a context-aware feature embedding network and a camera-aware manifold alignment strategy. To better extract comprehensive features of persons in images, we propose a saliency feature attention algorithm by cropping feature maps into regions and transforming them into context features. We optimize our model based on sub-domain alignment loss to alleviate the effect of cross-camera scene variation, which closes the distance between sub-domains composed of similar samples. Extensive experimental results and ablation experiments verify the effectiveness of our LMCL approach.

In recent years, Siamese trackers locate the target based on the cross-correlation results of the template and the search branch. Siamese trackers emphasized better features embedding and similarity matching. The latest work focuses on enriching the feature representation of the template. However, it is still difficult for the tracker to eliminate the side effect of the distractors in the search region, resulting in the decline in tracking accuracy. We propose a coattention guided Siamese network (CGS) for visual tracking from two aspects. To mine the inherent correlation between the template and search branch, a coattention block is proposed that can guide the target-aware feature learning during the tracking process. To wisely choose the target-specific features, a self-gated mechanism is exploited on the template features, which can adaptively reallocate the confidence of channels and strengthen the feature representation. We conduct extensive experiments on OTB2015, VOT2016, and GOT-10K benchmarks to evaluate the performance of the proposed method. Experimental results demonstrate that our CGS network performs favorable against other state-of-the-art methods. Moreover, our CGS runs at 79FPS, far above the real-time requirement.

The detection of pavement crack information plays an important role in the service evaluation of infrastructure engineering performance. The randomness of road noise, pollution, and spots make it difficult to ensure the accuracy of traditional image processing algorithms to detect cracks. Therefore, the crack identification technology under complex background conditions is particularly important. We propose a network architecture, named feature pyramid and attention enhancement network (FPAEN) for pavement crack detection. Through a new multiscale feature fusion module, the deep semantic information is integrated into the low-level convolution stage layer by layer, so the context information circulates throughout the whole network. In addition, the attention enhancement mechanism and the multiscale dilated convolution module are used to strengthen the network model’s ability to locate the crack pixels. To evaluate the performance of the proposed approach, we train FPAEN on the DeepCrack dataset and test it on four datasets. Experimental results show that our method outperforms other advanced crack detection approaches in F-score and mean intersection over union.

Existing methods for detecting surface defects in strip steel are mainly aimed at the case of low-resolution images, and little research has been done in the case of high-resolution images with a small percentage of valid targets. In response to the practical problems in the hot-rolled flattening stage, we propose a hot-rolled strip surface defect dataset called XLData-CLS, which contains a total of 5000 defective samples and 10,000 normal samples with an image resolution of 1344  ×  672  pixels. Based on this dataset, this paper improves the ResNet152 model by adding a feature extraction layer at the front of the model so that it can extract information more effectively for high-resolution images. Experimental results on the XLData-CLS dataset show that the proposed method achieves 99.00% classification accuracy on the test set, which is higher than other comparable deep learning models. Validation results on another datasets also demonstrate the accuracy of our method for high-resolution defective images.

Social media sites such as Facebook, Twitter, etc. have become commonplace in everyday life. With the emergence of the internet, the amount of data being created and communicated within social networks has increased exponentially. As much as we enjoy the exchange of information that occurs on social media, protecting and maintaining privacy demands a lot of effort. Users’ data that are to be kept private should be treated as confidential information. As a result, securing data exchange on the internet has become a critical and urgent issue. We have suggested a method for protecting user data which is encrypted by using the Rivest Cipher 4 (RC4) algorithm in multimedia images with the help of 3–3 least significant bits (LSB) techniques, with the added safety of Bernoulli map. We call this technique the “user data protection scheme.” 3–3 LSB is a message hiding process in which the first three stand for three LSB and the next three stand for all three channels (R, G, and B). In this method, we are using the Rivest Cipher 4 (RC4) encryption algorithm to encrypt the user’s data, which will then be embedded into the multimedia image in the next step. With the help of the Bernoulli map, we generate a random sequence of length similar to the size of the image. For every random number generated by the Bernoulli map, we determine the location in the plane and place three bits of secret message into the cover image using the LSB technique. Since we have used two different encryption techniques, i.e., RC4 for encryption of user data and Bernoulli map for embedding encrypted data at random locations over the cover image, our method is more secure than others. We performed this analysis to equate peak signal-to-noise ratio (PSNR) and mean square error. Our experimental result shows that 9 bpp can be embedded in color images with a PSNR higher than 42 dB, and for the gray scale image embedding capacity is 3 bpp with a PSNR higher than 42 dB, which provides better results. Apart from this, it produces less distorted stego-images than currently used methods (an average of 1.68 bpp).

Image denoising is a fundamental problem in image processing. Images may be contaminated by different kinds of noises in different imaging applications. Cauchy noise often arises in many practical applications, such as radar and sonar image processing. We consider removing Cauchy noise in the images. We develop a generalized low-rank model for Cauchy noise removal and design a proximal alternating algorithm to solve this nonconvex model. The convergence of the algorithm is also demonstrated. Numerical experiments illustrate that the proposed method can remove Cauchy noise in the images much better than the existing state-of-the-art methods in terms of both image recovered visual qualities and measure quantities.

Visual sensor-based human–computer interaction can provide environmental perception data of traffic road scenes for mechatronic intelligent vehicles. The existing three-dimensional (3D) reconstruction visualization schemes are mainly based on structured scene reconstruction with multiview geometry and are offline operations. It is difficult for large-scale autonomous driving scenes with wide baselines to ensure consistent reconstruction results in the wide field-of-view area and the center of the light-of-sight area. It is also difficult to achieve uniform reconstruction results for regions with drastic depth changes and inconsistent texture fineness. Therefore, we propose a 3D reconstruction-based visualization pipeline for autonomous driving scenes of wide baseline. First, the stereo geometric model is initialized. The parameters of distortion correction and polar calibration are calibrated. Second, the proposed feature chain code resampling (FCCR) is used to uniformly extract the features for regions with inconsistent texture fineness and depth variation. The filtered 3D data are obtained based on the stereo geometric model. Finally, the texture mapping of the mesh skeleton of 3D data is carried out by the bilinear combination of surrounding multilevel gradual textures. The visualization results of scene rendering and scene roaming are realized. The pipeline has been effectively verified on the simulated experimental platform, real vehicle test, and public dataset. It can meet the real-time 3D reconstruction visualization of wide-baseline scenes. The reconstruction texture is uniform, and the detail response is rich. We can present the intuitive, 3D, real-time, graphical physical quantities of scene data for drivers, intelligent vehicles, AR-HUD, and control systems. The method we present has research value and application significance for the practical engineering of electronic images.

With the continuous improvement of spatial and spectral resolution, the application of multispectral images has greatly increased in remote sensing. However, the amount of image data also increases sharply, which brings great pressure to data storage and transmission. To solve this issue, we propose an end-to-end image compression scheme according to spatial–spectral feature extraction, which can be implemented by a spatial–spectral memory unit (SSMU). Furthermore, to improve the feature extraction capability of this deep network, the multichannel grouping fusion module is adopted to reconstruct and fuse the image features. In the encoder of the proposed compression scheme, the SSMU first extracts spatial–spectral features along the spatial direction and the spectral direction, and the multichannel grouping fusion module extracts the spatial and spectral features of different levels by recombination and fusion of band features of multispectral images. Then, the extracted deep spatial and spectral features are compressed by downsampling. Next, the quantizer and entropy coding convert the data into a compressed bitstream. In the decoder, a reverse process is used to restore the original images. The experiments take the multispectral images of Landsat 8 and WorldView3 as the datasets to verify the superiority of our method and compare it with JPEG2000, 3D-SPIHT, and the CNN-based methods. The results show that the proposed method outperforms the JPEG2000, 3D-SPIHT, and CNN-based methods in PSNR, spectral similarity, and spectral angle mapping metrics at different bit rates.

Biometrics are widely used in security systems, but it has drawbacks such as its nonreusability, being susceptible to getting stolen, and being prone to unauthorized access. These issues can be tackled using cancelable biometrics. In this work, features extracted using Log-Gabor filters are processed according to the proposed max-min thresholding. The resulting binary features are transformed by random projection on a key matrix to create a cancelable template. The method was carefully evaluated using publicly accessible datasets of face near-infrared, palm print, palm vein, dorsal vein, wrist vein, and knuckle print. A significant improvement in performance over many recent methods was observed. Also, the effectiveness of the generated cancelable templates was observed in the general, stolen token, and changeable scenarios. The generated template is 25% of the original input size, which makes it efficient for low-end devices.

In almost all video applications, a video rate control algorithm (RCA) is used by the encoder. The RCA tunes the quantization parameter (QP) to match the encoded bit rate to the available capacity of the communication channel or storage media. Conventional RCAs usually utilize a rate-quantization (R-Q) or a rate-distortion (R-D) model for rate control. A content-based R-Q model for intra coding tree units (CTUs) of the high-efficiency video coding standard is proposed. The model is a convolutional neural network that observes pixels of a CTU and its intraprediction reference pixels and it estimates required bit counts for intracoding the CTU for all QP values simultaneously. The proposed model can be easily used by any video RCA. A given RCA just selects a proper QP for which the estimated bit counts are closer to the allocated bit budget. The evaluation results show a high accuracy for the model. According to simulation results, the mean absolute normalized bit error at CTU level is 19.66% and it decreases to 6.85% at the frame level. Compared with similar networks, the proposed structure has a very low computational complexity.

Monocular depth estimation is an essential step in scene geometry understanding. However, the depth maps predicted by the existing methods have the problems of loss of small target details and blurred edges. To this end, we propose a monocular depth estimation method based on multilevel feature fusion and edge optimization to obtain depth maps with rich depth details and precise edges. First, we improved the encoder part to make it better adapted for the depth estimation task. In addition, we fill the U-shaped network with a dense feature fusion layer (Dense-FL) to capture global context information and then combine the proposed structure attention module to further enhance the obtained feature information, so the depth map contains more detailed information. Finally, we design an edge optimization module to incorporate edge features into the training process and combine the proposed reweighted loss and image edge detail loss to constrain the network, so as to further improve the learning ability of the model to the edge of the object. The experimental results on the KITTI dataset show that the depth prediction results obtained by our method perform better at small targets and the edges of objects in the scene, and the structure of the objects is more complete. Generalization experiments on the Cityscapes and Sceneflow datasets also further confirmed the effectiveness and superiority of our proposed method.

During the last decade, graph regularized dictionary learning (DL) models have obtained a lot of attention due to their flexible and discriminative ability in nonlinear pattern classification. However, the conventional graph-regularized methods construct a fixed affinity matrix for nearby samples in high-dimensional data space, which is vulnerable to noisy and redundant sample features. Furthermore, the discrimination of the graph regularized representation is not fully explored with the supervised classifier learning framework. To remedy these limitations, we propose an adaptive graph-regularized and label-embedded DL model for pattern classification. Especially, the affinity graph construction in low-dimensional representation space and the discriminative sparse representation is simultaneously learned in a unified framework for mutual promotion. More concretely, we iteratively update the sample similarity weight matrix in representation space to enhance the model robustness and further impose a supervised label-embedding term on sparse representation to enhancing its discriminative capability for classification. The dictionary orthonormal constraint is also considered to eliminate the redundant atoms and enhance the model discrimination. An efficient alternating direction solution with guaranteed convergence is proposed for the nonconvex and unsmooth model. Experimental results on five benchmark datasets verify the effectiveness of the proposed model.

Performing high-accuracy object detection under low-illumination conditions is a difficult task. The fusion of infrared (IR) and visible images can provide a great reference value for human observation or other computer vision tasks. Therefore, we propose an image fusion-based method to detect objects under poor lighting conditions. It mainly proposes a whole set of process improvements to enable the detection of medium-sized and small objects in poorly illuminated images with high-accuracy. First, the saliency detection of the IR image extracts the salient area in the image to obtain the main objects. Second, the IR and visible-light images undergo multiscale decomposition and fusion to obtain the background information of the visible-light image. Finally, the salient area is dot-multiplied with the fused image, so it can remove unnecessary background information and interference information while acquiring the main objects. Then, the ablation experiment is used to detect the objects. Experimental results demonstrate the effectiveness and robustness of the method. Moreover, lower false detection efficiency and better detection accuracy were obtained, especially in the detection of objects under poor lighting conditions.

Computer-intensive tasks can be distributed to edge devices in edge computing, which has many benefits, including distributed processing, low latency, better security, privacy, etc. In traditional video surveillance, video is cached in the data center for post analysis and processing, but it is impossible to analyze the specific behaviors of specific objects intelligently in real-time scene. Taking this into consideration, we propose an efficient object detection and tracking framework via edge computing, which distributes the object detection and tracking tasks to edge cameras within the network instead of sending all the data to a centralized server. Its advantages can be summarized as follows: (1) instead of analyzing and processing video in the cloud, we analyze them locally, which greatly reduces the bandwidth and delay caused by the transmission of large amounts of data and ensures the security of the data. (2) we propose a neural network model compression framework for object detection with less storage and computing cost, which reduces the number of parameters and floating point operations by 39.3% and 17.28%. We use the detection results for tracking, combined with a simple tracking method, we can achieve a real-time object detection and tracking framework and more quickly determine the type and trajectory of moving objects. (3) We propose a communication module, which enables data interaction among cameras, and makes it easier for us to find the moving range of the object in the cross-camera area. In the experiment, network cameras and edge devices were used to build a framework, and an offline inference method was used to complete the detection and tracking of objects and judge the motion trajectory to verify the feasibility of the framework.

Most of existing local-based person re-identification (re-ID) methods extract powerful feature representations from multiple body regions of a pedestrian. However, local-based person re-ID methods employ horizontal or grid slicing to divide pedestrian images, which can easily cause network redundancy and misalignment of pedestrian body parts. Attention mechanisms have been recently introduced to computer vision tasks to strengthen local salient features and suppress irrelevant features simultaneously. However, attention mechanisms overemphasize local salient features, which may lead to potential significant information being missed. To address the above issues, we proposed a local information interaction enhancement network (LIEN) for person re-ID. The network integrates attention mechanism with local-based strategy to achieve mutually facilitating and complementary effects. Specifically, LIEN was composed of three parts: a backbone network, a dual attention module (DAM), and a local information interaction module (LIM). DAM was used to guide the backbone network to extract the local salient information, and LIM was used to mine the potential significant information that DAM ignored by exploring the correlations between different channel-level local features. With the combination of DAM and LIM, LIEN mined the salient features while obtaining more potential effective features. In particular, the way that it extracts local features, LIEN effectively avoided pedestrian misalignment caused by uniformly dividing pedestrian images. We evaluated the performance of LIEN on public person datasets, and experimental results indicated that LIEN achieved a more advanced performance.

Accurate and robust detection of prohibited items in x-ray images has been playing a significant role in protecting public safety. However, large-scale variation of prohibited items and diverse backgrounds in x-ray images bring in many challenges to the detection. We propose an effective weight-guided dual-direction-fusion feature pyramid network (WDFPN), making full use of multilevel features to solve the scale variation problem in cluttered backgrounds. Specifically, our WDFPN mainly consists of weight-guided upsample fusion pathway (WUFP), attention-based connection (AC), and downsample fusion pathway (DFP). WUFP uses channel-wise weights generated from high-level features to weight low-level features, reducing invalid information redundancy. AC transfers enhanced low-level detail information to DFP. Subsequently, DFP improves the localization capacity of the entire features pyramid by the bottom-up fusion pathway. Extensive experiments on the security inspection x-ray and occluded prohibited items x-ray datasets demonstrate the superiority of our WDFPN in detecting prohibited items.

In reality, deploying the traditional object detection algorithm on mobile and embedded devices is difficult due to the limited memory and computation resources. To solve this problem, we propose a lightweight, real-time detection algorithm, YOLOv5-R. First, inserting the efficient channel attention (ECA) module into the Ghost module achieves the information interaction between channels and suppresses the redundant features. And applying dense connections to it further improves feature reuse and network performance, forming a Ghost-ECA-Dense (GED) module. Utilizing the GED module to construct the F-GED, which is the backbone with fewer parameters and better performance, replace CSPDarknet53 in YOLOv5. Second, the redundant operations are replaced with Ghost modules and depthwise separable convolution in the neck and head of YOLOv5, and YOLOv5-R is constructed, which significantly reduces the network size and improves the inference speed. Finally, YOLOv5-R is deployed into the AI embedded device Jetson Nano for TensorRT acceleration. The experimental results indicated that the performance of YOLOv5-R outperformed the YOLOv5s. Specifically, the mean average precision was increased from 69.9% to 72.7%, the number of floating-point operations per second and parameters are reduced by 25% and 47.2%, respectively. The model weight was only 7.52 MB, and frames per second achieved 37, which can provide real-time detection.

Interferometric synthetic aperture radar (InSAR) is widely used in acquisition of digital elevation data as well as the settlement of Earth surface. Researching on the processing of space-born InSAR is of great importance in improving the accuracy of sensing data. The phase denoising method based on multi-resolution dictionary sparse coding (SC) is proposed. First, noisy blocks are extracted from the complex valued interferometric image. Then, the dictionary is trained by online method, which uses a portion of elements in the training set to update the dictionary column by column. And the trained dictionary is used to sparsely represent the noisy blocks by greedy method to denoise the image patches with different resolutions. The whole image of a given resolution is formed by combining the patches in the same resolution together. Finally, the denoising results of different resolutions are merged based on hypothesis testing and the phase estimations according to each pixels are computed. The experiments prove that the proposed denoising method based on multi-resolution SC achieves higher peak signal to noise ratio than traditional SC method.

Multispectral pedestrian detection has gained momentum in the research literature with its wide range of applications in car safety, video surveillance, and robotics. We introduce a noise-aware multispectral detector, called NAMPD, that is based on the one-stage YOLOv4 to produce fast and accurate detection. NAMPD merges the color and thermal streams at the decision level by a weighted sum, where the weight is computed using both the noise level and average luminance of a given image pair. We show that the noise level provides a distinction between nighttime and daytime and can be utilized to improve the detection performance. Experiments are carried out on the sanitized KAIST dataset. They demonstrate that NAMPD is able to achieve a log-average miss rate of 4.25%, which is better than that of the conventional IAF R-CNN, cross-modality interactive attention network, MSDS-RCNN, and MCFF. This mechanism can operate at a speed of 15 fps on the general GPU 1080 Ti, and, thus, is a viable option to be deployed in practical pedestrian detection systems.

Region dropout regularization strategies have proven to be highly effective at improving the generalization performance of convolutional neural networks (CNNs) in a variety of computer vision tasks including image classification, object detection, and semantic segmentation because these strategies enable models to focus on a wider range of image region information. However, for action recognition, models need to be able to extract not only useful spatial information but also important temporal and motion information, which cannot be satisfied by traditional regularization strategies. We propose a spatiotemporal dropout strategy to meet the need for regularization in spatial–temporal CNNs. We call it label guided spatial–temporal drop (LGST-Drop); it not only provides effectively structured dropout in the spatial dimension but also regularizes motion information in the temporal dimension. In addition, LGST-Drop’s mask is guided by the predicted categories of the model itself, which we called temporary labels. Extensive experiments on several standard datasets from action recognition domains show the usefulness of the proposed technique in comparison with the previous methods and theirstate-of-the-art variant algorithms.

The recent advent of deep convolutional neural networks (CNNs) in stereo matching has led to significant improvements. However, current CNN methods still face challenges in incorporating hierarchical context information with global dependencies and lacking the discriminative ability of feature representation to resolve matching ambiguities in ill-conditioned regions. To address the aforementioned problems, we propose an improved stereo matching framework that joins a stereo backbone network and an embedded independent multilevel attention subnetwork in an end-to-end trainable pipeline. The stereo backbone network applies a residual atrous spatial pyramid pooling integrated with channelwise attention to capture richer multiscale contextual information and selectively enhance discriminative features. This is followed by unary feature concatenation to construct cost volume for disparity prediction. To further improve performance, the embedded multilevel attention subnetwork learns global coherent contextual information to generate three attention streams, which are used to boost the unary feature representations with spatial encoding, enhance the quality of cost volume, and refine the disparity map, respectively. We show that appending the proposed multilevel attention subnetwork to the stereo backbone network produces significant improvements in matching accuracy. The experimental results on Scene Flow and KITTI 2012/2015 demonstrate that our method can achieve competitive performance in stereo matching.

Understanding vasculatures is important for endovascular intervention simulation (EIS). A recent trend attempts to represent vasculatures with three-dimensional (3D) surface meshes by multiview image and graphics-based techniques used in optical and laser scanners. Conversion from image volume data to 3D surface meshes, however, suffers from staircases and noise. Most existing methods in geometric processing often consider mesh smoothing in their independent settings. However, this approach either is ineffective in removing the artifacts or introduces additional types of artifacts (e.g., volume shrinkage and shape distortion). We propose an effective yet vascular-oriented mesh optimization framework. We first employ a weight-adaptive vertex resampling to eliminate staircases; then, we subdivide vascular meshes to denser ones; and finally, we exploit the constrained biquadratic Bezier surface fitting to produce optimized meshes. In EIS, clinicians are interested in the centerline of surface meshes as well as surface meshes. However, centerline extraction from vascular meshes is still challenging. We adopt an existing centerline extraction approach for vascular meshes based on the observation that human blood vessels are generally composed of piecewise cylindrical shapes. Tests on vascular data illustrate that our optimization approach achieves higher quality results regarding surface smoothness compared to previous approaches. In addition, our extraction approach effectively obtains complete centerlines from real data.

Facial expression recognition (FER) in computer vision is a very daunting computational task because of high-intra class variations. Researchers have developed methods for extracting features from images that apply conventional and deep learning approaches. When conventional approaches are used in FER system, the size of extracted features is very large, which may adversely affect the performance of the classifier. Therefore, in the process of FER, feature selection is an essential phase. We propose an improvement on competitive binary gray wolf optimizer in discrete search space named as Improved MOCBGWO, and it is utilized to perform feature selection within wrapper-based setup. Next, we applied the SVM and K nearest neighbors classifiers to selected features for investigating the performance of the proposed system through two publicly available standard datasets: CK+ and JAFFE. We compared the performance of the proposed improvement with binary gray wolf optimizer, binary moth flame optimizer, binary particle swarm optimizer, and competitive binary GWO for evaluating its efficacy. The experimental results reveal that our proposed improvement boosts the recognition accuracy along with reduction in the size of feature vector in a superior way as compared to other methods.

The increase in video streaming in recent years has led to the development of advanced video encoders to save bandwidth while meeting practical encoding complexity requirements. Video encoders, therefore, need to provide faster turnaround time through toolset optimizations and/or multithreading (MT). The multithread performance of a video encoder depends on the levels of parallel processing available in the encoder. For the libaom AV1 encoder, the default parallelism is limited to row/tile-based processing within a frame, resulting in low MT scaling even when configured with a high number of threads. We present our proposed frame-parallel MT approach for improving the existing libaom AV1 encoder’s performance. For encoding multiple frames in parallel, it is critical to consider their temporal encoding dependencies. To facilitate the parallel encoding of frames, some frames are prevented from acting as references by either marking them nonreference frames or temporarily removing them from the active reference list. The cross-frame dependencies of various encoding parameters are resolved by disabling or delaying their updates across the frames in a parallel encode set. The proposed method offers an average 31% encode-time reduction at the cost of a reasonable YUV-PSNR BD-rate loss of 1.17% for encoding ultra-high-definition sequences using single tile encoding for cpu-used = 2 with 32 threads. The corresponding MT scaling ratio also improved to 7.1x from 4.9x of the existing libaom implementation.

The ability to quickly detect and identify objects floating on the surface of the water can benefit numerous fields including environmental monitoring, civil infrastructure, and security applications. To address this need, we investigated the ability of multiple imaging sensors to perform machine learning-based object detection and classification for small objects on water. The sensors tested in this effort include a long-wave infrared polarimeter camera, a visible wavelength optical camera, and a low-light camera. Small waste objects such as trash bags, Styrofoam cups, wood, plastic bottles, cardboard, and aluminum cans were placed into a riverine environment on the water surface and data were gathered with each of the sensors. Artificial neural networks were trained based on the data gathered, and models were created to perform object detection and classification of those same objects in the riverine environment. The three-camera systems performed well but with clear advantages for specific environmental conditions and object types.

Single-photon sensitive image sensors have recently gained popularity in passive imaging applications where the goal is to capture photon flux (brightness) values of different scene points in the presence of challenging lighting conditions and scene motion. Recent work has shown that high-speed bursts of single-photon timestamp information captured using a single-photon avalanche diode camera can be used to estimate and correct for scene motion thereby improving signal-to-noise ratio and reducing motion blur artifacts. We perform a comparison of various design choices in the processing pipeline used for noise reduction, motion compensation, and upsampling of single-photon timestamp frames. We consider various pixelwise noise reduction techniques in combination with state-of-the-art deep neural network upscaling algorithms to super-resolve intensity images formed with single-photon timestamp data. We explore the trade space of motion blur and signal noise in various scenes with different motion content. Using real data captured with a hardware prototype, we achieved super-resolution reconstruction at frame rates up to 65.8 kHz (native sampling rate of the sensor) and captured videos of fast-moving objects. The best reconstruction is obtained with the motion compensation approach, which achieves a structural similarity (SSIM) of about 0.67 for fast-moving rigid objects. We are able to reconstruct subpixel resolution. These results show the relative superiority of our motion compensation compared to other approaches that do not exceed an SSIM of 0.5.

In recent years, deep learning-based image fusion methods have achieved promising fusion performance. However, existing fusion approaches always focus on increasing the network depth to enhance the feature extraction capacity. In addition, fusion is performed in a fixed feature space, which further degrades the performance. To this end, we proposed an end-to-end infrared and visible image fusion method based on a multibranch encoder, coined MEFuse. In principle, MEFuse increased the number of branches of the encoder, thus mapping the source images into different feature spaces, increasing the diversity of the extracted feature maps, and enhancing the learning capacity of the network. In addition, a multistage fusion strategy was designed to achieve multigranularity image fusion. First, the cascaded multimodal images were merged in the input stage; thus the primary visual fusion was realized in the coding process. Second, the feature maps from multiple coding branches were fused to fuse multimodal information comprehensively. Finally, the long-range lateral connection sent the input appearance of each encoding branch to the decoder to guide the image reconstruction process. Experiments on the benchmark data sets demonstrate that, compared with the existing methods, the proposed MEFuse achieved superior performance in both human perception and objective evaluation.

Few-shot learning (FSL) has been widely used for image classification when the training samples are limited and can effectively avoid overfitting. Most FSL methods are based on metric learning and tend to learn features that are conducive to classifying known classes of images. However, these methods cannot capture the semantic features that are important for classifying new classes of images. To address this issue, we proposed an improved few-shot image classification model based on multi-task learning termed SIM. It combines the self-supervised image representation learning task with the supervised image classification task, thus utilizing the complementarity of these two tasks. SIM has two stages, i.e., the pre-training stage and the meta-training stage. In pre-training stage, we learned the representation of training images via supervised learning and self-supervised learning. Then, in meta-training stage, we trained a linear classifier based on the learned representation. According to the experiments on four data sets, including three natural image data sets and one marine plankton image data set, SIM outperformed existing methods and achieved quite good performance in complex application scenarios.

Lipreading aims to decode the speech content from a moving mouth. It is a very challenging task because lip appearance variations and speech contents are coupled together in the subtle movements of lip region. Especially in the speaker-independent recognition scenario, training and testing data are totally different in distribution due to the diverse speaker identities, making the learned model generalize poorly in the testing task. We propose a Siamese decoupling lipreading network (SDLipNet) to address this problem. Specially, we exploit an encoder–decoder framework to establish a collaborative representation of speaker identities and speech contents, and utilize the identity-specific information to regularize the content feature space. The identity features are derived from a Siamese identity encoder trained with paired visual speech data from different speakers. In addition, we align the content representation with a prior Gaussian distribution by imposing a Kullback–Leibler divergence constraint between the two outputs of the Siamese content encoder. In this way, the learned content feature space is supposed to be universal to the target speaker domain. Extensive experiments on two lipreading benchmarks demonstrate that our proposed SDLipNet can achieve better performance in the speaker-independent recognition task compared with the state-of-the-art lipreading methods.

Recently, the number of cases of distributing malicious codes by exploiting homepages that provide an image search continues to increase, and malicious codes distributed through homepages are causing personal information infringement accidents and DDoS attacks. Due to the malware spread through web pages, privacy theft and infringement are getting serious and DoS attacks happen frequently. Distribution patterns of hidden malicious codes on the image search website were collected, and patterns of collected malicious codes and malicious scripts were analyzed. We have analyzed the malicious samples and derived some additional distribution patterns of web-based malware. Similar patterns are grouped together and a representative feature is then extracted from each group. Each category of the malicious samples contains malicious script codes and their variants. We have implemented a system to automatically detect malicious web sites using the malicious script patterns. The proposed malicious script pattern is expected to be available for the zero-day attacks.