2.1.

Advances in Occluded Face Recognition Algorithms

Recent advancements in occluded face recognition have significantly improved the robustness and accuracy of recognition algorithms under challenging conditions. The sparse representation classification (SRC) method, introduced by Wright et al.,⁷ became a foundational approach for occluded face recognition. By employing $L1$ norm regression, SRC derived sparse coefficients, which proved effective in classifying occluded faces. It attained recognition rates of 98.1% on the Extended Yale dataset and 94.1% on the augmented reality (AR) face database. In 2011, Zhang et al.⁸ proposed the collaborative representation method (CRC), which utilized least squares for coefficient estimation, thereby reducing computational complexity. CRC demonstrated recognition rates exceeding 90% on various experimental datasets. Ou et al.⁹ introduced a sparse representation-based classification (SSRC) method that incorporated an occlusion dictionary. SSRC achieved recognition rates of 97.2% on the Extended Yale B and 95.3% on the AR datasets for non-occluded faces, whereas for occluded faces, the rates were 95.8% and 92.7%, respectively. Zheng et al.¹⁰ devised an iterative robust coding technique using a Laplacian–Uniform hybrid approach, which ensured high recognition accuracy even under challenging conditions such as occlusion and pixel damage. Despite these improvements, traditional methods remained limited to “shallow” facial features, often missing finer details.

In recent years, the rapid advancement of deep learning has ushered in significant progress in occluded face recognition. Mundial et al.¹¹ leveraged supervised learning techniques to identify masked faces, achieving an accuracy rate of up to 97% through facial features extracted by deep neural networks. Vu et al.¹² combined deep learning with local binary pattern features to capture information from the eyes, forehead, and eye sockets of masked faces. These features were then merged with those learned from a face detector, creating a unified framework for masked face recognition. Their system recorded an $F1$-score of 87% on the COMASK20 dataset and 98% on the Essex dataset.

Montero et al.¹³ engineered an end-to-end face recognition model based on the ArcFace architecture, incorporating data enhancement and dynamic dataset mixing. This approach resulted in an average accuracy of 98% in recognizing masked faces. Hariri¹⁴ proposed an efficient method tailored for recognizing masked faces during the coronavirus disease 2019 (COVID-19) pandemic. The approach employed VGG-16, AlexNet, and ResNet-50 for feature extraction and quantization, followed by a multi-layer perceptron (MLP) classifier. On the real-world masked face recognition dataset (RMFRD), VGG-16 achieved a recognition rate of 91.3%, ResNet-50 reached 89.5%, and AlexNet secured 86.6%. Golwalkar and Mehendale¹⁵ introduced FaceMaskNet-21, a neural network optimized for masked face recognition using multiple convolutional and fully connected layers. Validated on various datasets, the network demonstrated 88.92% accuracy with an execution time of under 10 ms. Zhang et al.¹⁶ developed a lightweight occluded face recognition model based on MobileNetV2. They replaced the average pooling in the attention module with a depth-wise separable convolution and integrated an improved dual attention module. Their model achieved accuracies of 90% and 91% on the mask-labeled faces in the wild (LFW) and mask-AgeDB datasets, respectively. Huang et al.¹⁷ proposed a progressive learning loss for face recognition (PLFace) method, implementing a progressive training strategy for deep face recognition. PLFace adaptively adjusted the weights of masked and unmasked samples at different training stages. Experiments revealed an average accuracy of 77% on the RMFRD dataset, 99.7% on the LFW dataset, and 94% on the IJB-C 1:1 validation. Ge et al.¹⁸ introduced a convolutional visual self-attention network (CVSAN) that combined local convolutional features with self-attention for long-range dependencies. On the Masked VGGFace2 dataset, CVSAN surpassed ArcFace, increasing accuracy by 0.8% on LFW and boosting TPR@FAR = 0.1% from 89.90% to 95.16% on SM-LFW.

Cheng et al.¹⁹ implemented a face recognition system by Google (FaceNet) with transfer learning, using InceptionResNetV2, InceptionV3, and MobileNetV2. They incorporated a cosine annealing mechanism, which enhanced accuracy to $\sim 93\%$ across all three models. In recent studies, Zhong et al.²⁰ introduced masked uncertainty fluctuation to measure sample identifiability by combining feature amplitude and variance uncertainty. The approach resulted in an average accuracy improvement ranging from 1.33% to 13.28%. Faruque et al.²¹ designed a lightweight convolutional neural network model that integrated batch normalization, dropout, and depth normalization to optimize overall performance. Compared with other deep learning models, this model achieved a high recognition accuracy of 97%. Sharma et al.²² developed a novel dual method for masked face detection using AntelopeV2, which utilized the RetinaFace detection algorithm and the ResNet100 convolutional neural network for face detection and embedding generation. Experimental results indicated an accuracy of $\sim 98.5\%$. The research outlined above explored diverse approaches to occluded face recognition, addressing challenges through various models, data augmentation techniques, and feature fusion methods. However, several limitations and challenges persisted:

2.2.

Progress on GANs for Facial Recognition

The application of GANs in facial recognition tasks has gained significant attention, particularly in addressing challenges such as occlusion and low-quality images. Li et al.²³ introduced a masked face recognition method using deocclusion distillation, which combined GAN and attention mechanisms to predict and reconstruct facial features. Experiments demonstrated that this approach improved accuracy by 1.3% over VGGFace and 0.2% over VGGFace2. Fu et al.²⁴ presented a GAN-based unsupervised low-light image enhancement network with an attention module to improve image quality. Using an autoencoder, the method adapted enhancement across regions, highlighting details and reducing noise. It achieved a peak signal-to-noise ratio (PSNR) of 21.523 and a structural similarity index measure (SSIM) of 0.812 on the paired normal/lowlight images (PNLI) and low-light (LOL) test sets.

Chen et al.²⁵ enhanced face detection with an improved Xception model incorporating a local GAN. By replacing standard convolutions with inception blocks using dilated convolutions, the model effectively captured multi-scale features and achieved over 90% accuracy in detecting small-area faces. Zhang et al.²⁶ developed a domain embedding GAN for face repair, integrating three types of face domain knowledge into a hierarchical variational autoencoder to guide the repair process. Experiments showed that domain embedded generative adversarial network (DE-GAN) surpassed leading image inpainting methods on CelebA and CelebA-HQ datasets, achieving SSIM scores of 0.893 and 0.895 and PSNR scores of 26.132 and 26.208, respectively. Lin et al.²⁷ proposed a face de-identification method using GAN with a seven-layer network and two discriminators to boost feature extraction. Their model, evaluated through pixel loss, content loss, and adversarial loss, achieved over 90% recognition accuracy across various datasets.

Zhang et al.²⁸ introduced a GAN-based method that utilized contextual information to detect small-sized faces in complex environments. They generated virtual images with rich contextual information using GAN, fused these with real images, and created a comprehensive dataset for training deep learning models. Trigueros et al.²⁹ devised a method for generating realistic training data using GANs. The approach combined synthetic images with real images and employed a multi-scale generator network architecture to capture more details and variations. Experiments on the wider facial detection in the wild (WIDER FACE) and face detection data set and benchmark (FDDB) datasets demonstrated the effectiveness of their method in recovering clear, high-resolution faces from small, blurry ones. Yang et al.³⁰ constructed a semantic face restoration method using a dual discriminator DCGAN. Leveraging the VGG16 network to learn deep image features, their model achieved clearer and more realistic restoration results at the pixel level. Experiments on the CelebA dataset reported a PSNR of over 26 on most test datasets.

Hong et al.³¹ proposed a two-stage face inpainting method. The first stage predicted facial landmarks to provide geometric and symmetry information for the GAN. In the second stage, the masked face image and corresponding facial feature points were input into a GAN to inpaint the missing areas. In experiments, their method achieved SSIM and PSNR scores of 0.9 and above 30, respectively, outperforming the light adaptive face image normalization. Huang et al.³² introduced Cycle Style GAN, which integrated the pre-trained Style-GAN 3 network into the Cycle-GAN architecture for near-infrared to visible (NIR-VIS) cross-domain learning. This model synthesized realistic visible images from near-infrared (NIR) images and achieved a rank 1 accuracy of 99.6% on the CASIA NIR-VIS 2.0 database. The research mentioned above illustrates various contributions of adversarial networks in the field of face recognition. However, they still face certain limitations and challenges:

2.3.

Studies on Lightweight Network Architecture

The development of lightweight network architectures has become increasingly crucial for deploying deep learning models in resource-constrained environments, such as mobile devices. These architectures aim to reduce computational complexity and memory usage while maintaining high accuracy, particularly in tasks such as facial recognition where model efficiency is critical. Andrew and Menglong³³ introduced the MobileNet model, which employed depth-wise separable convolution to reduce parameters and computational complexity. In face attribute classification, MobileNet achieved 88.7% mean average precision (mAP) with just 1% of the computation. It recorded a mAP of 19.3% in common objects in context (COCO) object detection and maintained 79.4% accuracy in face embedding tasks, all while significantly reducing model parameters.

Sandler et al.³⁴ presented MobileNetV2, featuring an inverted residual architecture and linear bottlenecks, which improved efficiency and representation. Compared with MobileNetV1, MobileNetV2 achieved the same 22.1% mAP on COCO object detection while cutting parameters by 16%, computational load by 38%, and runtime by 26%. Howard et al.³⁵ further advanced this by proposing MobileNetV3, which integrated automated machine learning techniques to optimize its lightweight design. By employing network architecture search technology, the model automatically identified the optimal network architecture and introduced the efficient hard Swish activation function, further reducing computational overhead. Replacing SSDLite’s feature extractor with MobileNetV3 yielded a 27% speed improvement over MobileNetV2. Zhang et al.³⁶ developed ShuffleNet, which used group convolution and channel shuffle to lower computational complexity and enhance information flow. On the ImageNet 2012 dataset, experiments showed that ShuffleNet reduced classification error by 3.1% and computational complexity by 45 MFLOPs compared with MobileNet.

Ma et al.³⁷ introduced ShuffleNetV2, which enhanced group convolution and optimized feature transfer mechanisms. By simplifying network design and improving feature transfer efficiency, ShuffleNetV2 achieved better computational and storage efficiency on practical hardware, making it well-suited for low-power environments. At 500 MFLOPs, ShuffleNetV2 was 58% faster than MobileNetV2, 63% faster than ShuffleNetV1, and 25% faster than Xception. Tan and Le³⁸ presented EfficientNet, which employed compound scaling to optimize network width, depth, and resolution. This method ensured high efficiency across various resource constraints. On the ImageNet dataset, EfficientNet achieved a top accuracy of 97.1% with 66M parameters, surpassing MobileNetV2. Han et al.³⁹ introduced GhostNet, a model that created additional feature maps through linear transformations of existing ones, significantly lowering computational demands. Experiments showed that GhostNet outperformed other networks on the ImageNet dataset with a top 1 accuracy higher than MobileNetV3 by $\sim 0.5\%$ while maintaining similar latency. Liu et al.⁴⁰ developed a lightweight convolutional neural network for real-time semantic segmentation. The network used branched skip connections to capture contextual information and applied factorized dilated depth-wise separable convolutions to learn features from various scales. Despite its small size of 0.8M parameters, the network processed $1024\times 512$ images at 60 FPS on a single RTX 2080Ti graphics processing unit (GPU).

Chen et al.⁴¹ investigated a parallel design that combined MobileNet and Transformer, leveraging the strengths of MobileNet in local feature processing and Transformer in global interactions. In ImageNet classification tasks, this design outperformed MobileNetV3 within the low FLOP range of 25M to 500M FLOPs. Lyu et al.⁴² proposed a GPU-free real-time object detection method using a quantized single-shot multi-box detector (MobileNet-SSD), combining the lightweight design of MobileNet with the real-time detection of SSD. The quantized model significantly reduced computational and storage requirements. Experiments on a dataset of 22k monitoring images demonstrated a compression ratio of up to 21 times and a detection speed of nearly 25 FPS in a central processing unit (CPU)-only environment, with an mAP of 86.83% and a model size of 600 KB. Kavyashree and El-Sharkawy⁴³ enhanced the MobileNet baseline architecture and reduced its size to 2.3 MB through techniques such as weight quantization, model pruning, and channel pruning, achieving an accuracy of 89.13%.

Shi et al.⁴⁴ introduced DPNet, a dual-path network for efficient object detection with lightweight self-attention. It used a self-attention module in the backbone to encode global interactions and a multi-input version in the feature pyramid network (FPN) for cross-resolution interactions. On the COCO dataset, DPNet achieved 29.0% AP on the test-dev set with only 1.14 GFLOPs and a model size of 2.27M for $320\times 320$ images. Jia et al.⁴⁵ designed a recognition model based on an improved YOLOv7 combined with the lightweight MobileNetV3. MobileNetV3 was used for feature extraction, reducing the number of parameters while integrating the coordinate attention (CA) mechanism and the SIoU loss function to enhance accuracy. The model was tested on image datasets, achieving an accuracy of 92.3%. The research into lightweight network architectures is extensive but several challenges remain:

To address this issue, studying the integration of lightweight network architectures with DCGANs could enhance the stability of network training. This combination would improve the model’s ability to learn and interpret the complex features of occluded areas, thereby enhancing the accuracy of face recognition.

3.1.

Architecture of LW-DCGAN

The core architecture of LW-DCGAN features a generator and a discriminator. The generator utilizes a lightweight convolutional network architecture, integrating an FPN and an ARCM to progressively generate high-resolution images of occluded faces. Through the use of transposed convolution layers and Tanh activation, the generator upscales features to produce the final image output. The discriminator, on the other hand, extracts facial features through convolutional layers and incorporates a CA module to enhance feature representation. Furthermore, an auxiliary face recognition module is cascaded within the discriminator to support face recognition tasks. This design is intended to enhance feature extraction efficiency and image quality, thereby ensuring high-accuracy face recognition even in the presence of occlusion. The network architecture of LW-DCGAN is illustrated in Fig. 1.

Fig. 1

Architecture of LW-DCGAN.

3.2.

Generator Network

3.2.1.

Lightweight convolutional module

In the generator, the lightweight core component is the bottleneck block, which consists of a series of convolutional layers and activation functions. The first layer of the bottleneck block utilizes point-wise convolution to expand the number of channels, allowing for the extraction of more detailed feature information. The second layer employs depth-wise convolution to fuse features, integrating information from different levels. The final layer again uses point-wise convolution, this time for dimensionality reduction, which decreases the computational load and the number of parameters. In addition, squeeze-and-excitation modules are incorporated into the network. These modules dynamically adjust the importance of channels by learning the relationship between effective and ineffective weights, thereby enhancing the network’s expressive power and accuracy. Batch normalization (BN) is applied to stabilize and accelerate the training process, whereas rectified linear units (ReLU) are used as activation functions to introduce non-linearity. The detailed architecture of the bottleneck block is illustrated in Fig. 2.

Fig. 2

Architecture of the bottleneck.

3.2.2.

FPN module

The core of the FPN architecture is the establishment of top-down and bottom-up feature pathways. The top-down path is responsible for up-sampling high-level feature maps to match the size of the feature maps in the bottom-up path. Conversely, the bottom-up path connects low-resolution, high-level semantic features with high-resolution, low-level semantic features through feature fusion. This architecture enables shallow features to receive guidance from deeper features, thereby enhancing the detection capabilities of the shallow features. The entire process constructs a comprehensive feature pyramid, allowing features of different scales and semantic levels to be effectively utilized. In our model, we select layers 2, 3, 5, 9, and 12 from the feature extraction architecture to extract image features and merge them as multi-scale feature information within the FPN. The process of building the FPN network is illustrated in Fig. 3.

Fig. 3

FPN network architecture.

The input image undergoes a series of operations, including convolution and pooling, to form different scale feature layers C1, C2, C3, C4, and C5. These layers have an increasing down-sampling rate and decreasing resolution. The $\text{Input}\in R3*224*224$ with three channels and a size of $224\times 224$ undergoes down-sampling, resulting in feature maps C1 to C5 with sizes $C1\in R16*112*112$, $C2\in R16*56*56$, $C3\in R24*28*28$, $C4\in R48*14*14$, and $C5\in R96*7*7$. The convolution with a $1*1$ kernel to change the feature channel number to 64 and the subsequent 2× up-sampling are applied, resulting in different-sized feature maps $\mathrm{P}5\sim \mathrm{P}2$. P5 is calculated from C5, and P4 is derived by combining P5 and C4, and so forth. The size is $P5\in R64*7*7$, $P4\in R64*14*14$, $P3\in R64*28*28$, and $P2\in R64*56*56$. The final output feature map set is $P=[P2,P3,P4,\text{and}P5]$.

3.2.3.

ARCM module

To enhance feature extraction in the unobscured regions, LW-DCGAN incorporates ARCM into the generator’s design. This module boosts feature extraction capabilities, mitigates gradient vanishing issues, and addresses generator instability. The ARCM consists of three key components: a context enhancement (CE) module, a CA module, and a spatial attention (SA) module. The CE module employs four parallel branches, each utilizing $3\times 3$ convolutional kernels, aimed at maintaining the same receptive field size while minimizing the overall number of parameters. In this context, the receptive field refers to the perceptual range of each neuron within the network in relation to the input data. Each branch generates a feature map, which are then concatenated to form an enriched context feature map. The concatenation operation is crucial in integrating diverse feature information from different branches, thereby enhancing the model’s expressive capabilities. Following the cascading of the CE with the CA and SA modules, a skip connection is introduced to further stabilize the network. The structural framework of ARCM is depicted in Fig. 4.

Fig. 4

ARCM module.

The input to the ARCM is the output feature map collection from the FPN. Through deconvolution, the input feature maps are up-sampled by applying convolution, thereby unifying the size of all feature maps to a common resolution. These processed feature maps are then concatenated channel-wise, merging them into a single tensor. As the channel dimensions of P2, P3, P4, and P5 are already aligned, it is only necessary to concatenate them along the channel axis to form the tensor. This concatenation results in a context-enhanced feature map. The CA module then applies global average pooling (GAP) and global maximum pooling to the concatenated feature map, after which the results are passed through an MLP, which consists of a three-layer fully connected network. The final output is obtained through a normalized sigmoid function. The GAP operation is performed on the context-enhanced feature map to generate the channel attention feature map ${\mathrm{CA}}_{\mathrm{avg}}$, as illustrated in Eq. (1)

Eq. (1)

$${\mathrm{CA}}_{\mathrm{avg}}(c,h)=\left(\frac{1}{W}\right)\sum _{i=0}^{W-1}CE(c,h,i),$$

where $\mathrm{CE}(c,h,i)$ denotes the value of the feature map at channel $c$, height $h$, and width $i$. The symbol $W$ stands for the width of the feature map. Applying fully connected layers and a sigmoid activation function to ${\mathrm{CA}}_{\mathrm{avg}}$ results in the channel attention weight feature map ${\mathrm{CA}}_{\text{weight}}$, as shown in Eq. (2)

Eq. (2)

$${\mathrm{CA}}_{\text{weight}}=\sigma (W_{ca}\times {\mathrm{CA}}_{\mathrm{avg}}+b_{ca}),$$

where $W_{ca}$ means the weight of the fully connected layer, and $b_{ca}$ is the bias of the fully connected layer. $\sigma$ represents the sigmoid function. The fully connected layer receives ${\mathrm{CA}}_{\mathrm{avg}}$ as input and performs a linear transformation of weights and biases. The channel attention weight feature map ${\mathrm{CA}}_{\text{weight}}$ and CE are multiplied element by element to obtain the channel attention enhancement feature map ${\mathrm{CA}}_{\text{enhanced}}$. The workflow of the CA module is shown in Fig. 5.

Fig. 5

CA module.

The SA module first performs channel-wise average pooling and channel-wise max pooling, followed by a two-dimensional convolution. The resulting feature maps are then passed through a sigmoid function. Afterward, the feature maps are concatenated along the channel dimension. Finally, a skip connection is introduced to prevent information loss and mitigate gradient vanishing. The architecture of the spatial attention module is illustrated in Fig. 6.

Fig. 6

SA module.

By applying both an average pooling kernel and a max pooling kernel to the channel attention-enhanced feature map ${\mathrm{CA}}_{\text{enhanced}}$, we perform pooling operations in both horizontal and vertical directions. The feature map $H_{\text{pool}}$ in the horizontal direction is obtained, as shown in Eq. (3)

Eq. (3)

$$H_{\text{pool}}(c,h)=\left(\frac{1}{W}\right)\sum _{i=0}^{W-1}{\mathrm{CA}}_{\text{enhanced}}(c,h,i)+{\max }_{i=0}^{W-1}{\mathrm{CA}}_{\text{enhanced}}(c,h,i),$$

where ${\mathrm{CA}}_{\text{enhanced}}(c,h,i)$ symbolizes the value of the $i$’th element in channel $c$ and feature map $h$.

Simultaneously, in the vertical direction, we obtain the feature map $W_{\text{pool}}$ by applying average pooling and max pooling as shown in Eq. (4)

Eq. (4)

$$W_{\text{pool}}(c,w)=\left(\frac{1}{H}\right)\times \sum _{j=0}^{H-1}{\mathrm{CA}}_{\text{enhanced}}(c,j,w)+{\max }_{j=0}^{H-1}{\mathrm{CA}}_{\text{enhanced}}(c,j,w),$$

where $H$ stands for the number of elements considered along the height direction during pooling. ${\mathrm{CA}}_{\text{enhanced}}(c,j,w)$ denotes the value of the element in channel $c$ and feature map at the $w$’th column and $j$’th row.

The width of the feature map $W_{\text{pool}}$ is transposed to $W$, resulting in the transposed feature map $W_{\text{pool}\_\text{transpose}}$. Concatenated with the feature map $W_{\text{pool}\_\text{transpose}}$, the feature layer $M$ is obtained. It is subject to a $1\times 1$ convolution operation, resulting in the feature layer $M1$. Then, batch normalization and ReLU activation function are applied to $M1$, resulting in the feature layer $M2$. $M2$ is segmented, and the segmented parts are transposed and subject to a $1\times 1$ convolution operation. The resulting feature maps are then passed through the sigmoid function to generate the final spatial weights. The spatial weights in the height direction are represented by $H^{\prime }(i,k)$ as shown in Eq. (5), and the spatial weights in the width direction are represented by $W^{\prime }(j,k)$ as shown in Eq. (6)

Eq. (5)

$$H^{\prime }(i,k)=\sigma (H_{\text{pool}}(i,k)),$$

Eq. (6)

$$W^{\prime }(j,k)=\sigma (W_{\text{pool}}(j,k)),$$

where $i$ symbolizes the position coordinate along the height direction of the feature map, $j$ represents the position coordinate along the width direction to the feature map, and $k$ stands for the pixel value of a channel in the feature map.

Ultimately, the output of the ARCM module is ${\mathrm{SA}}_{\text{enhanced}}$, as shown in Eq. (7)

Eq. (7)

$${\mathrm{SA}}_{\text{enhanced}}=\sum _{k=1}^C\sum _{i=1}^H\sum _{j=1}^{W}G(i,j,k)\times H^{\prime }(i,k)\times W^{\prime }(j,k),$$

where $G(i,j,k)$ denotes the value at position $i$ in the original feature map.

3.2.4.

Network slimming

To compress the generator network model into a more compact size for efficient deployment in resource-limited environments, such as mobile devices, our goal is to reduce model parameters, simplify architectures, and accelerate the inference process. We employed network slimming, a technique that enables us to achieve a streamlined network without introducing excessive complexity. Following this, fine-tuning is performed to restore the original performance, ensuring that the generator’s feature extraction and image generation capabilities are preserved. In this context, the original generator model in LW-DCGAN is referred to as the teacher generator, whereas the slimmed-down version is known as the student generator.

In addition, we compute the scaling factor ${\gamma }_i$ for the BN layer. This factor directly influences the output of the convolutional layer, and if its value is too small, the overall output of that channel will be biased toward lower values. This implies that, during the forward propagation process, the channel carries less information. The scaling factor ${\gamma }_i$ can thus serve as a basis for determining the importance of convolutional channels, as shown in Eq. (8)

Eq. (8)

$${\gamma }_i=s\times \frac{1}{\sqrt{{{\sigma }_i}^2+ϵ}},$$

where $s$ is a constant scaling parameter applied during normalization, which directly affects the output of the convolution layer. ${\sigma }^2$ is the variance of the $i$’th channel. $ϵ$ is a very small positive number. Use ${10}^{-5}$ as the value of $ϵ$. ${\gamma }_i$ is initialized to 1 and then gradually adjusted through the training process.

In the optimization objective of the generator, the $L1$ regularization term is introduced as a penalty to limit the numerical size of ${\gamma }_i$. An $L1$ regularization term is added to the original loss function to penalize the sum of the absolute values of the scaling factors. The optimization goal is given by the following expression, as shown in Eq. (9)

Eq. (9)

$$L_{\text{new}}=L_G+\alpha \sum _i|{\gamma }_i|,$$

where $L_G$ represents the original generator loss function, $L_{\text{new}}$ stands for the total loss after introducing the $L1$ regularization term, and $\alpha$ is the coefficient of the $L1$ regularization term, controlling the impact of the regularization term on the total loss. $i$ denotes different channel layers.

Correction of the scaling factor ${\gamma }_i$ is given by the following, as shown in Eq. (10)

Eq. (10)

$${\gamma }_i=\text{sign}({\gamma }_i)\times \max (|{\gamma }_i|-\lambda \times l_r,0),$$

where the sign function converts elements greater than 0 to 1 and those less than 0 to $-1$, and $l_r$ stands for the learning rate. $\lambda$ is the $L1$ regularization strength that controls the intensity of scaling factor decay during the correction process.

Subsequently, all ${\gamma }_i$ are sorted, and the clipping threshold is selected according to the clipping ratio. Fifty percent of the channels are clipped, and the threshold reflects the middle ${\gamma }_i$ value. Then, the scaling factor on the convolution channel of each layer is compared with the threshold. If it is greater than the threshold, the weight parameters of the layer are retained; otherwise, the parameters are set to zero. This step allows many convolution channels with weights of 0 to exist in the model, achieving the sparseness of the model. Record the number of channels and channel numbers that retain weights in each convolutional layer, and redesign the depth of each layer of the network based on the number of channels to obtain a new, more streamlined student generator model. The remaining weights in the old model are put into the new model to realize the construction of the student generator.

In the streamlined network, the number of channels in the fully connected layers is reduced from 256 to 128. Consequently, the channel count in the transposed convolution layers is also decreased, leading to a reduction in the number of parameters in each layer and a decrease in the overall network complexity. Figure 7 illustrates the process of network slimming. In Fig. 7, $R$ represents the scaling factor greater than the threshold, whereas $r$ denotes the scaling factor smaller than the threshold.

Fig. 7

Slimming process of the generator.

Although the algorithm has streamlined the number of channels and network layers, it retains the core functionality necessary to ensure that the network continues to effectively generate high-quality images. The structural details of the slimmed-down LW-DCGAN generator network are outlined in Table 1, which presents the output shapes, operations, convolution kernel sizes, strides, and activation functions utilized at each level.

Table 1

The slimmed-down generator network architecture.

Layer	Output shape	Conv kernel/stride
Input	z_dim, label_dim	—
LabelEncoding	(7, 7, 128)
Dense	(7, 7, 128)
Reshape	(7, 7, 128)	—
Conv2DTranspose	(14, 14, 64)	3 × 3/2
BatchNormalization	(14, 14, 64)	—
FPN	(14, 14, 64)	—
ARCM	(14, 14, 64)	—
Conv2DTranspose	(28, 28, 32)	3 × 3/2
BatchNormalization	(28, 28, 32)	—
Conv2DTranspose	(56, 56, 16)	3 × 3/2
BatchNormalization	(56, 56, 16)	—
Conv2DTranspose	(112, 112, 8)	3 × 3/2
BatchNormalization	(112, 112, 8)	—
Segmentation head	(224, 224, 3)	—
Conv2DTranspose	(224, 224, 3)	3 × 3/2
Tanh activation	(224, 224, 3)	—

3.3.

Discriminator Network

The discriminator plays a crucial role in LW-DCGAN by distinguishing and categorizing images generated by the generator from real images, prompting continuous optimization by the generator. To ensure accurate feature capture from the input data, an attention mechanism is introduced into the discriminator, enhancing the original DCGAN design rather than employing lightweight processing. This attention mechanism improves the accuracy and effectiveness of the discriminator by enabling it to focus on important image regions. The network architecture of the LW-DCGAN discriminator is outlined in Table 2.

Table 2

Discriminator network architecture.

3.3.2.

Face recognition module

To simplify the network architecture of the face recognition module and reduce the number of parameters, we utilize pooled features as the input for the face recognition module, directing them to the softmax layer without incorporating an additional fully connected layer. This streamlined design leverages deep features extracted before the discriminator, enabling the face detection module to perform its tasks without unnecessary complexity.

The softmax layer, a standard output layer in deep learning neural networks for multi-class classification problems, transforms the network output into a probability distribution. Predicted probabilities for each category range between 0 and 1, with the sum across all categories equaling 1. During training, the output of the neural network passes through the softmax layer, yielding a predicted probability distribution, which is then compared with the true labels’ one-hot encoding. The cross-entropy loss function quantifies the dissimilarity between these two distributions, converting the difference between the network’s predictions and the actual labels into a scalar value. This scalar value serves as a metric for evaluating the accuracy of the model’s predictions.

Through optimization algorithms such as gradient descent, the optimizer seeks to minimize the cross-entropy loss function, aligning the predictions more closely with the actual labels. This design not only preserves the integrity of feature extraction prior to the discriminator but also makes the entire network architecture more lightweight, making it suitable for deployment in resource-constrained environments. The architecture of the face recognition module is illustrated in Fig. 8.

Fig. 8

Architecture of the face recognition module.

3.4.

Related Loss Functions

3.5.

Training Algorithm of LW-DCGAN

During the training process of LW-DCGAN, the generator $G$ is used to convert the real data images into the generated image $G\_\text{images}$. The task of the discriminator $D$ is to distinguish images and $G\_\text{images}$. $z$ is a random variable, and $pz(z)$ specifies the probability distribution of this random variable $z$. $D_r$ expresses the prediction result of the discriminator on the real image, and $D_f$ is the prediction result of discriminator for the fake image. $m$ represents the number of samples in a mini-batch during each training iteration. ${\theta }_g$ represents the parameter set of the generator $G$, whereas ${\theta }_d$ refers to the parameters in the discriminator $D$. $\nabla {\theta }_g$ is the gradient of the loss function with respect to ${\theta }_g$, whereas $\nabla {\theta }_d$ represents the gradient of the discriminator with respect to ${\theta }_d$. $W_g$ is the current values of the weights in generator $G$, whereas $W_d$ expresses the current weight parameters of the discriminator $D$. ${\eta }_g$ defines the learning rate of the generator $G$, whereas ${\eta }_d$ is the learning rate for the discriminator $D$. Using $\sum _{i=1}^m\nabla {\theta }_g{L}_{\text{Generator}}(x_i)$, we compute the gradient of the generator loss function for each example $x_i$, and then sum these gradients to get the total gradient of the generator loss function over the mini-batch. Similarly, $\sum _{i=1}^m\nabla \theta d{L}_{\text{Discriminator}}(x_i)$ calculates the gradient of the discriminator loss function for each sample $x_i$ and then sums these gradients to obtain the total gradient of the discriminator loss function on the mini-batch. This total gradient is used to update the weights of the discriminator to minimize the discriminator loss function. The training algorithm is shown in Table 3.

Table 3

Pseudo-code of LW-DCGAN training algorithm.

4.1.

Preprocessing of Dataset

The experimental dataset, CelebA-Mask, comprises over 24,000 face images from more than 4000 individuals, each meticulously annotated with detailed facial features such as hair, eyes, mouth, nose, and facial contours. CelebA-Mask, chosen for its multi-label segmentation properties, serves as an ideal foundation for this study.⁴⁶

For training set A, 4000 original face images were selected from this dataset. To address the diverse types of occlusions found in real-world scenarios, 4000 unoccluded images were also included for training set B. During data preparation, variability was introduced by applying element-wise multiplication, adding randomly positioned and sized black occlusions to the original images in training set B. Throughout the training process, batches of data from both datasets A and B were randomly selected. Segmentation labels from dataset A were fed into the generator to enrich facial structural information, whereas pixel-level details from dataset B were directly used as input for the generator. A fivefold cross-validation approach was employed, dividing the entire dataset into five subsets. Four subsets were used for training, whereas the remaining subset served as the test set. This process was repeated five times, with each subset serving as the test set once, ensuring a comprehensive evaluation. The test set involved in each training iteration is referred to as testing set 1. Figure 9 displays samples from the dataset.

Fig. 9

Partial display of the dataset. (a) Set A. (b) Set B.

To comprehensively evaluate the performance of LW-DCGAN in the occluded face recognition tasks, we additionally selected three widely used datasets as test sets. These three datasets are no longer involved in the training process and are only used as result tests. The Caltech occluded faces in the wild (COFW) dataset is specifically designed to study occluded faces, featuring numerous images with various occlusions such as hats, glasses, and hands, which facilitates testing the performance of the model under complex occlusion conditions.⁴⁷ The LFW dataset includes images with occlusions such as glasses and hats and is primarily used to assess the model’s performance under natural occlusion conditions.⁴⁸ The masked face recognition v2 (MFR2) dataset, introduced during the COVID-19 pandemic, addresses face recognition challenges with masks. It includes images with various mask types, such as medical masks, N95 masks, and cloth masks, as well as other occluders such as sunglasses and hats, simulating partially occluded facial scenes in real-world scenarios.⁴⁹ Table 4 displays the varying occlusion ratios across the datasets.

Table 4

Data set distribution.

4.2.

Experimental Setup and Parameters

The LW-DCGAN algorithm model developed in this study was trained using the TensorFlow 2.5.0 framework, with acceleration provided by an NVIDIA GeForce RTX 2080 Ti GPU. The training was conducted on a computer system equipped with an Intel(R) Core(TM) i5-11320H @ 3.20 GHz processor, Intel(R) UHD Graphics 630 adapter, 16 GB of memory, and a 64-bit operating system. The model was programmed in Python 3.8.3 using the PyCharm 2021 integrated development environment.

The experimental environment offered ample computing resources and stable software support, ensuring the efficient training of the LW-DCGAN model and the achievement of accurate results. The parameter settings used in the experiment are detailed in Table 5.

Table 5

Experimental parameters.

To better utilize the face segmentation label information during the training process of training set A, an additional channel is added at the input layer to incorporate the segmentation labels corresponding to the facial images. A branch is then introduced to handle this segmentation label input, which includes a $1\times 1$ convolution layer for mapping label features that are subsequently fused with the main network.

Considering the added task of predicting segmentation, a $1\times 1$ convolution prediction branch is inserted just before the output of the final transposed convolution layer. This branch has the same number of output channels as the segmentation label channel and is designed to extract segmentation feature maps. During the initialization phase, the additional channel parameters of the generator are set to a non-trainable state, effectively freezing these parameters. Throughout the training loop, by selectively enabling or restoring the training status of these additional channel parameters, we can flexibly control the extent to which the generator utilizes multi-label information when processing different dataset groups. For datasets that do not use multi-label segmentation, the focus remains on mean squared error loss, feature matching loss, and adversarial loss.

4.3.

Ablation Experiment

Ablation experiments help determine the relative importance of various components within the model. By comparing performance differences after removing specific components, we can identify which factors contribute the most to overall model performance. The evaluation metrics used in this experiment include accuracy, recall, SSIM, and PSNR. Accuracy, a commonly used metric in classification models, represents the proportion of correctly classified samples to the total number of samples. It is calculated as shown in Eq. (23)

Recall rate refers to the proportion of samples that the model correctly predicts as positive samples among all the samples that are positive. It is as shown in Eq. (24)

The SSIM is employed to gauge the similarity between two images, with a value closer to 1 indicating higher similarity. It is represented by Eq. (25)

The experiment is divided into three groups. In experiment group A, the full architecture of LW-DCGAN is maintained, including both the FPN and ARCM components. In experiment group B, the FPN is removed from LW-DCGAN while retaining ARCM, resulting in a network denoted as LW-DCGAN (ARCM). In experiment group C, ARCM is excluded whereas FPN is retained, creating the network referred to as LW-DCGAN (FPN). All three groups utilize the same training dataset and apply a cosine annealing learning rate decay strategy. Figure 10 illustrates the changes in each metric from 0 to 200 epochs across these ablation experiments.

Fig. 10

Comparison of evaluation metrics in ablation experiments. (a) Accuracy. (b) Recall. (c) SSIM. (d) PSNR.

By comparing the performance indicators of LW-DCGAN, LW-DCGAN (ARCM), and LW-DCGAN (FPN), several valuable conclusions can be drawn regarding the roles of FPN and ARCM in LW-DCGAN. First, LW-DCGAN demonstrated the highest performance, with an accuracy rate of 88%, a recall rate of 90%, an SSIM index of 0.877, and a PSNR of 28.3 after 200 epochs. The metrics for LW-DCGAN (ARCM) declined compared with LW-DCGAN, showing a 6% decrease in accuracy, a 4% decrease in recall, a drop of 0.035 in SSIM, and a reduction of 0.6 in PSNR. This suggests that FPN has a significantly positive impact on the performance of image generation and classification tasks. In contrast, when ARCM was removed while retaining FPN in LW-DCGAN (FPN), accuracy decreased by 13%, recall by 10%, SSIM by 0.082, and PSNR by 0.95. These significant declines in metrics indicate that the attention mechanism plays a crucial role in the model’s performance. Overall, LW-DCGAN, incorporating both FPN and ARCM, achieved superior performance.

4.4.

Generalization Experiment

In the generalization experiment, we meticulously recorded the accuracy, recall, generator loss, discriminator loss, and changes in PSNR and SSIM for both the training set and testing set 1 throughout the iteration process. In addition, the final performance metrics of the model, including accuracy, recall, SSIM, and PSNR, were evaluated on widely used datasets such as COFW, LFW, and MFR2. This analysis primarily serves to assess how well the model generalizes to unseen data. Figure 11 illustrates the accuracy and recall of the model on the training set and testing set 1 during the iterative training process.

Fig. 11

Accuracy and recall in generalization experiment. (a) Accuracy. (b) Recall.

As the number of epochs increased, both the training set accuracy and testing set 1 accuracy showed gradual improvement. Initially, at epoch 0, the accuracy was relatively low, but as training progressed, it steadily increased until reaching a saturation point. The accuracy on the training set rose from 0.20 to 0.88, whereas the accuracy on testing set 1, after some initial fluctuations, showed a significant upward trend, increasing from 0.15 to 0.85. Similarly, recall also improved for both datasets, with the training set stabilizing at 0.9 and testing set 1 at 0.87. This suggests that the LW-DCGAN model demonstrates a strong generalization ability in recognizing occluded faces, achieving high accuracy and recall on testing set 1. In addition, a comparison of accuracy and recall between the training set and testing set 1 indicates that the model did not exhibit significant signs of overfitting. The experiment further evaluates the LW-DCGAN model using both generator loss and discriminator loss. Figure 12 illustrates the generator loss.

Fig. 12

Generator loss. (a) MSE loss and feature matching loss. (b) Adversarial loss.

The data show a gradual decrease in both mean square error loss and feature matching loss as training progresses. On the training set, the generator’s loss decreases from an initial value of 1.47 to 0.93. This trend aligns with the characteristics of LW-DCGAN, indicating that the model gradually learns more effective generation and discrimination strategies during training. Meanwhile, adversarial loss fluctuates throughout the entire training process, reflecting the oscillations caused by the ongoing competition between the generator and discriminator. However, as shown in Fig. 12, adversarial loss exhibits an overall downward trend.

To form the comprehensive loss function for the discriminator, we combine binary cross-entropy loss with gradient penalty loss. The resulting discriminator loss rates are recorded in Fig. 13, which illustrates an overall fluctuating downward trend in the loss rates for both the training set and testing set 1.

Fig. 13

Discriminator loss.

The variation curves of PSNR and SSIM on the training set and testing set 1 are illustrated in Fig. 14. From the graph, it is evident that although PSNR experiences significant fluctuations, it shows an overall upward trend. Meanwhile, SSIM gradually increases in both the training set and testing set 1, indicating the progressive enhancement of image quality throughout the training process.

Fig. 14

PSNR and SSIM variation curve. (a) PSNR. (b) SSIM.

To further verify the generalization ability of LW-DCGAN, we evaluated the performance of the model on COFW, LFW, and MFR2. Table 6 shows the comparison results of LW-DCGAN on the training set, testing set 1, COFW, LFW, and MFR2 in terms of accuracy, recall, SSIM, and PSNR.

Table 6

Performance of LW-DCGAN on different datasets.

On the COFW dataset, which includes complex occlusions, the model achieves an accuracy of 82% and a recall of 87%, with a PSNR of 27.6 and an SSIM of 0.867, demonstrating robustness in challenging conditions. On the LFW dataset, featuring natural occlusions, the model records a high accuracy of 94% and a recall of 91%, with a PSNR of 28.5 and an SSIM of 0.874. Finally, on the MFR2 dataset, which focuses on occlusions from masks, the model performs with an accuracy of 86%, a recall of 88%, a PSNR of 27.9, and an SSIM of 0.877. Overall, these results validate LW-DCGAN’s strong performance and adaptability across diverse occluded facial recognition tasks.

4.5.

Comparative Experiment

The performance of the LW-DCGAN algorithm was further evaluated through comparison experiments with other GAN variants, including GAN, DCGAN, WGAN-GP, and the Pix2Pix network. Comparing LW-DCGAN with GAN serves as a benchmark to assess its lightweight face recognition capabilities, given GAN’s foundational role in generative models. The comparison with DCGAN allows for an analysis of whether LW-DCGAN outperforms more complex deep convolutional architectures. WGAN-GP, which addresses issues such as training instability and mode collapse through improved loss functions, is compared with LW-DCGAN to evaluate the impact of a lightweight design on training stability. Finally, Pix2Pix, a conditional generative adversarial network for image translation, is compared with LW-DCGAN to determine whether the latter excels in occluded face recognition tasks. The parameter settings for each model are provided in Table 7.

Table 7

Comparison of adversarial network parameter designs.

To better compare the performance of LW-DCGAN in face recognition, we selected FaceNet, ArcFace, VGGFace, and SphereFace (based on ResNet-64) as benchmarks. FaceNet is recognized for its robustness in embedding space optimization using triplet loss, ArcFace for its superior inter-class separability through angular margins, VGGFace as a well-established benchmark for evaluating general recognition performance, and SphereFace for its enhanced angle-based inter-class separation. This evaluation helps position LW-DCGAN relative to these established models. The experimental parameters of the four models are shown in Table 8.

To intuitively evaluate the performance of various networks in occluded face recognition, we primarily compare the models based on accuracy and recall. To assess the convergence and stability of different models and observe the variation of these metrics during the training process, the same training dataset and learning rate cosine annealing strategy are employed for each model. Figure 15 illustrates the variations in accuracy and recall for different models from epoch 0 to 200.

Fig. 15

Comparison of network models. (a) Accuracy. (b) Recall.

In comparative experiments, LW-DCGAN demonstrates significant superiority in training accuracy, achieving a final accuracy of 88% and a recall rate of 90%. Among the GAN-related models, DCGAN and WGAN-GP perform relatively well, with accuracies of 85% and recall rates of 87% and 88%, respectively. However, GAN and Pix2Pix show poor performance. Among the face recognition models—FaceNet, ArcFace, VGGFace, and SphereFace (based on ResNet-64)—SphereFace outperforms the others, achieving an accuracy of 80% and a recall rate of 83%. In contrast, VGGFace performs the worst on datasets with severe occlusion. To further illustrate the advantage of LW-DCGAN in a network scale, we compare these models in terms of model memory, parameters, and inference time. All values are shown in Table 9.

Table 8

Comparison of face recognition model parameter designs.

Table 9

Comparison of model size and inference time.

From the performance metrics listed in the table, it is evident that after the optimization, the memory usage of the LW-DCGAN generator model is reduced by 4.6 MB compared with the DCGAN, with a decrease of 1M in the number of parameters and a reduction of 0.07 s in inference time.