3.1.

Data Input

In the network, the input data consist of either real images of simulated craters, which consists of images captured by a visible-light polarization camera, or images converted from the simulated crater dataset. The dataset is introduced in Sec. 4. We labeled all images in the dataset. For each target detection, three images containing different types of target information were input into the base network to extract the features.

3.2.

Base Network

First, the CNN part of a VGG²⁸ network was adopted as the base network of our multimodal network for training. We used very small $3\times 3$ receptive fields for convolution with each pixel of the input. However, in the experiment, the training speed was slow because there were too many model parameters. For the final network, we choose a ResNet²⁹ network with batch normalization layers and activation layers as the base network of the multimodal network. Experiments showed that the output feature of the ResNet-4 network contains the most discriminative information and yields the best detection performance.

Figure 3 shows the network architecture of a residual block, and Fig. 4 presents the network architecture of the base network. Each residual block has a batch normalization layer and an activation layer to avoid the disappearing gradient problem and speed up learning. The architecture of each residual block is the same. Res2, Res3, and Res4 stages are composed of res_units (residual blocks), which extract the initial features of the image and sharpen the edges in the image.

Fig. 3

ResNet50 res_units network architecture. $f1$ represents the input filter numbers, $k$ is the kernel, $s$ is the stride, and $p$ is the padding.

Fig. 4

Base network architecture. $k$ is the kernel, $s$ is the stride, $p$ is the padding, Res2, Res3, and Res4 uses the res_units as the parameters. (a) Simulated crater dataset, (b) IQU dataset, (c) I dataset, (d) Q dataset, (e) U dataset, and (f) P dataset.

3.3.

Fusion Network

Our fusion network is similar to a deep fused network¹⁷ architecture and a deep fusion network is multi-input. Network fusion is a process of combining multiple base networks, such as $K$ base networks $\{H_{L_1}^1,\cdots H_{L_K}^K\}$. The conventional fusion, in general, includes two approaches: feature fusion, fusing the feature representations extracted from the networks together, and decision fusion, fusing the scores computed from the networks. Our method focuses on feature fusion and the fusion can be formulated in the function form $H_{(x_0)}=F[H_{L_1}^1(x_0),\dots .,H_{L_K}^K(x_0)]$, where the fusion function $F(.)$, in our method, is the sum of the representations:

3.4.

Detection Network

In the target detection network, the detection network of faster region-CNN (Faster R-CNN)³⁰ is used. Faster R-CNN is a two-stage detector mainly consisting of three major components: shared bottom convolutional layers, a region proposal network (RPN), and a region-of-interest (ROI)-based classifier. In our method, we only use the two components of RPN and ROI.

First, feature figures are a shared bottom feature figure. Based on that feature figure, RPN generates candidate object proposals where afterward the ROI-wise classifier predicts the category label from a feature vector obtained using ROI pooling. The training loss is composed of the loss of the RPN and the loss of the ROI classifiers:

Both training losses of the RPN and ROI classifiers have two loss terms: one is to classify the accuracy of prediction probability and the other is a regression loss on the box coordinates for better localization.

3.5.

Transfer Learning

Neural networks are trained with data. They obtain the information from the data and convert this information into the corresponding weights. These weights can be extracted and transferred to other neural networks, which enables us to “transfer” these learned features without having to train another neural network from scratch. Some researchers use a VGG16 or VGG19 pretrained model to perform the initial training on a network. According to the characteristics of the visible-light polarization image, we input visible-light polarization images into a single-model network, extract the features in the network, and train the single model. We use this model as the pretrained model of the multimodal fusion network. Because the pretrained model has good generalization performance, we can use an analogous structure and the weights directly when training using the new dataset to improve the accuracy of target detection.

4.1.

Datasets

We collected visible-light polarization images of craters that we simulated at a test site to verify our methods. The ground types of the test site include soil, sand, grassland, and other types. We created a simulated crater and used the visible-light polarization camera from different angles and different heights to obtain images for the simulated crater dataset, which is also called the uncharacterized dataset. Using the Stokes equation, the images in the simulated crater dataset were processed and used to create a dataset called the incident light intensity and linear polarization information (IQU) dataset, which belongs to the characterized dataset. Using the LBP algorithm, the texture images corresponding to the simulated crater dataset were also obtained. We combined the IQU dataset and texture diagrams with the simulated crater dataset and processed them to get four semicharacterized datasets: the I dataset, Q Dataset, U dataset, and P dataset. The Stokes equation expression is as follows:

The simulated crater dataset contains 2403 visible-light polarization images and the IQU dataset contains 2403 characterized images. Moreover, the I dataset contains the simulated crater dataset, corresponding texture images, and 801 I images; the Q, U, and P datasets are similar to the I dataset, except that I images are replaced by the Q, U, and P images, respectively, As is shown in Fig. 5.

Fig. 5

Some of the images in the six datasets used in experiments are shown: (a) uncharacterized dataset, (b) characterization dataset, (c), (d), (e), and (f) the semicharacterization dataset. (a) Visible light polarization image, (b) polarization image, (c) polarization feature image, (d) mean fusion image, (e) Laplace fusion enhancement, (f) wave fusion enhanced, (g) Brovey fusion image, (h) multifocus fusion enhancement image, and (i) our method.

4.2.

Training Parameters

We trained all networks on a NVIDIA Geforce GTX 1080 Ti GPU. The proposed framework was implemented using the MXNet toolbox. The size of the input images was $256\times 256$. The network was trained with a minibatch size of 16, a learning rate of 0.001, 10 training epochs, and a learning rate decay every seven epochs. Further, the optimization method was the Adam optimizer. These parameters were constant in our experiments. We used 80% of the images in the dataset as a training dataset and the remaining 20% of the images as the test dataset.

5.1.

Traditional Polarization Detection Experiments

In Fig. 6, we show the target detection results obtained by different methods, where the final image was obtained using our proposed method. It can be concluded from Fig. 6 that the test results obtained by the previous methods require manual observation, but our method can automatically and accurately detect the target and mark the target area.

Fig. 6

Above nine pictures in order: the first eight images are obtained by using the previous methods of image fusion to detect the simulated crater, and the last one is generated using our proposed method to detect the simulated crater in the image. (a) Original image, (b) simulated crater real box, (c) faster R-CNN (VGG) test image, (d) faster R-CNN (ResNet) test image, (e) FV1 test image, and (f) FR1 test image.

5.2.

Single-Modal Experiments

The results obtained by our method and single-modal methods for the simulated crater dataset are compared. The single-modal networks consist of a faster R-CNN (VGG) network, faster R-CNN (ResNet) network, and the Yolov3³¹ network. The resulting models were validated on a test dataset, as shown in Table 1.

Table 1

Experimental comparison table of the simulated crater dataset. FV represents our method to use VGG16 base networks, FR represents our method to use ResNet50 base networks.

According to the precision and mean average precision (mAP) metrics, our model obtains better results than most previous methods. In addition, because two base networks are used in the proposed network, the two models generated are different in size. The ResNet50 network is a better base network than the VGG16 base network. Moreover, its detection precision is higher and model size is smaller, so we obtain a better lightweight network model. In addition, this result demonstrates that using a multimodal network framework and base network to extract features and fuse them to detect targets is effective.

5.3.

ResNet50 Fused Experiments

In order to get the feature figure with the most target information, we find out which stage of ResNet50 has the best fusion effect based on the receptive field³² formulation:

We hypothesize that if the receptive field of the output in the ResNet_4 network covers the original image, the precision will be maximized. We used the IQU dataset as the input dataset to verify this. The ResNet50 network was split into five parts, namely ResNet_1, ResNet_2, ResNet_3, ResNet_4, and ResNet_5. The features of each stage were outputted and then analyzed. The five results are shown in Table 2. The detection precision increases gradually after each of the outputs of ResNet_1, ResNet_2, ResNet_3, and ResNet_4 is fused. However, when the feature output by ResNet_5 is fused, the detection precision decreases. Therefore, the output of the fourth layer of the ResNet network is the best.

Table 2

Using ResNet50 as the base network, it is judged that the fusion of feature diagrams in a certain part of the base network can make the best detection effect, and five experiments have been carried out and the precision rates have been obtained.

5.4.

Multimodal Experiments

For each labeled image in each training dataset, three images with different modals were input to the three networks in the base network to obtain three base network features. The outputs were then fed into the fusion network and detection network. Continuous learning in the network generated the final model. We then verified the multimodal-fusion detection model on the verification dataset, and the results show that the precision obtained is better than the precision of the single-modal model.

The model test results for 12 experiments are shown in Table 3. We used precision, mAP, and model size to evaluate our results. Moreover, we set the intersection over the union (IoU) threshold to 0.8 in the experiment. When $\mathrm{IoU}\ge 0.8$, the simulated crater is correctly detected. As can be observed in Table 1, the test precision of the multimodal fusion detection model is higher than that of the single model. In Table 3, FV represents our method to use VGG16 base networks, CS represents simulated crater dataset, FR represents our method to use ResNet50 base networks. Therefore, FV–CS and FR-CS are experiments using simulated crater datasets, FV-IQU and FR-IQU are experiments using the characterized datasets, and the rest are experiments using semicharacterized datasets. As can be seen from Table 3, the best precision is obtained in the FV-IQU and FR-IQU experiments using the visible-light polarization images characterized by the Stokes equation. These experimental results show that the polarization information is beneficial for target detection. In the multimodal-fusion detection network, the detection results using the ResNet50 base network are better than those using the VGG16 base network. Moreover, we use two different base networks to obtain different model sizes. The size of the network model generated by the VGG-based network was 1.5 G, and the model size obtained by ResNet50-based network was 198.3 MB, which is a factor of 7.7. Hence, it is possible to obtain a lighter multimodal fusion detection model. We, therefore, conclude the following: in multimodal fusion detection networks, polarization information is beneficial for target detection in visible-light polarization images. At the same time, the use of different base networks for target detection leads to different performances.

Table 3

FV-SC represents the use of a VGG16 base network to experiment with simulated crater dataset. Similarly, FR-SC represents the use of the ResNet50 base network to experiment with simulated crater dataset. Therefore, the name of after ‘-’ represents used the dataset, and before ‘-’ represents the used method.

5.5.

Qualitative Results on a Visible-Light Polarization Image

Our trained multimodal fusion detection model was tested on the images of the simulated crater dataset. Figure 7 shows the detection results of a visible-light polarization image. The red boxes represent the true position and size of the crater, and the blue boxes indicate the coordinates predicted by various models. We selected a visible-light polarization image taken on sand for detection. There is a clear difference between the target and background in images taken of craters on soil and grass, but the sand is light in color and reflective, so a target in sand is difficult to accurately detect. However, the use of this challenging image better demonstrates that our method can accurately detect targets in images.

Fig. 7

The test results of different models on the same image.

Figure 7(a) shows the original image of the simulated crater, where the crater is located in the top right of the image. Figure 7(b) shows the ground truth. Figures 7(c) and 7(d) both show the prediction results of a single-modal method. In these images, the red and blue boxes do not overlap much, indicating that the prediction results are poor. Figures 7(e) and 7(f) are the results of our method. Here, the overlapping parts of the two boxes are large. These figures show that the multimodal fusion detection model using the ResNet-based network better detects the simulated crater. These results demonstrate that our proposed multimodal fusion detection networks can take full advantage of multiple types information provided by visible-light polarization images. Of these proposed models, the lightweight multimodal fusion detection model yields the best detection performance on the simulated crater dataset.