Open Access
30 August 2023 Synthetic aperture radar and optical image registration using local and global feature learning by modality-shared attention network
Xin Hu, Yan Wu, Zhikang Li, Xiaoru Zhao, Xingyu Liu, Ming Li
Author Affiliations +
Abstract

The registration of synthetic aperture radar (SAR) and optical images is a meaningful but challenging multimodal task. Due to the large radiometric differences between SAR and optical images, it is difficult to obtain discriminative features only by mining local features in the traditional Siamese convolutional networks. We propose a modality-shared attention network (MSA-Net) that introduces nonlocal attention (NLA) to the partially shared two-stream network to jointly exploit local and global features. First, a modality-specific feature learning module is designed to efficiently extract shallow modality-specific features from SAR and optical images. Subsequently, a modality-shared feature learning (MShFL) module is designed to extract deep modality-shared features. The local feature extraction module and the NLA module in MShFL extract deep local and global features to enrich feature representations. Furthermore, a triplet loss function with a cross-modality similarity constraint is constructed to learn modality-shared feature representations, thereby reducing nonlinear radiometric differences between the two modalities. The MSA-Net is trained on a public SAR and optical dataset and tested on five pairs of SAR and optical images. In the registration results of five pairs of test SAR and optical images, the matching rate of the MSA-Net is 5% to 15% higher than that of other compared methods, and the matching errors of the matched inliers are on average reduced by about 0.28. Several ablation experiments verify the effectiveness of the partially shared network structure, the MShFL module, and the cross-modality similarity constraint.

CC BY: © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 International License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.
Xin Hu, Yan Wu, Zhikang Li, Xiaoru Zhao, Xingyu Liu, and Ming Li "Synthetic aperture radar and optical image registration using local and global feature learning by modality-shared attention network," Journal of Applied Remote Sensing 17(3), 036504 (30 August 2023). https://doi.org/10.1117/1.JRS.17.036504
Received: 5 October 2022; Accepted: 8 August 2023; Published: 30 August 2023
Advertisement
Advertisement
KEYWORDS
Synthetic aperture radar

Image registration

Feature extraction

Radio optics

Mining

Education and training

Image fusion

Back to Top