A fusion algorithm for infrared and visible images based on the dual-branch of CNN and transformer

Zhuang Wu; Weifeng Liu; Xueen Wang

doi:10.1117/12.3004972

20 October 2023 A fusion algorithm for infrared and visible images based on the dual-branch of CNN and transformer

Zhuang Wu, Weifeng Liu, Xueen Wang

Proceedings Volume 12916, Third International Conference on Signal Image Processing and Communication (ICSIPC 2023); 129160S (2023) https://doi.org/10.1117/12.3004972
Event: Third International Conference on Signal Image Processing and Communication (ICSIPC 2023), 2023, Kunming, China

Abstract

Infrared images provide brightness information of targets, while visible images provide detailed information about targets. The fusion of infrared and visible images can provide the most comprehensive description of the scene. Existing deep learning-based fusion methods mostly utilize convolutional neural networks (CNNs) to extract local features for fusion, but they do not exploit the global contextual relationships of the images. On the other hand, using a complete Transformer to process the segmented image does not leverage the local features effectively. In this paper, we propose a dual-branch fusion algorithm combining CNN and Transformer. The dual-branch adopts a parallel structure, allowing for maximum utilization of both local features and global representations. By processing the sequence features generated by the Transformer branch, we ensure their resolution and dimension are consistent with the local features for fusion. The fusion strategy employs a combination of L1-norm and averaging, and our model is trained in two stages. Comparative experiments with various fusion algorithms demonstrate that the proposed algorithm achieves good fusion results and shows improvement in objective metric evaluations.

(2023) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Zhuang Wu, Weifeng Liu, and Xueen Wang "A fusion algorithm for infrared and visible images based on the dual-branch of CNN and transformer", Proc. SPIE 12916, Third International Conference on Signal Image Processing and Communication (ICSIPC 2023), 129160S (20 October 2023); https://doi.org/10.1117/12.3004972

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
7 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Image fusion

Transformers

Feature extraction

Infrared radiation

Infrared imaging

Visible radiation

Feature fusion

Show All Keywords

Keywords/Phrases

Search In:

Publication Years