AVR: attention based salient visual relationship detection

Hualiang Li; Jianming Lv; Qinzhe Xiao; Yuzhong Liu; Siqi Wu; Xingjie Huang

doi:10.1117/12.2644438

12 October 2022 AVR: attention based salient visual relationship detection

Hualiang Li, Jianming Lv, Qinzhe Xiao, Yuzhong Liu, Siqi Wu, Xingjie Huang

Proceedings Volume 12342, Fourteenth International Conference on Digital Image Processing (ICDIP 2022); 123422C (2022) https://doi.org/10.1117/12.2644438
Event: Fourteenth International Conference on Digital Image Processing (ICDIP 2022), 2022, Wuhan, China

Abstract

Visual relationship detection aims to locate objects in images and recognize the relationships between objects. Traditional methods treat all observed relationships in an image equally, which causes a relatively poor performance in the detection tasks on complex images with abundant visual objects and various relationships. To address this problem, we propose an attention based model, namely AVR, to achieve salient visual relationships based on both local and global context of the relationships. Specifically, AVR recognizes relationships and measures the attention on the relationships in the local context of an input image by fusing the visual features, semantic and spatial information of the relationships. AVR then applies the attention to assign important relationships with larger salient weights for effective information filtering. Furthermore, AVR is integrated with the priori knowledge in the global context of image datasets to improve the precision of relationship prediction, where the context is modeled as a heterogeneous graph to measure the priori probability of relationships based on the random walk algorithm. Comprehensive experiments are conducted to demonstrate the effectiveness of AVR in several real-world image datasets, and the results show that AVR outperforms state-of-the-art visual relationship detection methods significantly by up to 87.5% in terms of recall.

Citation Download Citation

Hualiang Li, Jianming Lv, Qinzhe Xiao, Yuzhong Liu, Siqi Wu, and Xingjie Huang "AVR: attention based salient visual relationship detection", Proc. SPIE 12342, Fourteenth International Conference on Digital Image Processing (ICDIP 2022), 123422C (12 October 2022); https://doi.org/10.1117/12.2644438

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available