Two-stage video scene segmentation method based on multimodal semantic interaction

Tengfei Wang; Yihui Liao

doi:10.1117/12.3055769

2 December 2024 Two-stage video scene segmentation method based on multimodal semantic interaction

Tengfei Wang, Yihui Liao

Proceedings Volume 13443, Fifth International Conference on Computer Vision and Information Technology (CVIT 2024); 1344303 (2024) https://doi.org/10.1117/12.3055769
Event: 2024 5th International Conference on Computer Vision and Information Technology (CVIT 2024), 2024, Beijing, China

Abstract

This paper proposes a two-stage video scene segmentation method based on multimodal semantic interaction. The method divides the video scene segmentation task into two stages: shot audio-visual representation and multimodal scene segmentation. In the first stage, the method leverages the high correlation and complementarity between audio-visual information by using an interactive attention module to deeply explore audio-visual semantic information. Simultaneously, it introduces a self-supervised learning strategy to improve the model's generalization ability by utilizing the temporal structure characteristics of scenes. In the second stage, the method constructs a multimodal feature fusion module, learning a unified shot representation from the audio-visual representation based on the attention mechanism. Additionally, it builds a visual discrimination loss to regulate the influence of audio-visual features, further enhancing the discriminative power of shot representation. Experimental results on the MovieNet benchmark dataset show that the proposed method can achieve more accurate video scene segmentation.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Tengfei Wang and Yihui Liao "Two-stage video scene segmentation method based on multimodal semantic interaction", Proc. SPIE 13443, Fifth International Conference on Computer Vision and Information Technology (CVIT 2024), 1344303 (2 December 2024); https://doi.org/10.1117/12.3055769

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available