From image captioning to video summary using deep recurrent networks and unsupervised segmentation

Bogdan-Andrei Morosanu; Camelia Lemnaru

doi:10.1117/12.2310071

13 April 2018 From image captioning to video summary using deep recurrent networks and unsupervised segmentation

Bogdan-Andrei Morosanu, Camelia Lemnaru

Proceedings Volume 10696, Tenth International Conference on Machine Vision (ICMV 2017); 106960P (2018) https://doi.org/10.1117/12.2310071
Event: Tenth International Conference on Machine Vision, 2017, Vienna, Austria

Abstract

Automatic captioning systems based on recurrent neural networks have been tremendously successful at providing realistic natural language captions for complex and varied image data. We explore methods for adapting existing models trained on large image caption data sets to a similar problem, that of summarising videos using natural language descriptions and frame selection. These architectures create internal high level representations of the input image that can be used to define probability distributions and distance metrics on these distributions. Specifically, we interpret each hidden unit inside a layer of the caption model as representing the un-normalised log probability of some unknown image feature of interest for the caption generation process. We can then apply well understood statistical divergence measures to express the difference between images and create an unsupervised segmentation of video frames, classifying consecutive images of low divergence as belonging to the same context, and those of high divergence as belonging to different contexts. To provide a final summary of the video, we provide a group of selected frames and a text description accompanying them, allowing a user to perform a quick exploration of large unlabeled video databases.

Citation Download Citation

Bogdan-Andrei Morosanu and Camelia Lemnaru "From image captioning to video summary using deep recurrent networks and unsupervised segmentation", Proc. SPIE 10696, Tenth International Conference on Machine Vision (ICMV 2017), 106960P (13 April 2018); https://doi.org/10.1117/12.2310071

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
8 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Image segmentation

Video

Data modeling

Statistical modeling

Image classification

Image processing algorithms and systems

Image processing

RELATED CONTENT

Scenic classification methods for image and video databases
Proceedings of SPIE (November 21 1995)

Multilevel model for 2D human motion analysis and description
Proceedings of SPIE (January 10 2003)

Multiscale document segmentation using wavelet-domain hidden Markov models
Proceedings of SPIE (December 22 1999)

A new online learning algorithm with application to image segmentation
Proceedings of SPIE (March 01 2005)

Model guided diffeomorphic demons for atlas based segmentation
Proceedings of SPIE (March 12 2010)

Fast block-based image segmentation for natural and texture images
Proceedings of SPIE (May 30 2000)

An improved video matting technique based on GrabCut
Proceedings of SPIE (July 08 2011)

Subscribe to Digital Library

Receive Erratum Email Alert

Keywords/Phrases

Search In:

Publication Years