Evaluation methodology for query-based scene understanding systems

Todd P. Huster; Timothy D. Ross; Jared L. Culbertson

doi:10.1117/12.2177007

22 May 2015 Evaluation methodology for query-based scene understanding systems

Todd P. Huster, Timothy D. Ross, Jared L. Culbertson

Proceedings Volume 9476, Automatic Target Recognition XXV; 947605 (2015) https://doi.org/10.1117/12.2177007
Event: SPIE Defense + Security, 2015, Baltimore, MD, United States

Abstract

In this paper, we are proposing a method for the principled evaluation of scene understanding systems in a query-based framework. We can think of a query-based scene understanding system as a generalization of typical sensor exploitation systems where instead of performing a narrowly defined task (e.g., detect, track, classify, etc.), the system can perform general user-defined tasks specified in a query language. Examples of this type of system have been developed as part of DARPA’s Mathematics of Sensing, Exploitation, and Execution (MSEE) program. There is a body of literature on the evaluation of typical sensor exploitation systems, but the open-ended nature of the query interface introduces new aspects to the evaluation problem that have not been widely considered before. In this paper, we state the evaluation problem and propose an approach to efficiently learn about the quality of the system under test. We consider the objective of the evaluation to be to build a performance model of the system under test, and we rely on the principles of Bayesian experiment design to help construct and select optimal queries for learning about the parameters of that model.

Citation Download Citation

Todd P. Huster, Timothy D. Ross, and Jared L. Culbertson "Evaluation methodology for query-based scene understanding systems", Proc. SPIE 9476, Automatic Target Recognition XXV, 947605 (22 May 2015); https://doi.org/10.1117/12.2177007

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available