|
The detection of chemical agents with hyperspectral longwave infrared sensors is a difficult problem with many civilian and military applications. System performance can be evaluated by comparing the detected gases in each pixel with the ground truth for each pixel using a confusion matrix. In the presence of chemical mixtures the confusion matrix becomes extremely large and difficult to interpret due to its size. We propose summarizing the confusion matrix using simple scalar metrics tailored for specific applications. Ideally, an identifier should determine exactly which chemicals are in each pixel, but in many applications it is acceptable for the output to contain additional chemicals or lack some constituent chemicals. A performance metric for identification problems should give partially correct results a lower weight than completely correct results. The metric we propose using, the Dice metric, weighs each output by its similarity with the truth for each pixel, thereby giving less importance to partially correct outputs, while still giving full scores only to exactly correct results. Using the Dice metric we evaluated the performance of two identification algorithms: an adaptive cosine estimator (ACE) detector bank approach, and Bayesian model averaging (BMA). Both algorithms were tested individually on real background data with synthetically embedded plumes; performance was evaluated using standard detection performance metrics, and then using the proposed identification metric. We show that ACE performed well as a detector but poorly as an identifier; however, BMA performed poorly as a detector but well as an identifier. Cascading the two algorithms should lead to a system with a substantially lower false alarm rate than using BMA alone, and much better identification performance than the ACE detector bank alone.
|
