Generalizations of the Jaccard index and Sørensen index for assessing agreement across multiple readers in object detection and instance segmentation in biomedical imaging

Madeleine S. Durkee; Kyle Lleras; Karen Drukker; Junting Ai; Thao Cao; Gabriel Casella; Deepjyoti Ghosh; Marcus R. Clark; Maryellen L. Giger

doi:10.1117/1.JMI.10.6.065503

14 December 2023 Generalizations of the Jaccard index and Sørensen index for assessing agreement across multiple readers in object detection and instance segmentation in biomedical imaging

Madeleine S. Durkee, Kyle Lleras, Karen Drukker, Junting Ai, Thao Cao, Gabriel Casella, Deepjyoti Ghosh, Marcus R. Clark, Maryellen L. Giger

Author Affiliations +

Journal of Medical Imaging, Vol. 10, Issue 6, 065503 (December 2023). https://doi.org/10.1117/1.JMI.10.6.065503

Abstract

Significance

Manual annotations are necessary for training supervised learning algorithms for object detection and instance segmentation. These manual annotations are difficult to acquire, noisy, and inconsistent across readers.

Aim

The goal of this work is to describe and demonstrate multireader generalizations of the Jaccard and Sørensen indices for object detection and instance segmentation.

Approach

The multireader Jaccard and Sørensen indices are described in terms of “calls,” “objects,” and number of readers. These generalizations reduce to the equations defined by confusion matrix variables in the two-reader case. In a test set of 50 cell microscopy images, we use these generalizations to assess reader variability and compare the performance of an object detection network (Yolov5) and an instance segmentation algorithm (Cellpose2.0) with a group of five human readers using the Mann–Whitney U-test with Bonferroni correction for multiplicity.

Results

The multireader generalizations were statistically different from the mean of pairwise comparisons of readers (p < 0.0001). Further, these multireader generalizations informed when a reader was performing differently than the group. Finally, these generalizations show that Yolov5 and Cellpose2.0 performed similarly to the pool of human readers. The lower bound of the one-sided 90% confidence interval for the difference in the multireader Jaccard index between the pool of human readers and the pool of human readers plus an algorithm were −0.019 and −0.016 for Yolov5 and Cellpose2.0, respectively.

Conclusions

Multireader generalizations of the Jaccard and Sørensen indices provide metrics for characterizing the agreement of an arbitrary number of readers on object detection and instance segmentation tasks.

CC BY: © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 International License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Citation Download Citation

Madeleine S. Durkee, Kyle Lleras, Karen Drukker, Junting Ai, Thao Cao, Gabriel Casella, Deepjyoti Ghosh, Marcus R. Clark, and Maryellen L. Giger "Generalizations of the Jaccard index and Sørensen index for assessing agreement across multiple readers in object detection and instance segmentation in biomedical imaging," Journal of Medical Imaging 10(6), 065503 (14 December 2023). https://doi.org/10.1117/1.JMI.10.6.065503

Received: 10 August 2023; Accepted: 1 December 2023; Published: 14 December 2023

ACCESS THE FULL ARTICLE

JOURNAL ARTICLE
17 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY