2.1.1.

Original dataset

The original dataset consisted of 9860 patients retrospectively collected from the University of Chicago Medicine. This cohort was split at the patient level into 64% for training, 16% for validation, and 20% for testing using stratified sampling. COVID-19 prevalence was held constant across the three subsets (15.5%). Only the first CXR exam acquired within 2 days after a patient’s initial RT-PCR test for the SARS-CoV-2 virus was input to the model. CXRs had been acquired between January 30, 2020 and February 3, 2021; both standard and soft-tissue images were collected from stationary DES radiography units and portable radiography units (i.e., CXR exam type). The portable units generated soft-tissue images using post-processing algorithms. More information regarding this dataset can be found in Hu et al.;¹³ the test set from this study will be called the “original test set.”

2.1.2.

Current test set

Images that comprised the “current test set” were retrospectively collected from 5893 patients between March 15, 2020 and January 1, 2022 under a Health Insurance Portability and Accountability Act (HIPAA)-compliant, Institutional Review Board-approved protocol. Among these patients, 731 (12.4%) had tested positive and 5162 (87.6%) had tested negative for the SARS-CoV-2 virus. The current test set served only to assess the performance of the pre-trained model: no additional training or validation was performed. Patient images from the previous study and the current study were acquired from the same institution and were preprocessed in the same manner. Overall, the current test set followed the same curation process as the original dataset to minimize the impact of any confounding variables. A summary of the datasets is presented in Tables 1 and 2, where Table 2 categorizes the three manufacturers that were used to acquire the standard CXR images for both test sets: Canon Inc., GE Healthcare, and Fujifilm Corporation.

Table 1

Summary of datasets used, categorized by various factors, including type of units used to acquire the images.

Table 2

Number of patients categorized by manufacturer and CXR exam type for the original and current test set.

2.4.1.

Current test set only

Performance of the three classification algorithms was assessed by CXR exam type (portable versus DES) on the current test set.

2.4.2.

Original test set versus current test set

Comparisons of model performance between the original and current test sets were performed for standard CXRs when considering (1) the entirety of the two test sets (this comparison was repeated for the soft-tissue CXRs and fusion of the image types), (2) the original test set and only CXRs from the current test set acquired within the date range of the original test set (to control for the different virus variants), and (3) nonimmunized patients from both test sets (to control for the impact of disease severity due to vaccines).

The ability of the DL model to classify disease will depend on the severity of the disease as presented on a medical image. Therefore, the COVID severity of the CXRs in the original and current test sets was calculated, as studied by Li, et al.²⁴ Briefly, the COVID severity model computed a pulmonary x-ray severity (PXS) score, which is defined as the median Euclidean distance between the image of interest and “normal” images (i.e., absence of all pathologies²⁵). Specifically, the Euclidean distance, with respect to the imaging features on which the networks trained, was between the final two layers of twinned DenseNet-121 networks within a Siamese neural network. On a subset of 50 cases from the original test set chosen using stratified sampling, the manual modified radiographic assessment of lung edema scores (mRALE, based on the RALE score created by Warren et al.²⁶) were determined by a radiologist with over 20 years of experience. The PXS scores were assessed using a Bland–Altman plot²⁷ to display the agreement between the computed PXS and mRALE scores. The Spearman’s rank correlation coefficient was also calculated to assess the monotonic relationship between the PXS and mRALE scores. Model performance based on PXS score was evaluated after grouping the cases into four equally spaced bins. Due to the small numbers in the fourth bin for each of the test sets, however, the cases for the third and fourth bins were combined, resulting in three bins for analysis. The “obviousness” of each case was also qualitatively assessed by plotting the DL prediction scores of the cropped standard images from the original and current test sets.

Uniform manifold approximation projection (UMAP²⁸) was used to visualize the penultimate global average pooling layer of the standard model to qualitatively evaluate the COVID-19 classification task and to visualize the confusion matrix. A quantitative comparison of the UMAPs was performed using one-way multivariate analysis of variance (MANOVA) to test for a significant difference between the two bivariate means of the UMAPs; specifically, the bivariate means were tested for statistical significance using Pillai’s trace to calculate the F-statistic, which then resulted in its corresponding p-value. In addition, a comparison of demographics (i.e., age and sex distributions) between patient cohorts was performed, and an analysis of standard model performance with images of the various image dimensions previously described of the image preprocessing steps was conducted on the original test set.

Overall, these investigations were designed to better understand potential confounding factors that might have occurred during image acquisition, different variants of the COVID-19 virus, and patient age. Furthermore, clinical factors, such as vaccination status and severity of COVID, were assessed. Performance was evaluated using area under the ROC curve as the figure of merit (2000 bootstrapped samples to construct the 95% confidence intervals), with the DeLong test used to compare the uncorrelated ROC curves.²⁹

3.1.1.

Classification algorithm

The three classification algorithms corresponded to training the model with the original dataset using (1) cropped standard images, (2) cropped soft-tissue images, (3) and a feature fusion of the two image types. The pre-trained (i.e., no additional training or validation) DL model then was applied to the current test set and achieved the AUC values presented in Fig. 2.

Fig. 2

ROC curves for the classification of COVID-positive and COVID-negative patient CXRs using the current test set. No additional training or validation was performed. ROC curves were fitted using software as defined by Metz et al.³¹

The AUC value obtained using the cropped standard CXR images (0.67 [0.65, 0.70]) was significantly higher than that obtained using soft-tissue CXRs (0.65 [0.63, 0.67]). AUC values, however, failed to achieve statistical significance when comparing (1) cropped standard CXRs with both types of images (fusion) and (2) soft-tissue CXRs with both types of images. The results are displayed in Table 3.

Table 3

Comparisons of model performance for the three classification algorithms: standard images, soft-tissue images, or both types. P-values comparing the differences in AUC values were calculated using the DeLong test, with their corresponding confidence intervals (CIs), prior to multiple comparisons correction.30 Significance levels (α) and widths of the CIs were adjusted based on multiple comparisons.

3.1.2.

CXR exam type

When considering the CXR exam type (e.g., portable versus a DES unit), the AUC values achieved are shown in Table 4. AUC values failed to achieve statistical significance when comparing across CXR exam type for each classification algorithm, i.e., the type of unit did not appear to have an impact on the model’s performance in the task of classifying COVID-19.

Table 4

Model performance categorized by the different image types (classification algorithms) as acquired from the different radiography units for the current test set. 95% CIs are displayed in brackets. Majority of portable images were acquired on Canon Inc. units and majority of DES images were acquired on GE Healthcare and Fujifilm Corporation units.

3.2.1.

Entirety of both test sets

All AUC values obtained in Fig. 2 were lower than those obtained in Hu, et al.¹³ ($p<0.001$) as calculated using the DeLong test for uncorrelated ROC curves. Figure 3 shows the comparison of ROC curves and values between the two test sets.

Fig. 3

Comparison of ROC curves and AUC values between (a) original and (b) current test sets. AUC values for the three classification algorithms were consistently lower for the current test set compared with the original. Panel (a) is the same figure as Fig. 6 in Ref. 13 (reprinted with permission) and panel (b) is the same as Fig. 2.

3.2.2.

Date match

There were four main COVID-19 variants of concern (VOC) that underlie the datasets (as defined by the City of Chicago Department of Public Health):³² the original strain, B.1.1.7 (Alpha) beginning in January 2021 until June 2021, Delta from July 2021 to December 2021, and Omicron in December 2021 onward. Therefore, the original dataset had two VOC: the original variant and Alpha. The current test set had all four VOC. Table 5 displays the AUC values acquired when controlling for the variants.

Table 5

AUC values calculated for each of the four variants underlying the two test sets.

The original and Alpha VOC were controlled for when the current test set was limited to the image acquisition date range of the original test set. An AUC value of 0.66 [0.62, 0.70] (significantly different from the AUC of the original test set, $p<0.001$) was achieved when considering cropped standard CXRs acquired within the same image acquisition dates as the original test set. This date match corresponds to the overlap of the green histogram bars with the blue as shown in Fig. 4.

Fig. 4

Histogram of the imaging exam dates, categorized by the current test set, the original training set, and the original test set. The current test set had a much larger date range, spanning March 15, 2020 to January 1, 2022.

3.2.3.

Immunization status

The current test set achieved an AUC value lower than that of the original test set when comparing the cropped standard CXRs of nonimmunized patients from the two patient cohorts, as shown in Table 6. Further, to determine whether immunization status had an impact on the DL model’s performance in classifying COVID-19 status, a comparison between immunized and nonimmunized patients in the current test set was performed; differences in AUC values failed to achieve statistical significance ($p=0.60$).

Table 6

Summary of statistical analyses performed between the original test set and the current test set. AUC comparisons were conducted using the unpaired Delong test and all three comparisons between the test sets were statistically significantly different (p<0.001).

3.2.4.

COVID severity

COVID severity was assessed for both patient cohorts; differences between the PXS scores failed to achieve statistical significance ($p=0.17$). PXS scores also failed to achieve a statistically significant difference when matching the image acquisition dates between the test sets ($p=0.06$). The robustness of the severity score itself was evaluated based on the 50 cases selected in Fig. 5 using the Spearman’s rank correlation coefficient ($\rho =0.74$, $p<0.001$) as well as a Bland–Altman plot to display agreement (Fig. 6). The Bland-Altman plot showed that the PXS score (calculated using the standard CXRs) was on average lower than the radiologist’s assessment [Fig. 6(b)].

Fig. 5

Stratified sampling of the 50 random cases from the original test set to determine the robustness of the PXS scores. 10 cases were randomly chosen from each part of the box and whisker plot to ensure an equal representation of cases from all possible PXS scores assigned.

Fig. 6

(a) A scatter plot of a subset of images (from the original test set) displaying the linear correlation between the mRALE score calculated by the radiologist and the COVID severity as determined by the DL model described by Li et al.,²⁴ where the shaded blue region is the 95% CI. (b) Bland–Altman plot displaying the agreement between the methods of assessing COVID severity. The outlier outside the 95% boundary was a collapsed left lung with possible effusion.¹²

AUC values resulting from cases in the three PXS score bins for the test sets (details presented in Table 7) are displayed in Fig. 7. The highest AUC value for the original test set was achieved for the third bin, which contained cases with the highest COVID prevalence (45.05%). The highest AUC value for the current test set resulted from the second bin, which contained cases with the second highest COVID prevalence (35.09%). Though, the 95% CIs increased with bin edges since the number of cases decreased. The COVID prevalence increased with increasing bin edges, predictably, as higher PXS scores indicate greater radiographic evidence of abnormality. Overall, the original test set yielded AUC values consistently larger than those of the current test set. The distribution of severity scores split according to positive and negative cases, per test set, is shown in Fig. 8.

Table 7

Definition of PXS score bins for the test sets.

Fig. 7

Bar plot depicting the resulting AUC values when controlling for PXS scores using histogram bins. The 95% CIs were calculated by bootstrapping the AUC values 2000 times.

Fig. 8

Histogram plots of the original (a) and current (b) test sets. Both distributions have strong right skews with higher frequency of positive cases having larger PXS scores.

3.2.5.

DL model predictions

Figure 9 presents histograms displaying the prediction scores assigned by the model for the original and current test sets. The distributions display a higher proportion of images from the original test set at low ($\le 0.25$) and high ($\ge 0.75$) prediction scores relative to the images from the current test set, which resulted in the observed higher performance on the original test set. There was a slightly higher count for the current test set at scores in the middle of the plot, i.e., less certain predictions assigned by the DL model.

Fig. 9

Histogram of the prediction scores of the DL model for both test sets. The distribution (a) before February 3, 2021 and (b) the entire data range. February 3, 2021 was chosen as the cutoff date as that is the last date which had an overlap of CXR acquisitions between the two patient cohorts (see Fig. 4). The histograms were normalized to have equal area.

3.2.6.

UMAP visualization

UMAP visualizations indicated that the model perceived the two sets of CXRs nearly identically (Fig. 10). This observation was supported by the MANOVA analysis, which generated an F-statistic of 1.9014 and a $p$-value of 0.1494, failing to achieve a statistically significant difference between the two bivariate means of the UMAPs generated for the original test set and for the current test set. However, variation existed as the percentages of true positives (TPs) and false positives (FPs) were different between the original test set (TP = 8%, FP = 11.9%) and the current test set (TP = 4.8%, FP = 12.9%).

Fig. 10

UMAP visualization of the confusion matrix for (a) the original test set and (b) the current test set. A similar decision variable was chosen by the deep net for both patient cohorts, classifying positive cases from negative cases (division between blue and orange dots). Overall, the model returned a higher percentage of TPs and lower percentage of FPs for the original test set than for the current test set.¹²

3.2.7.

Patient demographics

Despite the nearly identical mean and median ages between the two test sets (Table 8), the distributions of age (Fig. 11) yielded statistically significant differences based on the Wilcoxon rank-sum test ($p=0.006$).

Table 8

Statistics of the age distributions for the original dataset (which includes training, validation, and test cases) and the current test set.

Fig. 11

Histogram of patient age, categorized by the original dataset (which includes training, validation, and test cases) and current test set.¹²

There were more male patients than female in the original dataset [5088 men and 4772 women (52% male)] and current test set [2979 men and 2914 women (51%)]. Table 9 summarizes AUC values obtained when dividing the test sets with respect to sex. Statistical differences occurred ($p<0.05$) when comparing the AUC values of the original test set to the current test. There were no significant differences within each test set.

Table 9

Distributions of sex for the original and the current test set with their corresponding AUC values and COVID prevalence.

3.2.8.

Image dimensions

When using CXRs from the original test set that were resized by the U-Net-based model to the large lung region dimensions (bottom panel in Fig. 1), the performance of the DL model decreased for all three classification algorithms relative to the published original test set results [Fig. 3(a)], obtaining AUC values of 0.74 [0.71, 0.77] for standard CXRs alone, 0.72 [0.69, 0.75] for soft-tissue CXRs alone, and 0.69 [0.66, 0.73] for both types of images. The feature fusion algorithm experienced the largest decrease in AUC values.

To determine the impact of image resizing further, the large lung region images were resized to the small lung region dimensions using the Image module from the PIL Python package. This additional resizing resulted in AUC values closer to those obtained from the original test set analysis: 0.76 [0.73, 0.79] for the standard CXRs alone, 0.72 [0.69, 0.76] for the soft-tissue CXRs alone, and 0.75 [0.72, 0.79] for both types of images. Once more, the classification algorithm incorporating both types of images experienced the largest decrease in classification performance. While the images in this additional analysis were resized to the smaller dimensions using the same Python packages as the U-Net-based model, the cropped images were not identical to the original test set as determined by evaluating the pixel values. The value of U-Net cropping was demonstrated using uncropped standard CXRs of the current test set, for which the model achieved an AUC value of 0.58 [0.56, 0.61], substantially lower than the 0.67 reported throughout this work.