2.1.

IVOCT Data Acquisition and Preparation for Training and Testing

The data used for this study comprise IVOCT-acquired images of patients diagnosed with coronary artery disease. The IVOCT images were acquired from the University of Malaya Medical Center (UMMC) catheterization laboratory using two standard clinical systems: Illumien and Illumien Optis IVOCT Systems (St. Jude Medical). Both systems have an axial resolution of $15\mu \mathrm{m}$ and a scan diameter of 10 mm. The Ilumien system and the Ilumien Optis system have maximum frame rates of 100 and 180 fps, respectively. The study was approved by the University of Malaya Medical Ethics Committee (Ref: 20158-1554), and all patient data were anonymized.

In total 64 pullbacks were acquired from 28 patients [25%/75% male/female, with mean age 59.71 ($\pm 9.61$) years] using Dragonfly^™ Duo Imaging Catheter with 2.7 F crossing profile when the artery was under contrast flushing (Iopamiro^® 370). The internal rotating fiber optic imaging core performed rotational motorized pullback scans for a length of 54 or 75 mm in 5 s. These scans include multiple pre- and poststented images of the coronary artery at different locations. These pullbacks were randomly assigned to one of two groups with a ratio of 7:3, i.e., 45 pullbacks were randomly designated as training sets and the remaining 19 as test sets. Excluding images depicting only the guide catheter, each pullback contains between 155 and 375 polar images. These images contain a heterogeneous mix of images with the absence or presence of stent struts (metal stents or bioresorbable stents or both), fibrous plaques, calcified plaques, lipid-rich plaques, ruptured plaques, thrombus, dissections, motion artifacts, bifurcations, and blood artifacts. The original size of each pullback frame was $984\times 496\text{pixels}$ (axial × angular dimension), and was subsampled in both dimensions to $488\times 248\text{pixels}$ to reduce training and processing time. For each image, raw intensity values were converted from linear scale to logarithmic scale before normalizing by mean and standard deviation.

Gold-standard segmentations were generated on both training and test sets by manual frame-by-frame delineation using ImageJ²⁹ in Cartesian coordinates, according to the document of consensus,¹⁴ whereby a contour was drawn between the lumen and the leading edge of the intima. The contour was also manually drawn across the guidewire shadow and bifurcation at locations that best represent the underlying border of the main lumen, gauged by the adjacent slices. The manual contour of the lumen border for each image was subsequently converted to polar coordinates, smoothed and interpolated to 100 points using cubic B-spline interpolation method for CNN training and testing.

2.2.

Convolutional Neural Networks Regression Architecture and Implementation Details

Using our linear-regression CNN model, in each polar image we infer the radius parameter of the vessel wall at 100 equidistant radial locations, rather than the more conventional approach of classifying each pixel within the image. This has the advantage of avoiding the physiologically unrealistic results that may arise from segmentation of individual pixels. The lumen segmentation was parameterized in terms of radial distances from the center of the catheter in polar space.

The general flow of the proposed CNN model is shown in Fig. 1. Our network consists of a simple structure with four convolutional layers and three fully connected layers, including the final output layer. All polar images were padded circularly left and right before being windowed for input. The window dimension was $488\times 128\text{pixels}$ centered on each individual radial point, therefore yielding 100 inputs and 100 evaluated radial distances per image.

Fig. 1

Overview of the linear-regression CNN segmentation system (refer to text for details).

The details of the network architecture are presented in Table 1. In the network architecture, a filter kernel of size $5\times 5\times 24$ with boundary zero-padding was applied for all convolutional layers, yielding 24 feature maps at each layer. In the first layer, a stride of 2 was also applied along the angular dimension to reduce computational load. The first three layers were also max-pooled by size $2\times 2$. Each fully connected layer contains 512 nodes. Exponential linear units³⁰ were used as the activation functions for all layers, including both convolutional and fully connected layers, except the final layer. Dropout with keep probability of 0.75 was applied to the fully connected layers FC1 and FC2, to improve the robustness of the network.³¹ The final layer outputs a single value representative of the radial distance between the lumen border and the center of the catheter for the radial position being evaluated.

Table 1

Linear-regression CNN architecture for lumen segmentation at each windowed image. The output is the radial distance at the lumen border from the center of the catheter. CN, convolutional layer; FC, fully connected layer.

The objective function used for the network training is the standard mean-squared error. Starting from a random initialization, the weight and bias parameters are iteratively minimized by calculating the mean squared error between the gold-standard radial distance and the output of the CNN training. The Adam stochastic gradient algorithm was used to perform the optimization, i.e., minimization, of the objective function.³² The network was trained stochastically with a mini-batch size of 100 at a base-learning rate of 0.005. The learning rate was halved every 50,000 runs. The training was stopped at 400,000 runs where convergence was observed (i.e., when the observed losses had ceased to improve for at least 100,000 runs). The trained weights and biases of the network, amounting to $\sim 6.3$ million parameters, are subsequently used to predict the lumen contour on the test sets.

The neural network was designed in a Python (Python Software Foundation, Delaware) environment using the TensorFlow v1.0.1 machine learning framework (Google Inc., California). The execution of the network was performed on a Linux-based Intel i5-6500 CPU workstation with NVIDIA GeForce GTX1080 8GB GPU. The training time for 45 training sets was 13.8 h and the complete inference time for each test image was 40.6 ms.

2.3.

Validation

The accuracy of our proposed linear-regression CNN lumen segmentation was validated against the gold-standard segmentation of the test data pullback acquisitions, which were the aforementioned 19 manually delineated pullbacks. These pullbacks contain in total 5685 images. The accuracy was assessed in three ways: (1) on a point-by-point basis via distance error measure, (2) in the form of binary image overlaps, and (3) based on luminal area.

The first assessment involves point-by-point analysis on the 100 equidistant radial contour points from all images, whereby the mean absolute Euclidean distance error between the gold standard and predicted contours was computed for each image.

The second assessment was performed to evaluate the regions delineated as lumen. The amount of overlap between the binary masks as generated from the predicted contours and the corresponding gold standards was computed using the Dice coefficient and Jaccard similarity index.

The third assessment targeted at the luminal area, which is one of the clinical indices to locate and grade the extent of coronary stenosis for treatment planning. Luminal area was computed from the binary mask produced from the predicted contours and compared against the corresponding gold standard. We also performed a one-tailed Wilcoxon signed ranks test on the errors of the estimated luminal areas at the significance level of 0.001. Three-dimensional (3-D) surface models of the lumen wall were also generated for all pullbacks to facilitate visual comparison of the segmentation by manual contouring and automated contouring using the proposed CNN regression model.

2.4.

Dependency of Network Performance on Training Data Quantity

To understand the dependency of the network performance to the amount of training data required, we assessed the variation in accuracy of the 19 test pullbacks against different numbers of training datasets. Tests were performed with 10, 15, 20, 25, 30, 35, 40, and 45 pullbacks. The training pullbacks for each group were selected randomly. The number of training runs with different training sets was kept constant at 400,000 runs, with a similar base learning rate and learning rate decay protocol.

2.5.

Interobserver Variability Against Convolutional Neural Networks Accuracy

To quantify the allowable variation in segmentation, we performed an experiment to assess variation in the manual gold standard that would be generated by three independent observers.

One hundred images were selected randomly from five pullbacks of the test sets and the lumen manually delineated by three independent observers. The interobserver variability was assessed through Bland–Altman analysis, consistent with Celi and Berti in their study on the segmentation of coronary lesions.¹¹ Specifically, the signed differences among all possible corresponding pairs of luminal areas from all three observers were plotted against their mean area differences. Bland–Altman analyses were also performed on luminal areas evaluated by the CNN against the corresponding evaluation by all observers. These analyses provide an understanding of the total bias and limits of agreement (i.e., 95% confidence interval or $1.96\times$ standard deviation of the signed differences from the mean) among all observers themselves as well as between the CNN and the observers.

3.1.

Dependency of Network Performance on Training Data Quantity

The results assessing the impact of training data quantity on CNN accuracy are shown in Fig. 2. The value reported here is the mean positional accuracy of each point along the vessel wall. There was notable improvement in CNN accuracy with an increase in the training data quantity up until 25 training data sets. Beyond that, the mean absolute error per image varied little with increased data. However, the optimal CNN segmentation was obtained from training with the highest sample size, i.e., 45 pullbacks consisting of 13,342 training images, as summarized in Table 2. At 45 training pullbacks, the median of the mean absolute error per image as quantified using point-by-point analysis was 21.87 microns, whereas Dice coefficient and Jaccard similarity index were calculated as 0.985 and 0.970, respectively.

Fig. 2

Mean absolute error against different numbers of training data sets.

Table 2

Accuracy of CNN segmentation with 45 training pullbacks (n=13,342). The values are obtained based on the segmentation on 19 test pullbacks (n=5685).

Representative segmentation results are shown in Fig. 3. Apart from performing well on images with clear lumen border contrast Fig. 3(a), linear-regression CNN segmentation has shown robustness in segmenting images with inhomogenous lumen intensity (b), severe stenosis (c), blood residue due to suboptimal flushing (d)–(f), multiple reflections (g), embedded stent struts (h) and (i), malapposed metallic stent struts (j), malapposed bioresorbable stent struts (k), and minor side branches [(c), (i), and (l)]. Acceptable lumen segmentation was found at the shadow behind the guide wire and metallic stent struts across all images. Errors were observed to occur most frequently at major bifurcations (angle spanning $>\sim 90\mathrm{deg}$), where the appropriate boundary for segmenting the main vessel was ambiguous [Figs. 4(c)–4(d)]. Seventy-two percent of the 100 worst performing segmentations were found to contain major bifurcations and, at these locations, overestimation of the area of the main vessels was noted.

Fig. 3

Representative results from the test sets, (a) showing good segmentation from linear-regression CNN on images with good lumen border contrast, (b) inhomogenous lumen intensity, (c) severe stenosis, (d)–(f) blood swirl due to inadequate flushing, (g) multiple reflections (indicated by yellow arrow), (h) and (i) embedded metallic and bioresorbable stent struts due to restenosis, respectively, (j) malapposed metallic stent struts, (k) malapposed bioresorbable stent strut, and (l) minor side branch. Blue and red contours represent CNN segmentation and gold standard, respectively. Scale bar (a) represents 500 microns.

Fig. 4

Representative cases from the test sets, (a) and (b) showing reasonable lumen segmentation from linear-regression CNN on images with medium-sized bifurcations. (c) and (d) Poorer results were seen at major bifurcations, where the appropriate boundary for segmenting the main vessel was ambiguous. Blue and red contours represent CNN segmentation and gold standard, respectively. Scale bar (a) represents 500 microns.

Based on the results obtained with the optimal training quantity (45 pullback data sets), we calculated luminal area estimates in all 19 test pullbacks, as tabulated in Table 3. CNN segmentation yields median (interquartile range) luminal area of 5.28 (3.88, 7.45) ${\mathrm{mm}}^2$ matching well with the results of manual segmentation of 5.26 (3.93, 7.45) ${\mathrm{mm}}^2$ (i.e., gold standard). The median (interquartile range) absolute error of luminal area was 1.38%, which is statistically significantly below 2% ($p<0.001$) as tested by the one-tailed Wilcoxon signed rank test. Figure 5 shows two representative examples of the 3-D reconstructed vessel wall from two different pullbacks for visual comparison of CNN regression (middle column) against gold-standard manual (left column) segmentation. The vessel wall was color-coded with the cross-sectional luminal area. Difference in luminal area between CNN regression and gold-standard segmentation is color-coded on the vessel wall on the right column.

Table 3

Luminal area in 19 test pullbacks with optimal training.

Fig. 5

Reconstruction of vessel wall from two different pullbacks [(a) and (b)] for visual comparison of CNN regression segmentation against the gold-standard manual segmentation. Vessel walls (left and middle columns) are color-coded with cross-sectional luminal area. Difference in luminal area is displayed on the right. The axis is in mm and color bar indicates luminal area in ${\mathrm{mm}}^2$.

3.2.

Interobserver Variability Against Convolutional Neural Networks Accuracy

The Bland–Altman analysis among all three observers showed a bias (mean signed difference) of $0.0{\mathrm{mm}}^2$ and limits of agreement of $\pm 0.599{\mathrm{mm}}^2$ in terms of luminal area estimation [Fig. 6(a)]. Comparing the CNN with all observers, the bias was $0.057{\mathrm{mm}}^2$ and the variability in terms of limits of agreement was comparable at $\pm 0.665{\mathrm{mm}}^2$ [Fig. 6(b)]. These results suggest that automated segmentation had sub-100 micron bias to over-estimate luminal area, and that the variation between automated and manual estimates of luminal area was only slightly greater than the interobserver variability among human observers.

Fig. 6

Bland–Altman plot analysis of luminal area for all possible (a) pair-comparisons among different observers and (b) between CNN and observers for the 100 randomly selected images from the test set.