2.1

Deformation parameters estimation

Inspired by literature,¹¹ a changed 2D-UNet network with deep supervision¹² is selected as the front network to generate deformation parameters θ₁, θ₂, …, θ_k(k ≥ 1) in decoder path, and it can be substituted by other suitable network structures. Since convolutional layers of different depths have different receptive fields, different from Unet,¹³ this network simultaneously outputs the learned deformation parameters from features extracted at different scales. For the specific coronary artery correction task in this paper, the features of three different scales are selected to estimating deformation parameters at the same time and calculate loss respectively, and finally the total loss is calculated by the combination of the three losses. This configuration can not only increase the stability of the network during training, but also support pruning the network during testing, which can increase the testing speed while ensuring the correction accuracy, thereby reducing the amount of network parameters within a controllable range.

2.2

CT Value Conservation Network based on Spatial Transformer (CTVC-STN)

In order to realize CT value conservation while ensuring the deformation, CTVC-STN network is proposed by improving on the basis of Spatial transformer Network as STN has shown some deficiencies in the end-to-end training of motion correction. First, STN introduces full connection layers to output an affine matrix θ with the size of 2 × 3 which increases the difficulty of training and limits that STN can only be used for small-size features.Secondly, STN uses conventional affine transformation and interpolation function to warp, in which the interpolation function can be bilinear interpolation, bicubic interpolation and thin plate spline interpolation. However, for coronary images with artifacts caused by different motion patterns, the simulated data as Fig.2 show that, ideally, the sum of CT value of the stationary and motion reconstructions are conserved. Therefore, this paper designs a deformation interpolation network CTVC-STN to keep the CT value conserved.

Figure 2:

One simulated case of the stationary reconstruction and motion reconstruction.The first image is the result of stationary reconstruction, and the rest three motioned images are reconstructed from different gantry starting positions and reconstruction angles. The CT value sums of the four images are approximatly equal. The CT value difference among them comes from system resolution error.

As illustrated in Fig.1, The Grid & Determinant Generator feeds with deformation parameters θs = θ₁, θ₂, …, θ_k, which are generated by front network. According to θ₁, θ₂, …, θ_k, input image is averagely discretized into k pixels. For the presupposed conditions, θ₁, θ₂, …, θ_k are the accurate deformation parameters corresponding to k pixels, so the coordinates of these points can be directly obtained. Furtherly, combined with sampling and bilinear interpolation, the coordinates and pixel values of all points in the initial deformed image are also obtained. Backward mapping is used for sampling, that is, the pixel value of each point of the deformed image is traversed to find the corresponding coordinates on the original image, and then the surrounding pixels are used for simple interpolation. This method avoids holes generated during forward mapping.

To realize CT value conservation of an image before and after deformation, ideally, when one pixel p of deformed image is contributed by q pixel grids on the original image where q > 1, as shown in Fig.3 e), q > 2, for these q pixels, calculating the ratio of covered area of each pixel grid of initial image. Then, the sum of the pixel values of these q grids multiplied by the corresponding ratio should be the exact pixel value of that pixel. An alternative solution is used here to alleviate the situation that computational complexity increase as initial image size becomes larger. To our knowledge, in the two-dimensional space, the geometric mathematical significance of the determinant of the matrix represents the directed area surrounded by two vectors.¹⁴ Based on this mathematical theory, the area of each pixel in the affine transformation grid can be obtained.

Figure 3:

A sample interpolation example. a) initial image of 4x4. b) result of bilinear interpolation, 2x2. c) result of proposed interpolation method with CT value-sum conservation. d)initial image with 2x2 focused ROI. e) the result of bilinear with deformation parameters θ. f)the result of proposed method. b),c) is still in the focused ROI, while e),f) not, so CT value conservation can be obtained in b),c), but invalid in e),f).

A sample interpolation example is shown in Fig.3 a)-c). Geometrically, consider the pixels of the image as squares rather than points and the pixel value as the center points of the input’s corner pixels. Fig.3 a) is the initial image with a size of 4x4, Fig.3 b) is the result of bilinear interpolation and sampling with scaling factor of 1/2. As it illustrated in Fig.3 b), the summation of all pixel value is non-conservation. According to the affine matrix of each pixel of this transformation and its determinant are as follows:

Therefore, the value summation of the image obtained by multiplying Fig.3 b) by ||θ|| is equal to the initial image, as shown in Fig.3 c).

The proposed method maintains CT value conservation meets the following constraints: First, if the deformed image exceeds the size of the deformed grid, the boundary pixels will be lost in the sampling process,which are illustrated in Fig.3 e),f). Second, at the ideal limit resolution, even very exaggerated deformations will become very smooth. Therefore, the method can realize the conservation of CT value under ideal conditions. However, due to the difficulty of implementation and computational complexity, very precise grids are not used during implementation, which will cause errors. However, the following experiments show that the method can control the error within the clinically acceptable range, as shown in Fig.5.

3.1

Image acquisition

The combination network above is performed using supervised end-to-end training strategy. To train this model, the ground truth used in the training process are anonymous motion-artifacts-free cases with United Imaging Healthcare(UIH) uCT ATLAS devices. Referring to the forward model for simulating cardiac motion method proposed in related literature¹⁵¹⁶ and our knowledge of cardiac beating patterns, artificial motion vector fields is generated to simulate all kinds artifacts. The artificial blurred data are input data for training. 9600 samples of 2D coronary patches with the size of 64x64 based on the above artifact simulation methods were generated. The samples were divided into 80% training data and 20% validation data. Test data set involves real motion blurred cases to examine the effectiveness of network and simulation method.

3.2

Neural Network Training

The training was performed on an NVIDIA TITAN RTX for 1000 epochs using an Adam optimizer with 0.1 decay, The batch size is 16 and the loss function is Mean Square Error(MSE).

3.3

Evaluation

Real data from several clinical patients with severe artifacts were used to test the trained network model.

The number of deformation parameters directly affects the accuracy of artifact correction. Fig.4 shows the trend of different deformation parameters on the loss convergence as the training epoch increases. It can be seen that since coronary motion is a relatively complex non-rigid deformation, it is impossible to correct the deformation of the entire image with a single parameter, so the loss is maintained at a relatively high level. With the increase of deformation parameters, the network can gradually learn complex motion deformation, and its number is positively correlated with the correction result. When the number of θ generated by the network is the same as the number of pixels in the input image, it is equivalent that each pixel has its own specific displacement vector, and a more accurate shape correction can be achieved under this configuration.

Figure 4:

The trend of different numbers of deformation parameters on the loss convergence as the training epoch increase. The loss is Mean Square Error.

When Mean Square Error (MSE) and structural similarity index measure (SSIM) were used as the loss function, the above corrected CT value will be slightly deviated. The network needs to add the directed area of the deformation vector for further numerical correction. As shown in Fig.5, TPS and STN failed to maintain the image CT value conservation before and after correction, while CTVC-STN can achieve approximate conservation of CT value within the range of loss not exceeding 1e-6. Real cases were also tested to demonstrate the effectiveness of this method as shown in Fig.6, which shows that the designed network has a good performance in correcting drastic artifacts of coronary images in three planes.

Figure 5:

The x-y plane of one image patch is visualized before and after different methods. Among these, CTVC-STN shows best correction both in shape and CT value. The CT value sum of Input is 3914156, CTVC-STN: 3914182, STN-biliear: 3909930, STN-TPS: 3908637, and Target: 3914188. CTVC-STN: can obtain an aproximate value coservation with a value error ratio of 1e-6. Figure b),c),d) show the differences between ground truth and the Results of STN-bilinear c), STN-TPS d), our method b).

Figure 6:

One clinical case to show that the proposed pipeline is robust in related artifacts in three planes images of patient’s coronary.