A.

CT Image Denoising with DNNs

In this work we assume to have high-dose images y ∈ ℝ^m×n as well as low dose images x ∈ ℝ^m×n during training time. The aim of any deep-learning based denoising method is then to find a function f(· ; θ) with parameters θ, such that

where f is realised by a DNN. In recent years most improvements on finding an optimal f focused on alterations of the architecture and training scheme. While earlier work utilized pixelwise losses (in image or feature space) which lead to smooth predictions and lack high-frequency information [4, 6], many recent methods are being trained as GANs, leading to extremely realistic denoising results [3, 5].

B.

Invariances of DNNs

Our work is based on reference [7], where the authors seek to reconstruct and interpret the invariances of image classification DNNs using invertible neural networks (INNs).

Given a network f(x) we can analyze any internal latent representation z thereof by decomposing f into f(x) = Ψ(z) = Ψ(Ф(x)). To then explain z we need to know what information of the input x is captured in z and to what information Ф is invariant to (and is thus missing in z). To this end, the authors of [7] employ a VAE comprised of an encoder E and a decoder D that is trained to learn a complete data representation = E(x) by reconstructing the input from s.t. ||D(E(x)) − x|| is minimized.

Since the complete data representation now not only contains the information captured in z but also its invariances v, we need to disentangle v and z by learning a mapping

Here, it is assumed that invariances v can be sampled from a Gaussian distribution, i.e. p(v) = 𝒩(v|0,1), and the mapping t is realized through a normalizing flow [8–10], a sequence of INNs between the simple (normal) distribution p(v) and the complex distribution .

Since t is invertible, we can generate new that only differ in their realization of the invariances by first sampling v ~ p(v) and then applying the inverse mapping of t

To visualize in the low dose image space we can reconstruct them using the previously trained decoder D: .

A.

Dataset

For all our studies the Low Dose CT Image and Projection dataset [11] is employed. The dataset comprises 50 head scans, 50 chest scans, and 50 abdomen scans acquired at routine dose levels with a SOMATOM Definition Flash (Siemens Healthineers, Forchheim, Germany) CT scanner. Additionally the dataset provides simulated low dose reconstructions (at 25% dose for abdomen/head and at 10% dose for chest scans) which were used as input to the denoising networks. We split the dataset into 70%/20%/10% for training/validation/test across all patients and trained with a weighted sampling scheme such that slices from each patient were sampled with equal probability.

To make results between different methods comparable we trained and validated all denoising networks as well as the invariance reconstruction method on the same training/validation split of our data.

B.

Denoising Methods

While our method can be used to provide post-hoc invariance analysis for any (trained) DNN-based denoising method, for simplicity, we here focus on interpreting the invariances of two well-known denoising methods:

Chen et al. [4] proposed a simple three-layer convolutional neural network which was trained to minimize (1) using an L₂ loss. The authors trained their network on patches of size 33 × 33 using an SGD optimizer and showed that their method can outperform conventional state-of the art methods.

Yang et al. [5] improved on previous works by training a Wasserstein GAN (WGAN) [12] in combination with a perceptual loss [13] in feature space. Furthermore, they utilize a deeper generator compared to [4] and train the network on larger patches of size 64 × 64.

We trained both [4] and [5] on the dataset described in Sec. III-A using the hyperparameters as described in the original papers. Whenever hyperparameters were not stated by the authors, we ran a grid-search and used the parameters that result in the lowest validation loss.

C.

Recovering Invariances

Similar to reference [7] we first learn a complete data representation = E(x) for a given low dose image x by training a VAE g(x) = D(E(x)). Our encoder is based on a ResNet-101 [14] and our decoder on a BigGAN [15] where the conditioning on the class is replaced by a conditioning on the latent representation . To improve reconstruction quality the VAE is trained together with a critic C as a WGAN and instead of training it on entire 512 × 512 pixels images we train it on 128 × 128 pixels patches.

For both of the two denoising networks evaluated, we train three conditional INNs (cINNs) to learn to reconstruct invariances at three different layers in the network.

For Chen et al. [4] we do so at layer 1, 3, and 5 and for Yang et al. [5] at layer 1, 7, and 13 (refer Tab. I). Each of the cINNs, t is composed of four invertible blocks, where each block is composed of coupling blocks [16], actnorm layers [17], and shuffling layers. For each invertible block, the conditioning on the denoising network representation z is realized by concatenating an embedding h = H(z), where H is a shallow network, with the input to the respective block.

TABLE I:

Overview of generator architectures used in Chen et al. [4] and Yang et al. [5]. Kernel sizes of the 2D convolutions are indicated by k and their number of filters by f. Final nonlinearities of the original architectures were omitted to accommodate for the normalization of our data.

For each network and layer we then reconstruct different samples of the invariances = D(t⁻¹(v, z)), v ~ 𝒩(0,1). Additionally, we can compute the standard deviation over a large set (here 250) of samples to highlight regions with high variation across the reconstructed invariances.

A.

Denoising Methods

We find that the results from both denoising networks are similar to those reported in the respective original papers (Fig. 1). Due to the L₂ loss in image space the results from [4] appear smooth and lack structural fidelity. This is alleviated by training with an adversarial loss and consequently our results for [5] look much more realistic with higher details and noise structures very similar to those present in the high dose images.

Fig. 1:

Denoising performance of Chen et al. [4] and Yang et al. [5] for six different dataset samples (columns). Blue arrows indicate regions where the networks produced errors in the reconstruction of anatomical details.

However, we find that both methods are unable to correctly reconstruct anatomical details in several cases (refer Fig. 1, blue arrows). This is particularly problematic when the network is trained in an adversarial setting, where those false anatomies can look very convincing to the radiologist.

B.

Reconstructed Invariances

The reconstructed invariances for both networks and two different samples (ref. Sec. III-C) are provided in Fig. 2. For each sample we also show the low dose input image x, the high dose ground truth image y, the reconstruction of the complete data representation , and the denoised image f(x).

Fig. 2:

Best viewed in color. Analysis of Chen et al. [4], (a) & (b), and Yang et al. [5], (i) & (ii). Provided are low dose input image x, high dose ground truth image y, VAE network reconstruction (Sec. III-C), denoised image f(x), five reconstructed samples from the space of invariances, and the standard deviation over 250 invariance samples. Red arrows highlight errors in the VAE reconstruction and blue arrows highlight regions in the reconstructed invariances.

From this we find that both denoising methods are invariant to several anatomical features to some extent (Fig. 2; blue arrows). We also find a higher overall variance of the invariances in homogeneous regions of the image for [4], indicating that it is more invariant to the specific realization of noise in the low dose input image. However, when inspecting the VAE reconstructions we also find major deviations from the original low dose image x (Fig. 2, red arrows), which may explain some of the differences between the reconstructed invariances and x.