Jinwei Zhang^{1,2}, Zhe Liu^{1,2}, Shun Zhang^{2}, Pascal Spincemaille^{2}, Thanh D. Nguyen^{2}, Mert R. Sabuncu^{1,3}, and Yi Wang^{1,2}

A Fidelity Imposing Network Edit (FINE) method is proposed for solving inverse problem that edits a pre-trained network's weights with the physical forward model for the test data to overcome the breakdown of deep learning (DL) based image reconstructions when the test data significantly deviates from the training data. FINE is applied to two important inverse problems in neuroimaging: quantitative susceptibility mapping (QSM) and undersampled multi-contrast reconstruction in MRI.

Introduction

DL-based image reconstruction approaches typically establish network weights using supervised learning and may not perform well on a test data that deviates from the training dataFINE is applied to two important inverse problems in neuroimaging: quantitative susceptibility mapping (QSM) and undersampled multi-contrast reconstruction in MRI.

**Data acquisition and processing: **

For QSM, we obtained 6 healthy subjects, 8 patients with multiple sclerosis (MS) and 8 patients with intracerebral hemorrhage (ICH), with $$$256\times256\times48$$$ matrix size and $$$1\times1\times3mm^3$$$ resolution. MRI was repeated at 5 different orientations per healthy subject for COSMOS reconstruction^{(2)}, which was used as the gold standard for brain QSM. For multi-contrast reconstruction, we obtained and co-registered T1w, T2w, and T2FLAIR axial images of 237 patients with MS diseases, with $$$256\times176$$$ matrix size and $$$1mm^3$$$ isotropic resolution.

**Supervised Training:**

For QSM, we
implemented a dipole inversion network using a 3D U-Net^{(3,4) }$$$\phi(;\Theta)$$$ with weights $$$\Theta$$$ for mapping from local
tissue field $$$f$$$ to COSMOS QSM $$$\chi$$$. Five of the 6 healthy subjects were used for training, giving a
total number of 12025 $$$64\times64\times16$$$ patches, in which $$$20%$$$ were selected randomly as
validation set. We employed the same combination of loss function as in (4) in training the network with Adam optimizer^{(5)} (learning rate 0.001, epoch 40), resulting a 3D U-Net $$$\phi(;\Theta_0)$$$.

For multi-contrast
reconstruction, we employed a 2D U-Net^{(6) }$$$\phi(;\Theta)$$$ with weights $$$\Theta$$$ for mapping from a fully
sampled T1w image $$$v$$$ to a fully sampled T2w
image $$$u$$$. 8800/2200/850 slices were extracted from multi-contrast MS dataset
as the training/validation/test dataset. We used the L1 difference between the
network output and target image as the loss function in training the network
with Adam^{(5)} (learning 0.001, epoch 40), resulting a 2D U-Net $$$\phi(;\Theta_0)$$$. A similar 2D U-Net was established for T2FLAIR images.

**Fidelity Imposing Network Edit (FINE):**

For QSM, given a new local field $$$f$$$, the network weights $$$\Theta_0$$$ from supervised training was used to initialize the weights $$$\Theta$$$ in the following minimization: $$\hat{\Theta} = \arg\min_{\Theta}||W(d*\phi(f;\Theta)-f) ||^2_2$$where $$$W$$$ the noise weighting, $$$d$$$ the dipole kernel.

For multi-contrast reconstruction, given undersampled axial T2w k-space data $$$b$$$ and corresponding fully-sampled T1w image $$$v$$$, the network weights $$$\Theta_0$$$ from supervised training was used to initialize the weights $$$\Theta$$$ in the following minimization: $$\hat{\Theta} = \arg\min_{\Theta}||UF\phi(v;\Theta)-b||^2_2$$where $$$U$$$ the binary random variable density k-space under-sampling mask, $$$F$$$ the Fourier Transform operator.

Above equations were solved using Adam^{(5)} (learning rate 0.001, iterations stop when relative decrease of the loss function between two consecutive epochs reached 0.01). The final reconstruction of the edited network was $$$\hat{\chi}=\phi(f;\hat{\Theta})$$$ for QSM and $$$\hat{u}=\phi(v;\hat{\Theta})$$$ for T2w image. Similarly, T2FLAIR images were reconstructed.

For comparison, total variation regularization (MEDI^{(7)} in QSM, DTV^{(8) }in multi-contrast reconstruction), Supervised Training (DL), DL based L2 regularization (DLL2) were used. The fidelity cost ( $$$||W(d*\chi-f) ||_2$$$ in QSM, $$$||UFu-b||_2$$$ in multi-contrast reconstruction) and structural similarity index (SSIM)^{(9)} were calculated for each method, with COSMOS^{(2)} as ground truth for QSM and fully sampled T2w/T2FLAIR images as ground truth for multi-contrast reconstruction.

The
differences between $$$\Theta_0$$$ and $$$\Theta$$$ were shown in Figure 1 applying FINE in reconstructing QSM of an MS patient. FINE
changed substantially the weights only in the encoder and decoder parts of the
network (Figure
1b&c). Compared to FINE, a randomized $$$\Theta$$$ initialization in above equations using a truncated normal distribution (Figure 1e) (deep image prior)^{(10)} caused substantial changes of weights in all layers (Figure 1f&g) and resulted in markedly inferior QSM (Figure 2d&h).

Fidelity cost and SSIM for QSM are presented in Table 1(a), those for T2w/T2FLAIR in Table 1(b). Both quantitative analysis showed superior performance of FINE compared to the other methods. For QSM, more fine structures are shown clearly in FINE for healthy subject (Figure 2) and MS patient (Figure 3), shadow artifacts were markedly suppressed in FINE for ICH patient (Figure 3). For multi-contrast reconstruction, structural details such as white/grey boundary were clearly depicted in FINE (Figure 4&5).

1. Suciu O, Mărginean R, Kaya Y, Daumé III H, Dumitraş T. When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks. arXiv preprint arXiv:180306975 2018.

2. Liu T, Spincemaille P, De Rochefort L, Kressler B, Wang Y. Calculation of susceptibility through multiple orientation sampling (COSMOS): a method for conditioning the inverse problem from measured magnetic field map to susceptibility source image in MRI. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 2009;61(1):196-204.

3. Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. 2016. Springer. p 424-432.

4. Yoon J, Gong E, Chatnuntawech I, Bilgic B, Lee J, Jung W, Ko J, Jung H, Setsompop K, Zaharchuk G. Quantitative susceptibility mapping using deep neural network: QSMnet. NeuroImage 2018.

5. Kingma DP, Ba J. Adam: A method for stochastic optimization. arXiv preprint arXiv:14126980 2014.

6. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. Lect Notes Comput Sc 2015;9351:234-241.

7. Liu J, Liu T, de Rochefort L, Ledoux J, Khalidov I, Chen W, Tsiouris AJ, Wisnieff C, Spincemaille P, Prince MR. Morphology enabled dipole inversion for quantitative susceptibility mapping using structural consistency between the magnitude image and the susceptibility map. Neuroimage 2012;59(3):2560-2568.

8. Ehrhardt MJ, Betcke MM. Multicontrast MRI reconstruction with structure-guided total variation. SIAM Journal on Imaging Sciences 2016;9(3):1084-1106.

9. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 2004;13(4):600-612.

10. Ulyanov D, Vedaldi A, Lempitsky V. Deep image prior. arXiv preprint arXiv:171110925 2017.

Figure 1.
a): 3D U-Net’s weights for each layer before FINE. b): Weight change in
absolution value after FINE. The encoder and decoder parts of U-Net experienced
substantial weight change by FINE. c): Portion of weights that experienced
relative change above 10% in each layer after FINE. d): Reconstructed QSM after
FINE.
e): 3D U-Net’s
weights for each layer before deep image prior update. Weights are initialized
using a truncated normal distribution. f): Weight change in absolution value
after deep prior update. All layers of U-Net experienced substantial weight
change by deep prior. g): Portion of weights that experienced relative change above
10% in each layer after deep prior update. h): Reconstructed QSM after deep
prior update. Deep prior failed in this case.

Figure 2.
Comparison of QSM of one healthy subject reconstructed by (from left to right)
COSMOS, MEDI, DL, DLL2 and FINE. More detailed structures are recovered after
fidelity enforcement. Structures in occipital lobe were more clearly depicted
in FINE and DLL2 than in MEDI and DL.

Figure 3. Axial images from representative MS and ICH
patients. From left to right: local field, QSM reconstruction by MEDI, DL, DLL2
and FINE, respectively. For MS patient, central veins near the ventricle were
better depicted in FINE and DLL2 than in MEDI or DL. Same structures were
discernible on local field as well. For ICH patient, hypo-intense artifact was
observed close to ICH in DL and DLL2 but suppressed in MEDI and FINE.

Figure 4.
Comparison of T2w and T2FLAIR reconstructions. From left to right: fully
sampled ground truth, under-sampled k-space reconstruction by DTV, DL, DLL2 and
FINE, respectively. First row: reconstructed image. Second row: magnitude of
reconstruction error with respect to truth. Third row: zoom-in images. FINE
provided a clear recovery at white/grey border, while DTV and DL suffered from
over-smoothing and DLL2 suffered from noise.

Figure 5. Comparison of
T2FLAIR reconstructions.

Table 1. Fidelity cost and SSIM comparison of reconstruction methods for QSM healthy subject, T2w and T2FLAIR test dataset.