Moritz Blumenthal^{1} and Martin Uecker^{1,2,3,4}

^{1}Institute for Diagnostic and Interventional Radiology, University Medical Center Göttingen, Göttingen, Germany, ^{2}DZHK (German Centre for Cardiovascular Research), Göttingen, Germany, ^{3}Campus Institute Data Science (CIDAS), University of Göttingen, Göttingen, Germany, ^{4}Cluster of Excellence “Multiscale Bioimaging: from Molecular Machines to Networks of Excitable Cells” (MBExC), University of Göttingen, Göttingen, Germany

Deep learning offers powerful tools for enhancing image quality and acquisition speed of MR images. Standard frameworks such as TensorFlow and PyTorch provide simple access to deep learning methods. However, they lack MRI specific operations and make reproducible research and code reuse more difficult due to fast changing APIs and complicated dependencies. By integrating deep learning into the MRI reconstruction toolbox BART, we have created a reliable framework combining state-of-the-art MRI reconstruction methods with neural networks. For demonstration, we reimplemented the Variational Network and MoDL. Both implementations achieve similar performance as implementations using TensorFlow.

- efficient implementations of numeric operations allowing training in reasonable time;
- a framework for composing $$$f(\mathbf{x};\mathbf{\theta})$$$ of small building blocks and computing its gradients;
- implementations of training algorithms.

Non-linear operators (nlop) and automatic differentiation have been included in BART for non-linear and model-based reconstruction. An nlop consists of a forward operator $$$F$$$ and linear operators $$$\mathrm{D}_iF_o$$$ modeling the derivative and its adjoint for each pair of inputs $$$i$$$ and outputs $$$o$$$ of $$$F$$$: $$\begin{aligned}F:\mathbb{C}^{N_1+\cdots+N_I} & \to\mathbb{C}^{M_1+\dots+M_O}\\\left[{\mathbf{x}_1,\dots,\mathbf{x}_I}\right] & \mapsto\left[F_1(\mathbf{x}_1,\dots,\mathbf{x}_I), \dots, F_O(\mathbf{x}_1,\dots,\mathbf{x}_I)\right]\\\\{\left.\mathrm{D}_i{F_o}\right|}_{[{\mathbf{x}_1,\dots,\mathbf{x}_I}]}:\mathbb{C}^{N_i} & \to\mathbb{C}^{M_o}\\\mathrm{d}\mathbf{x}_i & \mapsto \left(\left.{\frac{\partial F_o}{\partial\mathbf{x}_i}}\right|_{[{\mathbf{x}_1,\dots,\mathbf{x}_I}]}\right)\mathrm{d}\mathbf{x}_i\\{\left.\mathrm{D}_i{F_o}^H\right|}_{[{\mathbf{x}_1,\dots,\mathbf{x}_I}]}:\mathbb{C}^{M_o} & \to\mathbb{C}^{N_i}\\\mathrm{d}\mathbf{y}_o & \mapsto \left(\left.{\frac{\partial F_o}{\partial\mathbf{x}_i}}\right|_{[{\mathbf{x}_1,\dots,\mathbf{x}_I}]}\right)^H\mathrm{d}\mathbf{y}_o\;.\end{aligned}\;.$$ The derivatives are always evaluated at the last inputs of $$$F$$$. Note that nlops generally do not distinguish between inputs corresponding to weights or input data. Sophisticated nlops, such as neural networks, can be composed of basic nlops by chaining or combining them and linking or duplicating their arguments as depicted in Fig. 2. The derivatives of the composed nlops are constructed automatically.

We implemented several basic nlops often used to construct neural networks such as convolutions, ReLUs and batch-normalization. Reflecting the complex nature of MRI-data, nlops act on complex numbers. To integrate data-consistency in MRI-reconstruction networks, we implemented nlops modeling a gradient update $$$A^H(A\mathbf{x} - \mathbf{k})$$$ and the regularized inversion of the normal operator $$$\left(A^HA + \lambda\right)^{-1}\mathbf{x}$$$. Here, $$$A=\mathcal{PFC}$$$ is the composition of the multiplication with $$$\mathcal{C}$$$oil sensitivity maps, the $$$\mathcal{F}$$$ourier transform and the projection to the sampling $$$\mathcal{P}$$$attern. As proposed in [3], we use the conjugate gradient algorithm to compute $$$\left(A^HA + \lambda\right)^{-1}\mathbf{x}$$$ and its derivatives with respect to $$$\mathbf{x}$$$ and $$$\lambda$$$ - already demonstrating the benefit of being able to use implementations of traditional reconstruction methods.

The training algorithms Adam

- M. Uecker et al., "BART Toolbox for Computational Magnetic Resonance Imaging", Zenodo, DOI: 10.5281/zenodo.592960
- K. Hammernik et al., “Learning a variational network for reconstruction of accelerated MRI data”, Magnetic Resonance in Medicine, vol. 79, no. 6, pp. 3055–3071, 2018
- H. K. Aggarwal, M. P. Mani, and M. Jacob, “MoDL: Model-Based Deep Learning Architecture for Inverse Problems”, IEEE Transactions on Medical Imaging, vol. 38, no. 2, pp. 394–405, 2019
- T. Pock and S. Sabach, “Inertial Proximal Alternating Linearized Minimization (iPALM) for Nonconvex and Nonsmooth Problems”, SIAM Journal on Imaging Sciences, vol. 9, no. 4, pp. 1756–1787, 2016
- D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization”, 2014, arXiv:1412.6980
- Tensorflow-ICG: https://github.com/VLOGroup/tensorflow-icg (commit: a11ad61)
- MoDL implementation: https://github.com/hkaggarwal/modl (commit: 428ef84)
- Variational Network implementation: https://github.com/VLOGroup/mri-variationalnetwork (commit: 4b6855f)