Multi-site diffusion MRI harmonization in the presence of gross pathology: How far can we push?
Suheyla Cetin Karayumak1, Marek Kubicki1, and Yogesh Rathi1

1Department of Psychiatry, Harvard Medical School, BOSTON, MA, United States


We present a multi-site diffusion MRI (dMRI) data harmonization method using CycleGAN network with segmentation loss (CycleGANS). This method aims to learn an efficient mapping of dMRI signal using rotation invariant spherical harmonics features from the same set of subjects across sites. At the same time, it has potential to learn the tumor pathology (if exists) during harmonization. We compare our CycleGANS network with the CycleGAN network. We show that our CycleGANS network has better multi-site diffusion MRI data harmonization accuracy. Moreover, our method shows up to 60% improvement on the prediction of tumor pathology.


Harmonization of multi-site diffusion MRI (dMRI) datasets can dramatically increase the statistical power of neuroimaging studies and enable comparative studies pertaining to several brain disorders1,2. Given the importance of the problem, initially, several data pooling methods1,2,3,4 have been published which are based on removing statistical differences from diffusion tensor imaging derived metrics. Recently, model-free dMRI harmonization methods have been proposed which can be used to harmonize the raw dMRI signal across sites. Based on the dMRI data on hand, there have been successful attempts to harmonize raw dMRI signal5-10. However, none of the methods have been tested on harmonizing of diffusion MRI data with gross pathologies. In this work, we propose a multi-site dMRI data harmonization using CycleGAN with segmentation loss (CycleGANS) which has two goals: (i) learning an efficient mapping of dMRI signal from the same set of subjects across sites; (ii) preserving existing tissue abnormalities or pathologies during harmonization.


In our previous work5-8,10, rotation invariant spherical harmonics (RISH) have been shown to successfully represent different micro-structural tissue properties of the dMRI signal for each spherical harmonics (SH) order. They can be appropriately scaled to modify the dMRI signal without changing the principal directions of the fibers5. In this work, using HCP Prisma RISH-features (up to 6th order: {L0, L2, L4, L6}) as input, our first goal is to learn a nonlinear mapping of Prisma to Connectom scanner using the paired RISH-features. To this end, we first match the resolutions, then spatially align Connectom and Prisma RISH-features for each subject. Later, we segment the white matter, gray matter and CSF from the L0 RISH-feature using Geodesic Information Flow (GIF) software17. Next, we construct CycleGAN14-16 network with aligned Prisma RISH-features as input and Connectom RISH-features as output with their corresponding segmentation maps (Figure 1). The cycle consistency and adversarial loss were shown16 to only constrain the network to learn a global mapping that matches the marginal distribution but not the conditional distribution pertaining to the tissue variabilities. Therefore, a model trained using these losses does not guarantee to preserve the pathologies (e.g. tumor). In addition to adversarial and cycle-consistency loss, “segmentation loss”15 can be included as a multi-class cross entropy loss: $$$L_{segm}(S_P,S_C,G_P,G_C)=-Y_P log(S_P(G_P(C)))+-Y_C log(S_C(G_C(P)))$$$, where $$$Y$$$ denotes the ground truth segmentation map. The segmentation loss can regularize the generators and preserve the variability of the tissues (white matter, gray matter or CSF). Segmentation map can be extended to pathologies by supervising the pathology as a semantic label.

Experiments and Results

Harmonization. We used 16 HCP healthy subjects with dMRI scans obtained from both Siemens Prisma and Skyra Connectom scanners11, where we learned the mapping for $$$b=3000$$$ shell from Prisma to Connectom scanner using RISH-features. The final signal was estimated by multiplying Prisma SH coefficients with the voxel-wise scale between Connectom and harmonized Prisma RISH-features. We generated Fractional-Anisotropy (FA), Mean-Diffusivity (MD), mean Diffusion-Weighted-Imaging (DWI) and RISH-features L0, L2 using the Connectom data and our harmonized results. Training was done using leave-one-out cross validation and Normalized Mean Squared Error (NMSE) was computed on all measures (See Table 1 for the performance of CycleGAN and CycleGANS networks).

Validation on pathology. To test the performance of the harmonization on data with pathology, we simulated tumor on Prisma and Connectom dMRI data. Independent dataset that includes a low-grade tumor subject (scanned on a Siemens scanner at Brigham and Women’s Hospital) was used for simulation, where its T1, T2 and dMRI data were co-registered. Then, the steps in Figure 2 were applied to simulate tumor for Prisma and Connectom dMRI data (See Figure 3 for RISH-features of tumor data).

Next, we ran two experiments: (1) we trained CycleGAN and CycleGANS networks with 16 healthy subjects and tested on data with simulated tumor; (2) we trained CycleGAN and CycleGANS with simulated tumor as input. For CycleGANS, the tumor was included as a semantic label in the segmentation map. Figure 4 depicts the qualitative comparison of the methods for RISH-Feature L0. Our CycleGANS network is observed to perform better in prediction of tumor when compared to CycleGAN. Quantitatively, CycleGANS showed up to 60% improvement when compared to CycleGAN on the prediction of tumor for Connectom data.

Discussion and Conclusion

In this paper, we proposed CycleGANS multi-site dMRI data harmonization method, which has potential to learn the tumor pathology during harmonization. We note that this work is preliminary and extensive validation on real tumor data or different types of pathologies is required to further understand the power and limitations of this technique.


The authors would like to acknowledge the following NIH grants which supported this work: R01MH102377 (PI: Dr. Marek Kubicki), R01MH097979 (PI: Dr. YogeshRathi).


1. Jahanshad, N. et al. Multi-site genetic analysis of diffusion images and voxelwise heritability analysis: A pilot project of the enigma-dti working group. NeuroImage 81, 455-469 (2013).

2. Kochunov, P. Multi-site study of additive genetic effects on fractional anisotropy of cerebral white matter: Comparing meta and mega analytical approaches for data pooling. NeuroImage 95, 136-150.

3. Pohl, K.M. et al. Harmonizing dti measurements across scanners to examine the development of white matter microstructure in 803adolescents of the ncanda study. Neuroimage 130, 194-213 (2016).

4. Fortin, J.P. et al. Harmonization of multi-site diffusion tensor imaging data. Neuroimage 161, 149-170 (2017).

5. Mirzaalian H. et al. Inter-site and inter-scanner diffusion mri data harmonization. NeuroImage 135, 311-323 (2016).

6. Mirzaalian H. et al. Multi-site harmonization of diffusion mri data in a registration framework. Brain Imaging and Behavior, (2017).

7. Mirzaalian, H. et al. Harmonizing Diffusion MRI Data Across Multiple Sites and Scanners. Springer. p. 12-19, (2015).

8. Cetin Karayumak, S. et al. Retrospective harmonization of multi-site diffusion MRI data acquired with different acquisition parameters. Neuroimage 184, 180–200 (2018).

9. Tax C. et al. Cross-vendor and Cross-protocol harmonisation of diffusion MRI data: a comparative study. ISMRM 2018.

10. Cetin Karayumak, S. et al. Harmonizing diffusion MRI data across magnetic field strengths. Medical Image Computing and Computer Assisted Intervention- MICCAI 2018 pp 116-124 and Lecture Notes in Computer Science book series (LNCS, volume 11072).

11. Van Essen DC, Smith SM, Barch DM, et al. The WU-Minn Human Connectome Project: An Overview. NeuroImage. 2013;80:62-79. doi:10.1016/j.neuroimage.2013.05.041.

12. Avants, B.B., Tustison, N.J., Song, G., Cook, P.A., Klein, A., Gee, J.C.A reproducible evaluation of ANTs similarity metric performance in brain image 0 registration. NeuroImage 54, 2033-2044, 2011.

13. Fedorov A. et al. 3D Slicer as an Image Computing Platform for the Quantitative Imaging Network. Magn Reson Imaging. 2012 Nov;30(9):1323-41.

14. Jun-Yan Zhu et al., Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, ICCV, 2017.

15. Zhang, Z. et al. Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency, CVPR, 2018.

16. Jue Jiang et al., Tumor-Aware, Adversarial Domain Adaptation from CT to MRI for Lung Cancer Segmentation, MICCAI, 2018.

17. Cardoso, M.J. et al., Geodesic information flows: Spatially-variant graphs and their application to segmentation and fusion. IEEE Transactions on Medical Imaging 34(9) (2015) 1976–1988


Figure 1 - CycleGAN network with segmentation loss (CycleGANS) for multi-site dMRI harmonization using RISH features and tissue segmentation map as input.

Figure 2 - Algorithm steps for tumor simulation on Connectom and Prisma dMRI data.

Figure 3 - RISH-features up to 6th order (L0, L2, L4 and L6) of Connectom and Prisma data for the same subject with simulated tumor.

Figure 4 - Harmonized L0 comparison between CycleGAN (with and without tumor data in training) and CycleGANS (with and without tumor data in training). CycleGANS is observed to perform better in prediction of tumor when compared to CycleGAN.

Table 1- The NMSE results for harmonization from Prisma to Connectom using CycleGAN and CycleGANS networks: NMSE was computed between Connectom data and our harmonized results for FA, MD, DWI and RISH-features (L0, L2).

Proc. Intl. Soc. Mag. Reson. Med. 27 (2019)