Deep convolutional neural networks for brain lesion segmentation in multiple sclerosis using clinical MRI scans
Sunny Nagam1, Glen Pridham1, and Yunyan Zhang1

1University of Calgary, Calgary, AB, Canada


Machine learning opens up a new opportunity for advancing our image pattern recognition abilities in medical imaging. In this study, we tested the potential of 3 new deep convolutional neural network-based learning methods for detecting brain MRI lesions in multiple sclerosis (MS). Using clinical scans available online from 10 patients, we found that the ResNet and SegNet achieved a promising dice score of 0.65 and 0.61 respectively, better than the generative adversarial network. Deep learning methods may be novel tools for optimal detection of brain MRI lesions, improving the management of patients with MS and similar disorders.


Multiple sclerosis (MS) is an inflammatory demyelinating disease of the central nervous system. It affects over 2 million people worldwide and is the leading cause of non-traumatic disability in young adults.1 Pathologically, MS is characterized by multi-focal lesions, and accurate detection of brain lesions has been a critical task in assessing the impact of new candidate therapies for patients.2 Although there are various imaging techniques developed for brain lesion segmentation, many suffer from low reproducibility or require manual intervention. Machine learning has opened up a new avenue for advanced pattern recognition, particularly its new form, deep learning. Over the past few years, various deep learning methods have emerged with the potential to enhance our disease characterizing abilities.3 Our goal was to compare the feasibility of 3 deep learning methods for segmenting brain white matter lesions in MS patients using clinical magnetic resonance imaging (MRI).


We investigated 3 common deep convolutional neural network (CNN)-based methods including residual neural networks (ResNets),4 deep convolutional encoder-decoder networks (SegNets),5 and generative adversarial networks (GANs).6 We used a free source MRI dataset from the 2008 MICCAI MS Lesion Segmentation Challenge to test our models. This dataset included standard clinical scans: T1- weighted, T2-weighted and FLAIR MR images, from 10 patients with MS, and corresponding lesion masks as ground truth for individual MRI slices (Fig. 1). Each MRI sequence contained 100 images, totaling 300 images per patient, 3000 for 10 patients. Prior to input into the networks, we performed image preprocessing to standardize the scans including 1) normalizing the MRI signal intensity to 0 to 1; and 2) co-registration to align images between MRI sequences per subject.

Building upon the reported architectures, we first customized the input and output layers of the networks, such that each method took all 3 anatomical MRI scans as input, in addition to the lesion masks, and allowed 2 output categories per voxel: 0 or 1 representing lesion or not. Moreover, we reduced the ResNet to 30 layers to decrease training time, and the SegNet included 8 layers (Fig. 2) For GANs, we tested different options and combinations for the generative and discriminative networks including the SegNet, and the customized ResNet.

The specific settings of the networks followed the optimized approaches as reported in the literature.7 This included Max Pooling to decrease feature space, ReLu to correct unsaturated gradients, Adam to optimize learning rate, and a sigmoid transfer function to generate binary output. We used a leave-one-out validation scheme to test the performance of the methods and a dice metric as a loss function to evaluate the learning accuracy.8 The dice score ranges from 0 to 1, the higher the better.


Visually, each method showed the potential to detect focal lesions using standard brain MRI scans. Quantitatively, the ResNet and SegNet performed the best with a dice score of 0.65 and 0.61 respectively. While the accuracy of the methods was relatively suboptimal when segmenting the entire shape of larger lesions or more densely packed lesions, both were visually confirmed to detect nearly all occurrences of the lesions regardless of lesion size (Fig. 3). Moreover, we found that it was also possible to use only FLAIR and T2-weighted MR images, with the addition of a second FLAIR image 4 slices apart as input (Fig. 4), to detect the lesions using ResNet and SegNet once the models were trained. The GANs method did not perform as well (dice score = 0.28) compared to either method.


Anatomical MRI is very sensitive to the signal abnormalities created by the underlying tissue structure. Although the intensity alone lacks specificity, each anatomical sequence includes spatial context which may be lesion-specific, such as the distribution of lesion voxel intensities, and their relationship to surrounding voxels. Deep learning is a multivariate process that takes thorough account of inter-voxel relationships,7 including the unique lesion-defining image textures. The ResNet is an established method as shown in the 2015 Large Scale Image Recognition Competition. While with reduced number of layers, our results were achieved based on the implementation of the original network. Similarly, the SegNet is also a tested method in prior applications based on encoder-decoder networks.5 GANs is a relatively new invention. The relatively low performance here may be due to 1) lack of maturity of our implementation; 2) inappropriate target of the application.


Both ResNet and SegNet demonstrate great potential to detect MS lesions using clinically available brain MRI scans, deserving further confirmation.


We thank the funding supports from the Natural Sciences and Engineering Council of Canada (NSERC), MS Society of Canada, and Alberta Innovates, Canada.


1. Ontaneda D, Thompson AJ, Fox RJ, et al. Progressive multiple sclerosis: prospects for disease therapy, repair, and restoration of function. Lancet 2017; 389: 1357-1366.

2. Miller DH, Albert PS, Barkhof F, et al. Guidelines for the use of magnetic resonance techniques in monitoring the treatment of multiple sclerosis. Ann Neurol 1996; 39:6–16.

3. Plis SM, Hjelm DR, Salakhutdinov R, et al. Deep learning for neuroimaging: a validation study. Frontiers in neuroscience 2014; 8: 229.

4. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. arXiv:151203385 2015.

5. Badrinarayanan V, Kendall A, Cipolla R. SegNet: A deep convolutional encoder-decoder architecture for image segmentation. arXiv: 2016; 1511.00561v3.

6. Wang Y, Yu B, Wang L, et al. "3D conditional generative adversarial networks for high-quality PET image estimation at low dose." Neuroimage 2018; 174: 550-562.

7. Vieira S, Pinaya WH, Mechelli A. Using deep learning to investigate the neuroimaging correlates of psychiatric and neurological disorders: Methods and applications. Neurosci Biobehav Rev 2017; 74(Pt A): 58-75.

8. Senders JT, Staples PC, Karhade AV, et al. (2018). "Machine Learning and Neurosurgical Outcome Prediction: A Systematic Review." World Neurosurg 109: 476-486.e471.


Fig. 1: Example input images to the deep learning networks. This includes 3 types of clinically available MRI scans from T1-weighted, T2-weighted, and FLAIR images, and the corresponding lesion mask for the MRI slice.

Fig. 2: Example SegNet and ResNet architectures. Panel I shows the diagram of the SegNet used in this study. The color boxes represent the network layers, and the numbers below represent the channels used at an individual layer. Panel II shows the scheme of a typical learning block of a ResNet (left) and an usual neural network (right). In ResNet, a special link is added to connect the learning results from the previous block (X) directly to the output of the next block to avoid information loss, particularly at deep layers of convolutional neural networks; He et al, arXiv:151203385 2015.

Fig. 3: Lesion detection examples using SegNet and ResNet. The lesion masks are provided along with the clinical MRI dataset with expert verification. Shown examples are obtained from different MRI slices from T1, T2, and FLAIR (see Fig. 1).

Fig. 4: Example learning results of SegNet and ResNet from the same MR images. Input images are focal patches from Flair and T2 scans only, with addition of a FLAIR image that is 4 slices away from the initial image of the same volume. This allows to present slightly different imaging patterns for improved learning but with similar lesion locations.

Proc. Intl. Soc. Mag. Reson. Med. 27 (2019)