Segmenting Brain Metastases Using Deep Learning on Multi-Modal MRI
Darvin Yi1, Endre Grøvik2,3, Michael Iv3, Elizabeth Tong3, Greg Zaharchuk3, and Daniel Rubin1

1Department of Biomedical Data Science, Stanford University, Stanford, CA, United States, 2Department for Diagnostic Physics, Oslo University Hospital, oslo, Norway, 3Department of Radiology, Stanford University, Stanford, CA, United States


Detecting and segmenting brain metastases is a tedious and time-consuming task for many radiologists, particularly with the growing use of multi-modal 3D imaging. Using deep learning to learn from the comprehensive pixel-wise labeled MRI-data, this work aims to train a fully convolution neural network for automatic detection and segmentation of brain metastases using multi-modal MRI. By training and testing on over 100 and 50 patients, respectively, including a variety of size and number of brain metastases from several primary cancers, this work provides a comprehensive investigation on the value and potential use of machine learning in a clinically relevant setting.


Attributed in large by advances in effective treatment regimens of primary tumors, there has been an explosion in the number of patients with metastatic cancer to the brain over the last decade. Detecting and segmenting brain metastases is a tedious and time-consuming task to most radiologists, particularly with the growing use of multi-modal 3D-imaging. To aid radiologists in this task, our work aims to train a fully convolution neural network (CNN) for automatic detection and segmentation of brain metastases using multi-modal MRI.


This retrospective study was approved by the Institutional Review Board. A total of 156 patients with brain metastases from several primary cancers were included. Pre-surgical MRI included pre- and post-gadolinium T1-weighted 3D fast spin echo (CUBE), post-gadolinium T1-weighted 3D axial IR-prepped FSPGR (BRAVO), and 3D CUBE fluid attenuated inversion recovery (FLAIR). The examinations were performed on a combination of 1.5T (n=18) and 3T (n=138) scanners. The ground truth was established by two experienced neuroradiologists, manually delineating and cross-checking the enhancing lesions using OsiriX MD. Training was performed using a 2.5D fully convolution network based on a GoogLeNet architecture1 (Fig. 1) using 7 slices from the aforementioned sequences as input, comprising a single center slice with 3 slices above and below. The network was modified to optimize segmentation and trained using TensorFlow. The output was a probability map on whether the voxel represents a metastasis ranging from 0-1. The dataset was split into 100/5/51 patients for training/validation/testing. The test set was equally and randomly selected within three different patient subgroups presenting with 1-3, 4-10, and >10 metastasis (17 cases in each subgroup). The results were evaluated by estimating the precision and recall, in addition to the Dice- and Intersection over Union (IoU) score, and by using ROC-curve statistics.


Mean patient age was 63±12 yrs (range: 29-92 yrs). Primary malignancies included lung, breast, melanoma, genitourinary, gastrointestinal, and miscellaneous cancers. Of the 156 patients included, 64 (41%) had 1-3 metastases, 47 (30%) had 4-10 metastases, and 45 (29%) had >10 metastases. Figure 2 shows an example case demonstrating the resulting probability map as an overlay on the BRAVO image-series. Figure 3 shows the individual ROC curves for all 51 patients in the test set. The area under the ROC-curve (AUC), averaged across all patients, was 0.96. Based on the ROC statistics, the average optimal threshold for including a voxel as a metastasis was 0.91 likelihood. Using this threshold, the average precision and recall, and the segmentation IoU-and Dice-score were estimated to 0.69, 0.55, 0.44, and 0.58, respectively. The distribution of these metrics within the subgroups are shown Table 1.


This study demonstrates that a modified 2.5D GoogLeNet CNN can detect and segment brain metastases on multi-modal MRI with high accuracy, comparable to similar reported results in the literature. Liu et al. used a CNN-based segmentation for stereotactic radiosurgery (SRS) planning and reported an average Dice-score and AUC of 0.67 and 0.98, respectively2. Charron et al. used a 3D CNN for detection and segmentation of brain metastases that were scheduled for irradiation3. Similar to our study, their network was trained using multi-modal MRI which proved to outperform deep neural networks trained on a single MRI contrast. Their network showed a sensitivity of 98% and 7.2 false positive per patient. One condition that separates our work from these previous studies is the strength of having diverse data, which may make is harder for the network to demonstrate an overall high performance. We include cases from both 1.5T and 3T multi-vendor scanners, and our data was not limited to patients receiving SRS, thus including more patients with many (>10) metastasis. The split between training/validation/testing in this study is somewhat unusual compared to similar studies in the literature. However, we chose to test on a large number of cases to understand how generalizable the network was and to facilitate subgroup analysis, enabling a better understanding of the network’s clinical performance and potential. In earlier stages of the study, we found that the network had a high performance on approximately half of the current training set, and that increasing the number of training cases did not provide significant improvement, thus justifying our use of a larger test set. Our results indicate that the networks segmentation performance is slightly better for patients with 4-10 metastases, which may result from an optimal tradeoff between total number and individual size of the metastases. This will be further investigated.


Using deep learning on multi-modal MRI may facilitate automatic and accurate detection and segmentation of brain metastases in a clinical setting.


No acknowledgement found.


1. C. Szegedy et al., Going deeper with convolutions, Proc. IEEE Conf. Comput. Vis. Pattern Recognit., (2015), pp. 1-9.

2. Y. Liu et al., A deep convolutional neural network-based automatic delineation strategy for multiple brain metastases stereotactic radiosurgery PLoS One, 12 (2017), e0185844,

3. O. Charron et al., Automatic detection and segmentation of brain metastases on multimodal MR images with a deep convolutional neural network. Comput. Biol. Med., 95 (2018), 43-54,


Table 1: Detection and segmentation performance

Figure 1: Flowchart showing the four image inputs used to train the neural network, the modified GoogLeNet architecture, and the resulting output colormap (overlaid a post-contrast BRAVO image) representing a probability map of whether the voxel represents a metastasis, ranging from 0-1 as indicated by the color bar.

Figure 2: Example case showing a 47-year-old female patient presenting with three brain metastases from lung cancer. The image mosaic shows the predictions (probability maps as indicated by the color bar), generated by the neural network, and delineated metastases (yellow circle) overlaid the post-contrast BRAVO image.

Figure 3: ROC curves for all 51 patients included in the test set. The average area under the ROC curve (AUC) was 0.96, ranging from 0.82 to 1.00. The ROC curves are color coded according to the subgroups (i.e. number of metastases). For the outlier, we retrospectively found that one of two manually delineated metastases most likely was an artifact, which was supported by follow-up scans. This may explain the deviating ROC curve and low AUC.

Proc. Intl. Soc. Mag. Reson. Med. 27 (2019)