Deep Learning with a Novel Surface Feature for Fully Automatic Quantification of Lesion Hyperintensities in Multiple Sclerosis
Peter Adany1, In-Young Choi2,3,4, Scott Belliston3, Jong Chul Ye5, Sharon G. Lynch3, and Phil Lee2,4

1University of Kansas Medical Center, Kansas City, KS, United States, 2Hoglund Brain Imaging Center, University of Kansas Medical Center, Kansas City, KS, United States, 3Neurology, University of Kansas Medical Center, Kansas City, KS, United States, 4Molecular & Integrative Physiology, University of Kansas Medical Center, Kansas City, KS, United States, 5Korea Advanced Institute of Science & Technology, Seoul, Korea, Republic of


Manual lesion segmentation presents major labor and limitations for quantitative MS lesion analysis, and recent improvements in deep learning promise more consistent, fully automatic lesion segmentation. However, convolutional neural networks still rely on learned thresholding of the arbitrary boundaries of diffuse hyperintensities. Therefore, we aimed to develop a new DL framework pairing a CNN and a custom surface feature that could detect hyperintense isocontour in 3 dimensions very sensitively. Our goal is to achieve detection of MS lesions and quantification of lesion hyperintensity volume with a new DL algorithm that combines traditional imaging and a specially designed surface feature.


MS lesions are generally analyzed by semi-automatic or manual methods, often with several raters. Inter-rater variability presents a major limitation in quantitative lesion volume analysis, stemming from inconsistencies in contouring the diffuse edges of hyperintensities. More consistent contouring is a key promise of automatic segmentation, and rapid improvements in deep learning (DL) have shown new promise for fully automatic lesion detection and segmentation. However, convolutional neural networks (CNNs) approaches effectively still determine lesion boundaries by thresholding, which can transfer poorly across different data sets. More fundamentally, diffuse hyperintense image regions by definition lack a discrete boundary. As DL based segmentation is still an emerging method, there are additional notable limitations such as their architectural tradeoffs and dependence on the quantity and accuracy of training data. Therefore, we aimed to develop a new DL framework pairing a CNN and a custom surface feature that could detect hyperintense isocontour in 3 dimensions very sensitively, bounded by the image noise floor. Our goal is to approach the quantification of lesion volume as integrated image hyperintensity, rather than spatial volume inside a boundary, by training a DL algorithm on the combined traditional imaging and specially designed edge feature data.


Imaging MRI (MPRAGE and FLAIR) were performed on 73 MS patients and 35 healthy controls at 3T. Automatic segmentation of gray matter, white matter and cerebrospinal fluid was performed using SPM8, and manual lesion contouring using Jim6 software (Xinapse Systems). Lesion rating and manual contouring were performed by a neurologist specializing in MS and trained lesion drawers. Training and validation data were split 80%/20% from MRI data sets and 20-30 full-sized image slices were extracted per data set. A custom CNN was implemented, adapted from U-Net1. Additional input channels included image intensity gradient magnitude (edge), position and surface geometry obtained from a custom 3D contouring algorithm. The custom contouring algorithm detects in-plane and out-of-plane isosurface directions, forming a cross-hair shape that is followed a predetermined distance (1~3cm) (Fig. 1.). This provides a set of in-plane (2D) and transverse (3D) isocontour segments that cross at the origin point, without adding the third dimension to the CNN architecture. The isocontours are sub-pixel resolved, reducing their dependence on the image resolution.


A new surface feature was implemented (Fig. 1) to detect diffuse hyperintense isocontours in 3D and convert them into a feature vector for machine learning. A new CNN was implemented based on the U-Net framework. The new CNN architecture uses convolution transpose layers to output smooth, full image-sized lesion prediction maps. Preliminary results show promise for the combined imaging and feature vector based CNN. The surface contour sensitively detects edge features, allowing 3D surface characterization of diffuse lesion hyperintensities, and thus facilitating the extraction of the hyperintense component.


We implemented a novel CNN using a newly developed surface feature for sensitive lesion characterization using DL. We improved the performance of DL-based lesion segmentation by extracting a lesion hyperintensity image component from data in a DL framework. This surface-extraction approach may allow a more consistent quantification of the hyperintense volume, rather than volume solids inside discrete boundaries, which could improve the consistency and robustness of MS lesion volume analysis.


Sharon G. Lynch is funded by the National Multiple Sclerosis Society, and she has participated in Multi-center MS trials and grants funded by Biogen, Teva, Novatis, Acorda, Opexa, Roche, Genentech, Genzyme, Sun Pharma, Vaccinex, Actelion, NMSS, and NIH.


1. Ronneberger O, Fischer S, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI 2015;234-241.


(A, B): Isocontours (yellow) of a FLAIR image obtained using the marching squares algorithm. (C): isocontours (cyan) of a dim hyperintensity. (D): sub-pixel resolved isocontours (red) shown for the same image as (C).

(A): Surface feature contours (1,2) (red) in a FLAIR image. Contour 1 hugs a large lesion. Each contour has an in-plane and orthogonally jutting out-of-plane direction, forming a flexible crosshairs fitting the local gradients. (B): Closer view, with the feature in plain white matter (3) and finding a very dim hyperintensity (4).

Example training validation results from the proposed CNN. Using the U-Net architecture, the output field is sized identically as the input. The regression approach may allow the extraction of hyperintensity from images in the proposed CNN framework.

Proc. Intl. Soc. Mag. Reson. Med. 27 (2019)