How to Benchmark DSC-MRI: the technical development of an anthropomorphic phantom for software validation 
Laura C. Bell1, Natenael B Semmineh1, Sudarshan Ragunathan1, and C. Chad Quarles1
1Barrow Neurological Institute, Phoenix, AZ, United States


Clinical adoption of DSC-MRI for brain cancer imaging is plagued by reproducibility concerns. To support and advance reproducible science for DSC-MRI, an anthropomorphic phantom is developed in order to benchmark post-processing pipelines and software platforms. In addition to this anthropomorphic phantom, the technical computation of relative cerebral blood volume (rCBV) is discussed since rCBV is not an absolute measurement in DSC-MRI. In summary, this DRO-based benchmark can then be used to characterize the accuracy of commonly employed DSC-MRI algorithms and clinical software.


Recently, there has been great momentum towards the standardization of DSC-MRI imaging protocols to alleviate concerns of reproducibility for clinical adoption [1-3]. A contributing aspect to this consensus was the use of a digital reference object (DRO) [4]. Building on these efforts, we focus on another obstacle that limits clinical utility, accuracy, and multi-site consistency: the lack of a validation of, and a benchmark for, DSC-MRI post-processing algorithms and software. A large aspect in the variability of post-processing stems from varying models of blood brain barrier (BBB) leakage correction for relative cerebral blood volume (rCBV) even with a standardized acquisition [5]. In an effort to continue to advance reproducibility, we aim to 1) establish an rCBV benchmark for DSC-MRI, and 2) demonstrate the technical development of an anthropomorphic DRO-based benchmark.


The original, population-based DRO encompasses 10,000 unique low-grade and high-grade tumor pixels that were simulated for both an intact- (Ktrans = 0) and a disrupted- (Ktrans > 0) BBB as well as normal-appearing-white-matter (NAWM) pixels (Ktrans = 0) (Figure 1). In addition to Ktrans, other kinetic parameter inputs (“ground truth”) include Fp (plasma flow rate), PS (the permeability surface area product), vp (vascular volume fraction), and ve (extracellular extravascular volume fraction).

DSC rCBV Benchmarking: Note that rCBV is not an input parameter. For DSC-MRI, the input parameters cannot directly be compared to those derived from a traditional signal analysis due to the biophysical relationship between the contrast-agent (CA) induced R2* relaxation rate change (∆R2*(t)) and intravascular CA concentration. This relationship is characterized by the CA relaxivity, r2*; which is influenced by the voxel-wise vascular geometry. Accordingly, the derived rCBV measures are a function of the input vp and the voxel-wise r2* values. With the DRO, the intravascular CA concentration is determined by the input parameters, including vp, and the associated ∆R2* time series is computed [6]. This enables the computation of r2* for each unique, voxel-wise vascular structure and appropriate scaling of vp to generate ground truth measures of rCBV. Ground truth rCBV (CBVtrue) can now be defined using the simulated intact-BBB DRO.
Next we demonstrate the ability to benchmark two post-processing leakage correction methods: the gamma-variate method and the Boxerman-Schmainda-Weiskoff (BSW) [7]. The prior recommendations were used for this DRO simulation (full dose preload and bolus injection, 65o flip angle, 30 ms TE, 1.5 s TR) [1].

Anthropomorphic DRO Development: To convert the DRO into an anthropomorphic structure, a representative patient DSC-MRI dataset was segmented into 6 regions: enhancing tumor, non-enhancing FLAIR, vessels, and white and gray matter. Semi-automatic segmentation of enhancing tumor from anatomical T1-weighted scans and edema from anatomical T2-weighted FLAIR scans were done [8]. The rest of the healthy tissue structures were automatically segmented using both the T1- and T2- weighted anatomical scans [9]. Second, both in vivo and in silico signal time curves were converted to ∆R2*(t) curves without any post-processing leakage correction methods applied for all segmentation regions. Third, each in vivo pixel was matched to an in silico pixel by determining the minimum the L2 norm for all in silico pixels representing the segmented region of interest.
Statistical Analysis: Concordance correlation coefficients (CCC) were evaluated between the computed r2* values (CBVestimated) and the vp (CBVtrue) to assess the agreement across all tumor pixels.


Table 1 demonstrates the ability to use the DRO to benchmark post-processing leakage correction methods. The CCC between the ground truth rCBV and that computed from the intact DRO, across the 10,000 pixels, is 0.922. Perfect agreement is not expected given that the DRO includes realistic noise and discrete sampling. Establishing this optimal agreement (CCC = 0.92), before adding in effects of contrast agent leakage or other pre-processing steps, provides a solid basis by which to benchmark pipelines. Without leakage correction, the agreement between CBVestimated from disrupted-BBB signal time curves and the ground truth (CBVtrue) is quite poor (CCC = 0.292) – as expected. This agreement improves slightly when a gamma-variate leakage correction method is applied to the data (CCC = 0.449) and improves drastically when the Boxerman-Schmainda-Weiskoff (BSW) method is applied (CCC = 0.917). These results not only indicate that the BSW approach is superior to the gamma-variate approach, but also the BSW approach is on par with the benchmark threshold of 0.922.

Figure 2 is an example of matched ∆R2*(t) curves between the in vivo and in silico data within a tumor and NAWM pixel (left) and the corresponding anatomical T1-weighted and CBV maps (right).


We demonstrate the development of a DSC-DRO based benchmark in order to support the standardization of image analysis. The development and application example of an anthropomorphic benchmark is ideal for software platforms (e.g. commercial, open-source, in-house) that expect realistic structures. For example, these software platforms may automatically identify regions of interest for analysis. In this work, we demonstrate the ability to use the benchmark for leakage correction methods. However, this benchmark will also be able to characterize numerous pre-processing (e.g. filtering and smoothing) and post-processing steps. We are expanding the benchmark to include higher-ordered perfusion parameters (e.g. cerebral blood flow), multi-echo DSC-MRI scans, and dynamic contrast enhanced (DCE) scans.


NIH/NCI R01 CA213158-01


[1] Welker K, Boxerman J, Kalnin A, et al. ASFNR recommendations for clinical performance of MR dynamic susceptibility contrast perfusion imaging of the brain. AJNR. Am. J. Neuroradiol. 2015;36:E41-51

[2] Boxerman JL, Quarles CC, Hu LS, et al. Consensus recommendations for a dynamic susceptibility contrast MRI protocol for use in high-grade gliomas. Neuro-Oncol. 2020; 22(9): 1262-1275.

[3] Kaufmann TJ, Smits M, Boxerman J, et al. Consensus recommendations for a standardized brain tumor imaging protocol for clinical trials in brain metastases. Neuro-Oncol. 2020; 22(6): 757-772.

[4] Semmineh N, Bell L, Stokes A, Hu L, Boxerman J, Quarles C. Optimization of Acquisition and Analysis Methods for Clinical Dynamic Susceptibility Contrast (DSC) MRI Using a Population-based Digital Reference Object. AJNR Am J Neuroradiol 2018.

[5] Bell LC, Semmineh NB, An H, et al. Evaluating the Use of rCBV as a Tumor Grade and Treatment Response Classifier Across NCI Quantitative Imaging Network Sites: Part II of the DSC-MRI Digital Reference Object (DRO) Challenge. Tomography. 2020; 6(2): 203-208.

[6] Semmineh NB, Stokes AM, Bell LC, Boxerman JL, Quarles CC: A Population-Based Digital Reference Object (DRO) for Optimizing Dynamic Susceptibility Contrast (DSC)-MRI Methods for Clinical Trials. Tomogr (Ann Arbor, Mich) 2017.

[7] Boxerman JL, Schmainda KM, Weisskoff RM. Relative cerebral blood volume maps corrected for contrast agent extravasation significantly correlate with glioma tumor grade, whereas uncorrected maps do not. AJNR. Am. J. Neuroradiol. 2006;27:859–67.

[8] Zhang, Y, Brady, M, Smith, S. Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm. IEEE Trans Med Imag, 20(1):45-57, 2001.

[9] M. Jenkinson and S.M. Smith. A global optimisation method for robust affine registration of brain images. Medical Image Analysis, 5(2):143-156, 2001.


Figure 1: The original DSC DRO encompasses 10,000 tumor pixels and NAWM pixels as labeled in this figure. Note that this DRO is population-based and spans both low- and high-grade tumor DSC time curves, and is not anthropomorphic in appearance which may create hurdles when attempting software validation.

Figure 2: An example of matched ∆R2*(t) curves between in vivo and in silico data within a tumor and NAWM pixel (Fig 2a) and the corresponding anatomical T1-weighted and CBV maps (Fig 2b).

Table 1: Preliminary data demonstrating the ability to use the DRO as a benchmark for leakage correction algorithms. Using the consensus acquisition protocol, we computed CBV for an intact-BBB and a disrupted-BBB. We chose two leakage correction methods to compare for CBV calculation: Boxerman-Schmainda-Weisskoff (BSW) and the gamma-variate (GV) methods. Using our ground truth CBV maps, we can compute the concordance correlation coefficient (CCC) between various estimated CBVs across the 10,000 pixels.

Proc. Intl. Soc. Mag. Reson. Med. 29 (2021)