Motion tracking using simultaneous MR and 4D ultrasound acquisition for image guided radiation therapy
Thomas KF Foo1, Bo Wang1, Jhimli Mitra1, David Mills1, L Scott Smith1, Heather Chan1, Aqsa Patel1, Shourya Sarcar1, Eric Fiveland1, Warren Lee1, Sydney Jupitz2, Alan McMillan3, James H Holmes3, Wes Culberson2, Michael Bassetti4, Andrew Shepard2, and Bryan Bednarz2

1GE Global Research, Niskayuna, NY, United States, 2Medical Physics, University of Wisconsin, Madison, WI, United States, 3Radiology, University of Wisconsin, Madison, WI, United States, 4Human Oncology, University of Wisconsin, Madison, WI, United States


An MR-compatible 4D ultrasound probe allows hands-free, simultaneous MR and ultrasound image acquisition. This new imaging capability provides a path for tracking tumor target motion during radiation therapy, as an alternative to an integrated MR-LINAC system. To facilitate this, the ability to track the motion of fiducial markers as an indication of respiratory state is essential. In our approach, as the MR images are acquired outside of the radiation therapy procedure, motion tracking of endogenous ultrasound fiducials is proposed to determine respiratory states.


MRI provides good delineation of tumor margins and is frequently used to guide interventional procedures, such as biopsy and surgical interventions by registering pre-acquired MR images to real-time ultrasound images. Real-time tracking of tumor targets for guiding radiation therapy has been demonstrated with combined MR-LINAC systems [1-4]. Simultaneous MR-ultrasound acquisition is proposed as an alternative approach to allow real-time image guidance during radiation therapy in a LINAC that uses pre-acquired MR images for assessing the motion of the tumor target. This approach facilitates the utilization of MR images with excellent soft-tissue contrast for image guidance without the need for a more complex MR-LINAC system.


An MR-compatible 4D ultrasound probe has been developed to allow hands-free, simultaneous MR-ultrasound image acquisition [5,6]. Pre-treatment simultaneously acquired MR-ultrasound images directly associates each MR image to a respiratory state as determined by 4D ultrasound. During therapy, the ultrasound probe used in the LINAC detects the respiratory state and matches that state to that acquired during the pre-treatment phase. In this manner, an MR image that corresponds to the current respiratory state can be displayed and used for tumor target tracking in the liver.

To determine the respiratory state, an endogenous fiducial marker, such as a blood vessel is first selected in a reference ultrasound image. The x-y-z displacement of the fiducial marker is detected using a fast block matching algorithm [7]. The displacement along each direction represents a continuum of respiratory states. These are then clustered into discrete respiratory states using agglomerative or hierarchical clustering [8,9]. Rather than clustering based on displacement in Euclidean space, distance measures using cosine distance or cosine similarity is used instead. The labels for each cluster are randomly assigned but the different respiratory states from positional differences are naturally clustered together. This tracking and clustering must be completed in <200 ms, hence the need for a GPU.

All studies were conducted in a GE SIGNA MR750 and Premier 3.0T MRI scanners with the MR-compatible ultrasound probe driven by a GE Vivid E95 ultrasound scanner. Ultrasound and MR data were streamed to an Intel Xeon workstation (512 GB RAM and NVIDIA GeForce GTX Titan X GPU). TTL signals from the MR scanner indicating the start and end of data acquisition for each slice location match MR images to 3D ultrasound volumes at each time point (Figure 1). Both the 2D MR fast gradient echo (FGRE) and 4D ultrasound images were acquired at about 4 fps.

Four healthy volunteers were consented under IRB-approved protocols. The ultrasound probe was placed on the right lateral abdominal wall to image the liver. Simultaneous MR-ultrasound images were acquired and used for testing the clustering and matching algorithms. To assess accuracy, MR images at each time point were subtracted from a reference state, and the mean squared error (m.s.e.) in a region-of-interest (Figure 2) was measured as an indication of the goodness of the match. This was done on the MR images as speckle in ultrasound images would result in a high m.s.e.


No significant artifacts were noted in the simultaneously acquired MR or ultrasound images. The ultrasound probe generated only short-range susceptibility artifacts (Figure 2). Ultrasound images acquired during active MR acquisition were generally artifact-free (Figure 3). In the initial experiments, it was noted that there was noticeable respiratory drift in the ultrasound data (Figure 4). This was attributed, in part, to the strap holding the ultrasound probe to the abdomen. With a more rigid strap, the magnitude of the drift was reduced.

For a single, 1-min MR acquisition at one slice location, the subtracted images and m.s.e. plots are shown in Figure 5 (using frame #23 as the reference). With clustering into 7 respiratory states and selecting a reference frame in the end-expiration state, the m.s.e. of the difference images in an ROI in the liver is shown. The subtracted images show the extent of the mis-registration for an identified end-expiration state (#71) compared to some other respiratory state (#78). As radiation therapy is typically performed with patients at end-respiration, the m.s.e. of the identified phases in the end-respiration state illustrates the feasibility of the MR-ultrasound approach for image guidance during radiation therapy.


The ultimate validation of the proposed approach is to conduct an initial series of MR-ultrasound acquisitions, and a second series after repositioning of the ultrasound probe. Without repositioning in a single study, the results of the tracking and clustering have been encouraging. The primary issue is how well the tracking and clustering correlates when there is some degree of difference in the repositioning of the ultrasound probe, the effect of respiratory drift, and physiologic changes between the pre-treatment and therapy phases.


Funding support: NIH R01CA190298.


1. Park JM, et al. Radiother Oncol 2016; 120: 279-85.

2. Mutic S, et al. Sem Rad Oncol 2014; 24: 196-9.

3. Raaymakers BW, et al. Phys Med Biol 2009; 54: N229-37.

4. Kermeijer KGW, et al. Front Oncol 2016; 6: 215.

5. Foo TKF, et al. Proc. 26th ISMRM, p. 4416.

6. Lee W, et al. IEEE Intl Ultrasonics Symp 2017; doi: 10.1109/ultsym.2017.8092223.

7. Shepard AJ, et al. Med Phys 2017; 44: 5889-900.

8. Pedregosa F, et al. J Machine Learning Research 2011; 12: 2825-30.

9. Zepeda-Mendoza ML, et al. (2013) Hierarchical Agglomerative Clustering. In: Dubitzky W, et al. (eds) Encyclopedia of Systems Biology. Springer, New York, NY.


Figure 1: Schematic of data streaming from MR and ultrasound to a GPU-equipped workstation, with the MR-compatible 4D ultrasound probe shown inset.

Figure 2: (left) T2-weighted FSE (TE/TR = 86.7/10909 ms), and (right) T1-weighted gradient echo FLEX images showing the approximate position of the ultrasound probe during simultaneous acquisition, and artifacts from the MR-compatible ultrasound probe.

Figure 3: Single frame showing 2 orthogonal views from 4D ultrasound. Acquisition was with harmonic mode (1.7/3.4 MHz) to a depth of 15-cm.

Figure 4: Beam-spaced displacement measures for ultrasound data and the result of the clustering. Images acquired corresponded to a 200-260 phase MR acquisition at each slice location.

Figure 5: Mean-squared error as measured over 200 phases in MR images of the liver. The location of the ROI in the liver, and the m.s.e. of the identified end-expiration phases (red circles) are shown. The difference images are also shown for the end-respiration state (#71) and some other state closer to inspiration (#78). The MR multi-phase images were clustered into 7 discrete respiratory states.

Proc. Intl. Soc. Mag. Reson. Med. 27 (2019)