Evaluation of the Statistical Detection of Change Algorithm for Screening Patients with MS with New Lesion Activity on Longitudinal Brain MRI

DISCUSSION

In this single-center study evaluating the utility of the automated SDC algorithm in assisting human readers to detect new lesions on longitudinal FLAIR images, we found that SDC was able to provide a perfect new lesion-detection sensitivity on a subject level. SDC achieved this excellent level of detection sensitivity while providing a moderate subject-level specificity of 0.67 (meaning about 2 of every 3 subjects without new lesion activity were correctly classified). These operating characteristics allow SDC to be used as a valuable screening tool and could accelerate the interpretation time of cases in which no new lesions are identified by SDC. In our study, for example, 114 of 200 patients had no new lesions on follow-up confirmed by SDC, suggesting that a more rapid expert human review of these cases (57% of our entire cohort) may be feasible. Such an increase in efficiency may enable expert human readers to allocate more time to interpret MR imaging cases flagged by SDC as being potentially positive for new lesions.

In the conventional radiology workflow, detecting new lesions that formed between 2 longitudinal scans of patients with MS is often performed by radiologists and other clinicians manually on a PACS monitor by comparing a large number of paired-but-imperfectly aligned FLAIR images. This approach is often time-consuming, mentally demanding, and error-prone, especially if there is a substantial image misalignment due to the difference in head orientation between the 2 scans. SDC overcomes these challenges by providing an automated detection of potential new lesions, which are then visually indicated to the reader on a pair of longitudinally registered images (Fig 1). Conceptually, SDC is formulated as an optimal change detector applied to the subtraction image, which can be proved mathematically by the Neyman-Pearson lemma to provide the best detection power for a given false-positive rate.³⁰

By design, SDC mimics a human reader in 2 key aspects: First, it applies an adaptive intensity threshold to the subtraction image on the basis of the level of noise in the image. For example, the threshold for the longitudinal signal change is automatically increased by SDC for more noisy images, which can be regarded as equivalent to setting a higher level of trust as often performed by a human reader when dealing with noise. Second, SDC uses signal from voxels in a local neighborhood to calculate the test statistic, helping to increase the detection reliability. Similar to change detection by humans, spatially spurious signals on the subtraction image are encoded by the algorithm to have a lower likelihood of being identified as true change. By virtue of being capable of operating at a very high sensitivity and a reasonable specificity, SDC enables the new lesion identification problem to be shifted from the traditionally difficult task of locating new lesions on unmarked images to a much easier task of confirming true-positives and eliminating false-positives in areas already marked by SDC. Therefore, SDC has a great potential to shorten new-lesion detection time while reducing reader fatigue.

Several studies have demonstrated the benefits of detecting new lesions on a subject level on longitudinally registered or subtracted standard 1-mm isotropic 3D FLAIR images. A study by Galletto Pregliasco et al,¹⁵ performed in 94 patients with MS, showed that using an automated coregistration-fusion method improved the detection rate of subjects with at least 1 new lesion from 46% to 59%. In another study conducted by Eichinger et al,¹³ in 106 patients with MS, 58% of subjects were identified as having at least 1 new lesion on the FLAIR subtraction image, with a similar proportion (59%) identified using the conventional reading method. In this study of 200 patients with MS, we saw an improvement from 8% by Reader to 15% by Reader + SDC in the subject-level detection rate using a minimum lesion volume threshold of 15 mm³. The observed relatively rare event of a new lesion occurring in our cohort is likely related to different patient characteristics.

Compared with the emerging neural network–based deep learning approaches for new lesion detection, the SDC algorithm does not require data labeling and specialized hardware (such as a powerful graphics processing unit) for network training. SDC can also be used as a computer-assisted detection and segmentation tool to help humans create labeled image data for network training more efficiently. In addition to the ease of implementation, SDC provides relatively fast processing of the subtraction image, taking only a few seconds per case using our Matlab implementation, which can be further improved for interactive use by a C/C++ implementation. However, the longitudinal registration step to generate the subtraction image takes about 2 minutes per case in our processing pipeline. The development of rapid image registration algorithms^31⇓-33 will, therefore, be essential for a successful deployment of SDC in a routine interactive workflow. Alternatively, SDC can be implemented as part of the image-reconstruction process, assuming that images from the prior study are available on the scanner. Further studies are needed to evaluate the clinical feasibility of these approaches.

The coregistration steps involved in the SDC processing pipeline are key elements of the SDC approach and offer a highly precise registration not possible in routine image-interpretation software solutions embedded in clinical PACS systems. In our pipeline implementation, we chose to perform longitudinal FLAIR image registration by registering the image to the T1-weighted half-space for 2 reasons: 1) The T1-weighted image has the same resolution (1-mm isotropic) but much better GM/WM contrast than the FLAIR image, and 2) in our data, the T1-weighted image has less noise than the FLAIR image because T1-weighted imaging was acquired with an acceleration factor of 2, while FLAIR was acquired with an acceleration factor of 4 (to keep the scan time reasonable). Consequently, the longitudinal registration was found to be generally more accurate when using T1-weighted images rather than FLAIR images. While a 1-mm isotropic 3D T1-weighted sequence is part of the MS brain MR imaging protocol recommended by the most recent clinical consensus,³ performing direct registration between the 2 longitudinal FLAIR images is a viable option when such T1-weighted images are not available.

This study has several limitations. First, the ground truth was determined on the basis of expert readings from only 1 experienced board-certified neuroradiologist in consensus with an MR imaging physicist with expertise in MS brain image analysis. Furthermore, because these readers were aware of the use of SDC, our study design was unable to fully eliminate the possibility of reader bias. While this study design is appropriate for the current early-stage evaluation of SDC technology development, the diagnostic performance metrics such as sensitivity and specificity reported in this study require further clinical validation in future multireader multicase studies³⁴ involving multiple board-certified radiologists and using a clinical consensus as the criterion standard. This is a key validation step to ensure that image-analysis techniques are ready for regulatory approval and clinical adoption.

Second, intracortical and subpial GM lesions were not considered because these lesions are difficult to visualize reliably with existing routine 3T imaging sequences, including FLAIR.³⁵ Third, in the current study design, the Reader method was performed in the routine clinical workflow by readers different from those involved in the Reader + SDC method. These differences make it difficult to determine the clinical significance, if any, of discrepancies in new lesion identification noted between the clinical radiologic report and retrospectively performed SDC-enhanced readings. For example, the radiologist issuing the clinical report in the Reader method had real-time access to the electronic health record and contrast-enhanced imaging while being subject to time constraints imposed in the clinical workflow and being responsible for total brain MR imaging interpretation, not only new lesion detection on FLAIR.

While our study design allows a direct comparison of SDC with the radiology report, regarded as a clinically established benchmark, future work will be focused on integrating the SDC algorithm into the clinical workflow, which will allow prospective comparison studies to be performed in a real-world setting. In the context of active MS, this will also enable the comparison of gadolinium-based T1-weighted sequences with emerging noncontrast approaches for new acute lesion detection such as those combining FLAIR subtraction with quantitative susceptibility mapping.³⁶ Fourth, a minimum lesion volume cutoff of 15 mm³ was used to mitigate the effect of noise on the false-positive rate of the SDC algorithm. Such punctate MS lesions are often considered inconsequential,¹ though further evidence on their role in the disease progression and outcome may be needed. Fifth, the PACS system used for clinical interpretations was not equipped with the capability to adjust for differences in scan angle or patient positioning or to coregister 3D images to enable section-by-section comparisons accurately. This capability is being enhanced in more modern PACS systems but was not readily available in the version of the PACS system used during the time of the study scanning. Sixth, validation on FLAIR images acquired on MR imaging scanners from other vendors or with different acquisition parameters is very important for successful clinical translation of SDC and will be addressed in a future study. Finally, linear longitudinal brain registration was used in this study, which was deemed sufficiently accurate for the annual follow-up interval in our cohort but may not capture nonlinear changes in the brain morphology during a longer interscan period. The use of a deformable motion model in the registration algorithm may be considered in such scenario.^12,24