Reliability of CTA Alone and in Combination with CTP in the Detection of Cerebral Vasospasm: Methodologic Issues on Reliability to Avoid Misinterpretation

I was interested in the article by Heitkamp et al¹ published in the March 2024 issue of the American Journal of Neuroradiology. The authors investigated the reliability of CTA alone and in combination with CTP in the detection of cerebral vasospasm and the decision to initiate endovascular treatment. Three neuroradiologists were asked to grade 15 intracranial artery segments in 71 cases using a tripartite scale (no, mild <50%, or severe >50% vasospasm). They reported that the interrater reliability for per-segment analysis of vessels was highly variable (κ = 0.16–0.61). When one focused on senior raters, the addition of CTP images resulted in higher interrater reliability for severe vasospasm (κ = 0.28 versus κ = 0.46) and subsequently higher concordance (κ = 0.23 versus κ = 0.73) for the decision to start endovascular treatment.

Although the article provides insight into the decision about using CTA in combination with CTP to increase the reliability of endovascular treatment, its conclusion is limited in 2 important methodologic and statistical ways. First, the value of κ depends on the prevalence in each category. It is possible to have the prevalence of concordance cells equal to 90% and discordance cells equal to 10% and, however, obtain different κ values (0.44 as moderate versus 0.81 as very good, respectively).^2⇓-4 Second, the Fleiss κ is a statistical measure for assessing the reliability of agreement among a fixed number of raters when assigning categoric ratings to a number of items. This use contrasts with other applications of κ such as the simple Cohen κ, which works only when assessing the agreement between not more than 2 raters or the intrarater reliability.

The authors concluded that CTA alone offers only low interrater reliability in the graduation of cerebral vasospasm. However, using CTA in combination with CTP might help, especially senior neuroradiologists, to increase the interrater reliability to identify severe vasospasm following aneurysmal SAH and to increase the reliability regarding endovascular treatment decisions. Considering the above-mentioned methodologic and statistical issues, such conclusions can easily be a misleading message.^2⇓-4 Briefly, the prevalence-adjusted and bias-adjusted Fleiss κ should be applied to correctly assess the reliability of CTA alone and in combination with CTP in the detection of cerebral vasospasm and the decision to initiate endovascular treatment.

• © 2024 by American Journal of Neuroradiology