Abstract
BACKGROUND AND PURPOSE: Cerebral vasospasm is a common complication of aneurysmal SAH and remains a risk factor for delayed cerebral ischemia and poor outcome. The interrater reliability of CTA in combination with CTP has not been sufficiently studied. We aimed to investigate the reliability of CTA alone and in combination with CTP in the detection of cerebral vasospasm and the decision to initiate endovascular treatment.
MATERIALS AND METHODS: This is a retrospective single-center study including patients treated for aneurysmal SAH. Inclusion criteria were a baseline CTA and follow-up imaging including CTP due to suspected vasospasm. Three neuroradiologists were asked to grade 15 intracranial arterial segments in 71 cases using a tripartite scale (no, mild <50%, or severe >50% vasospasm). Raters further evaluated whether endovascular treatment should be indicated. The ratings were performed in 2 stages with a minimum interval of 6 weeks. The first rating included only CTA images, whereas the second rating additionally encompassed CTP images. All raters were blinded to any clinical information of the patients.
RESULTS: Interrater reliability for per-segment analysis of vessels was highly variable (κ = 0.16–0.61). We observed a tendency toward higher interrater reliability in proximal vessel segments, except for the ICA. CTP did not improve the reliability for the per-segment analysis. When focusing on senior raters, the addition of CTP images resulted in higher interrater reliability for severe vasospasm (κ = 0.28; 95% CI, 0.10–0.46 versus κ = 0.46; 95% CI, 0.26–0.66) and subsequently higher concordance (κ = 0.23; 95% CI, −0.01–0.46 versus κ = 0.73; 95% CI, 0.55–0.91) for the decision of whether endovascular treatment was indicated.
CONCLUSIONS: CTA alone offers only low interrater reliability in the graduation of cerebral vasospasm. However, using CTA in combination with CTP might help, especially senior neuroradiologists, to increase the interrater reliability to identify severe vasospasm following aneurysmal SAH and to increase the reliability regarding endovascular treatment decisions.
ABBREVIATIONS:
- ACA
- anterior cerebral artery
- aSAH
- aneurysmal SAH
- DCI
- delayed cerebral ischemia
- IQR
- interquartile range
- TCD
- transcranial Doppler sonography
Despite advances in the acute management of aneurysmal SAH (aSAH), cerebral vasospasm is a frequent complication.1 It remains a risk factor of delayed cerebral ischemia (DCI) and an important predictor of poor outcome.2 Conservative treatment strategies include induced hypertension, maintaining euvolemia, and oral or intravenously administered calcium channel blockers to prevent and treat DCI. When conservative treatment strategies fail, the application of an intra-arterial vasodilator or angioplasty of the affected vessel segments has been proposed.3 DSA is considered the criterion standard to assess cerebral vasospasm.4 However, DSA is a time-consuming diagnostic technique and can be accompanied by rare-but-severe complications due to its invasive nature.5 Transcranial Doppler sonography (TCD) is used as a noninvasive screening method to determine patients at risk of cerebral vasospasm. However, this technique is highly operator-dependent, requires a sufficient acoustic window, and has a low sensitivity.6 Therefore, CTA is widely used to evaluate vessel narrowing in patients after aSAH and to guide further invasive treatment decisions.3 Unfortunately, it offers only a low sensitivity in detecting cerebral vasospasm, and interrater reliability has been reported to be moderate at best.7⇓-9 However, the interrater reliability of CTA in combination with CTP to assess cerebral vasospasm and guide treatment decisions has not been sufficiently studied.8,10,11
The aim of this study was to investigate the reliability of CTA alone and in combination with CTP in the detection of cerebral vasospasm and the decision to initiate endovascular treatment. We hypothesized that adding CTP to CTA in the assessment of vasospasm will increase the interrater reliability. In addition, we hypothesized that CTP would be useful in clinical decision-making regarding the need for invasive treatment.
MATERIALS AND METHODS
Study Design
This retrospective cohort study was conducted in accordance with the Guidelines for Reporting Reliability and Agreement Studies (GRAAS) (https://www.equator-network.org/wp-content/uploads/2012/12/GRRAS-checklist-for-reporting-of-studies-of-reliability-and-agreement.pdf).12 We conducted a single-center retrospective study at our tertiary stroke center, including all patients treated for aSAH between January 2019 and December 2022. Inclusion criteria for our study were the following: a complete baseline image data set including CTA, suspected vasospasm with follow-up multimodal CT including perfusion imaging, and subsequent DSA with the decision for endovascular treatment.
Imaging data were assessed in 2 stages by 3 neuroradiologists, each with different levels of experience, spanning 3, 9, and 15 years: 1) The first rating contained nonenhanced cranial CT and CTA images at baseline and follow-up imaging. 2) The second rating additionally included CTP images of the same patients. The second rating was performed at least 6 weeks after the first one to minimize recall bias. All raters were blinded to clinical information, the reason for follow-up imaging, and whether all patients received endovascular treatment. The decision to perform endovascular treatment was made by the treating neurointerventionalist. In 8 of 71 cases of vasospasm, the endovascular treatment was performed by raters participating in this study. Clinical data including patient, aneurysm, and treatment characteristics were obtained from medical records. Furthermore, the indication for conducting follow-up CT, along with the time interval between the admission and follow-up CT, was retrieved.
The study was approved by the local ethics committee (Ärztekammer Hamburg, Germany; 2022–300245-WF). All study protocols and procedures were conducted in accordance with the Declaration of Helsinki. Patient consent was not needed due to the retrospective nature of the study.
Diagnostic Protocol for the Detection of Vasospasm
Daily clinical and TCD assessments were conducted to detect neurologic deterioration or sonographic signs of vasospasm. TCD was performed by trained physicians or technicians, who measured the mean velocity in both the MCA and anterior cerebral artery (ACA) whenever possible. Following the Association of the Scientific Medical Societies in Germany (AWMF; https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4106509/) guidelines,13 the TCD measurements were recorded as the mean velocity (cubic centimeters). Elevated TCD measurements in the MCA were identified by a mean velocity exceeding 140 cm/s or doubling within a 24-hour period. If patients experienced a new neurologic deficit, deterioration in consciousness, or elevated TCD measurements, multimodal imaging including CTA and CTP was performed. Also refer to the Online Supplemental Data illustrating our local protocol of a diagnostic algorithm for the detection of cerebral vasospasm after aSAH.
Imaging and Ratings: Classification and Categories
All CT scans were performed on a 256 dual-section scanner (Somatom Definition Flash; Siemens). Details on image acquisition can be found in the Online Supplemental Data. One data set per patient included axial nonenhanced cranial CT images, thin-section axial CTA, and axial, coronal, and sagittal MIP images. In addition, each CTP data set for the second rating contained whole-brain coverage of the perfusion CBF, CBV, MTT, and time-to-maximum. CTP images were automatically processed using syngo.via (Version VB30A_HF91; Siemens). The imaging data and corresponding DICOM source information were fully anonymized before being provided to the raters in digital format. The raters performed the evaluations using a commercially available DICOM viewer (Horos: Version 3.3.6; Horosproject). According to a previous study,14 raters were asked to grade vasospasms by visual judgment using a tripartite scale for each vessel separately. The scale included no vasospasm, mild vasospasm <50% narrowing, and severe vasospasm >50% narrowing. The admission CTA was used as a reference when evaluating vasospasm on follow-up CTA.7,11,15 Arterial segments were predefined as follows: supraclinoid ICA; M1 and M2 segments of the MCA; A1 and A2 segments of the ACA; P1 and P2 segments of the posterior cerebral artery, as well as the basilar artery. Finally, readers were requested to determine whether endovascular treatment was recommended for the patients. Within the framework of the second rating, raters were additionally asked whether perfusion deficits were present.
Statistical Analysis
Descriptive statistics were performed to display clinical information, including patient, aneurysm, and treatment characteristics as well as the reason for follow-up CT being performed. The normality of data distribution was assessed using Shapiro-Wilk tests. Continuous variables are presented as mean (SD) for normally distributed variables and medians with interquartile ranges (IQR) for non-normally distributed variables. Categoric variables are described as counts and percentages. According to a previous study,16 a per-patient assessment with dichotomization between no vasospasm and severe vasospasm was calculated if the rater estimated narrowing to be severe for at least 1 of the ICA, M1, M2, A1, A2, P1, P2, or basilar segments. Interrater agreement for dichotomized variables was assessed using the Fleiss κ for all 3 raters and the Cohen κ for the 2 senior raters (M.B. and C.T.) using 95% bias-corrected confidence intervals obtained with 1000 bootstrap resampling. The interrater agreement of each vessel’s vasospasm rating was calculated using the Krippendorff α. Intrarater agreement was assessed using the Cohen κ for dichotomized variables and the Krippendorff α for the per-segment analysis of each vessel.
An illustration depicting the various statistical tests is included in the Online Supplemental Data. The level of agreement was interpreted according to Landis and Koch17 (slight agreement, 0–0.2; fair agreement, 0.21–0.4; moderate agreement, 0.41–0.6; substantial agreement, 0.61–0.8; and almost perfect agreement 0.81–1.0). Significant differences between point estimates or between k values were considered to exist when 95% confidence intervals did not overlap. The data analysis was conducted using Stata 18.0 (Stata/MP 18.0; StataCorp).
Data Availability
Data supporting the findings of this study are available from the corresponding author on reasonable request.
RESULTS
A total of 206 patients were treated for aSAH during our inclusion period. Of those, 47 patients met the inclusion criteria. One patient was excluded from final analysis due to severe imaging artifacts caused by coiling. The final analysis incorporated imaging data from 46 patients, comprising 25 patients who underwent CT imaging more than once. Therefore, a total of 71 CT examinations was assessed. Among the 25 patients who underwent 2 CT examinations, the median time between the first and second CT was 49 hours (IQR, 26–99 hours). The mean age was 56.6 (SD, 12.6) years, and 71.7% of patients were female. The median Hunt and Hess score was 3 (IQR, 2–4), and the median modified Fisher scale score was 4 (IQR, 3–4). Aneurysms were most frequently located at the ACA (37.0%) and ICA (34.8%). Most aneurysms were treated endovascularly, accounting for 82.6% of all cases. For further information regarding patient, aneurysm, and treatment characteristics refer to Table 1.
Patient baseline, procedural, and outcome characteristics
Reasons for suspected vasospasm and subsequent follow-up imaging are shown in Table 2. The median time from ictus to follow-up imaging was 9 days (IQR, 6–11.5 days). The most frequent indication for follow-up imaging was a neurologic deterioration (eg, reduced level of consciousness or a new focal deficit) in 42.3% of all cases. Solely elevated TCD profiles accounted for 23.9%, and neurologic deterioration together with elevated TCD profiles accounted for 26.8% of reasons to perform follow-up CT. Further reasons were screening for vasospasm in comatose patients without a TCD acoustic window (5.7%) and decreased partial pressure of brain tissue oxygen (1.4%). In 22 cases, patients were intubated while receiving follow-up imaging.
Reasons to perform follow-up CT examination (n = 71)
Table 3 and Figure 1 provide the distribution of interrater reliability stratified by each vessel segment. The left A1 segment accounted for the highest interrater reliability with substantial agreement (κ = 0.61; 95% CI, 0.48–0.74). In general, we observed a tendency toward higher interrater reliability in proximal vessel segments (M1, A1, P1), except for the ICA (Figs 1 and 2). The addition of perfusion maps in the second assessment did not substantially improve interrater reliability for the individual vessels (Fig 1). In a subsequent analysis, we investigated the interrater reliability for the detection of severe vasospasm in any arterial segment. Interrater reliability was fair for the first (κ = 0.27; 95% CI, 0.11–0.42) and second (κ = 0.31; 95% CI, 0.15–0.46) ratings of all raters. Most interesting, there was an increase from fair agreement in the first rating (κ = 0.28; 95% CI, 0.10-0.46) to moderate agreement in the second rating (κ = 0.46; 95% CI, 0.26-0.66) when focusing on senior raters. In addition, the restriction to proximal vessels (ICA, A1, M1, P1) further increased point estimates of the interrater reliability of the senior raters from κ = 0.51 (95% CI, 0.31–0.72) to κ = 0.59 (95% CI, 0.38–0.79) during the 2 assessments.
The interrater reliability of first and second ratings (Krippendorff α) is stratified by vessel segments. Vasospasm was graded on CTA using a tripartite scale for each vessel separately. The scale included no vasospasm, mild vasospasm <50%, and severe vasospasm >50%. Krippendorff α values are displayed by black triangles and blue squares for the first and second ratings, respectively. The black and blue lines indicate the 95% CIs.
Illustrative case of a patient without left-hemispheric vasospasm on follow-up imaging. The ICA, A1, and A2 segments of the ACA as well as the M1 and M2 segments of the MCA are highlighted in color. The color graduation indicates the interrater reliability (Krippendorff α) for the first rating. The graduation ranges from slight (α = .0) to substantial (α = .8) agreement. Note that proximal segments accounted for higher interrater reliability except for the ICA.
Interrater reliability and agreement for the graduation of vasospasm on CTa
Regarding the decision about whether endovascular treatment was indicated, the interrater reliability based on CTA alone was fair for all raters (κ = 0.23; 95% CI, 0.06–0.39) and reached moderate agreement (κ = 0.47; 95% CI, 0.30–0.64) when adding CTP images for the second assessment. If we took only senior raters into account, interrater reliability increased from fair (κ = 0.23; 95% CI, −0.01−0.46) to substantial (κ = 0.73; 95% CI, 0.55−0.91) across the course of the 2 ratings. Refer to Table 4 for further statistics regarding diagnosis and treatment decisions of vasospasm on CT.
Interrater reliability and agreement regarding graduation and treatment decisions of vasospasm on CT
Regarding intrarater agreement, we observed a broad spectrum of agreement among the raters, ranging from slight to substantial when focusing on the per-segment analysis of each vessel (Online Supplemental Data) separately. Fair intrarater agreement was observed for all 3 raters (rater 1: κ = 0.22; 95% CI, −0.02–0.47; rater 2: κ = 0.28; 95% CI, 0.06-0.51; rater 3: κ = 0.34; 95% CI, 0.17–0.52) concerning the question of whether to perform endovascular treatment (Online Supplemental Data).
DISCUSSION
In this single-center retrospective study, we investigated the interrater reliability of CTA alone and in combination with CTP for the detection of cerebral vasospasm after aSAH. Our second objective was to determine the interrater reliability regarding the CT-based decision about whether endovascular treatment was indicated. We observed that the interrater reliability of vasospasm on a vessel segment level was overall very low and highly variable. Higher interrater reliability was observed for proximal vessel segments except for the ICA. CTP did not improve the interrater reliability for the per-segment analysis. However, when focusing on senior neuroradiologists, we found that the addition of CTP images resulted in higher interrater reliability for severe vasospasm and subsequently higher concordance for the decision of whether endovascular treatment was indicated.
To date, most studies focused on the diagnostic accuracy of CTA compared with DSA for the detection of vasospasm.8,10,15,18 In previous studies, interrater reliability for detecting cerebral vasospasm on CTA alone ranged from moderate to substantial, challenging its sufficiency in diagnosing vasospasm and guiding further treatment decisions.7,15,18 We also observed low interrater reliability with CTA alone in our study, even when dichotomized for any severe vasospasm. One reason for the low or at least varying interrater agreements might be that there is no consensus on a definition of critical vessel narrowing or a reference standard for evaluating vasospasm, such as whether to use the baseline examination or the contralateral hemisphere as a comparison. For example, Letourneau-Guillon et al7 found 8 different classification systems with various cutoffs for the evaluation of vasospasm on CTA. Furthermore, the reduction of vessel diameters can be subtle, making it even more challenging when using grading systems with multiple increments. Especially in smaller vessels such as the M2 or A2 segments or even more distal, visual grading becomes more difficult. This issue might explain our observation of higher interrater reliability in proximal vessel segments, except for the ICA, which is consistent with findings in previous studies.7,18 Due to the relatively short and tortuous course of the supraclinoid ICA, raters may encounter challenges in establishing a detailed graduation of vasospasm in this vessel. Proximal vessels such as the M1 or the A1 segment typically have a larger diameter and a more extended horizontal course in comparison with their distal counterparts. As a result, it is assumed that changes in these proximal vessel calibers are easier to detect, particularly when contrasted with the inherently narrower calibers observed in distal segments. Therefore, focusing on proximal vessels may be more reliable for clinicians when evaluating vasospasm, but it also highlights the problem of accurately and reliably detecting relevant vasospasm in distal cerebral arteries.
It is not clear which cutoff value of vessel stenosis is most predictive of cerebral ischemia and would necessitate an adjustment of the treatment strategy.19,20 Thus, we also provided our observers with CTP images because these might indicate hemodynamically relevant vasospasm.21 We observed substantial agreement for all raters and almost perfect agreement for senior raters in the assessment of perfusion deficits. In contrast to CTA alone, perfusion imaging additionally yields insights into microvascular vasospasm, which may not necessarily affect large- or medium-sized vessels.22 It has been shown that CTP improves the detection of distal medium- and small-vessel occlusions in patients with embolic stroke,23 leading to the assumption that it can also enhance the detection of cerebral vasospasm. However, perfusion deficits due to vasospasm are certainly not confined to 1 vascular territory but can be diffuse or only affect the watershed areas between 2 vascular territories.24 This feature may be an explanation for the relatively low interrater reliability for the per-segment analysis despite additional CTP imaging data, because it can be challenging to assign the perfusion deficit to a distinct vessel segment.
We observed that CTP images increased point estimates regarding the detection of severe vasospasm and, clinically more important, the decision about whether endovascular treatment was indicated. This finding was particularly evident among experienced raters. In previous studies, CTA in combination with CTP has shown high diagnostic accuracy for vasospasm diagnosis, and perfusion deficits have been associated with DCI.10,25 So far, the role of perfusion imaging to guide treatment decisions in cerebral vasospasm and its benefit as a screening technique on clinical outcome remain unanswered. Perfusion imaging provides additional information about the blood supply at the tissue level and indicates possible tissue at risk. This information may help observers determine which of the caliber irregularities may be functionally relevant and therefore provides additional information to justify endovascular treatment options, such as intra-arterial administration of spasmolytic drugs or percutaneous transluminal angioplasty. Of note, it must be considered that CTP entails a substantially higher radiation exposure, especially in the case of multiple examinations.
The raters were not given any clinical information such as the presence of new focal neurologic symptoms or TCD elevations. However, information about the patient’s clinical status or neurologic deterioration is essential when assessing the risk of focal brain damage in patients with suspected vasospasm and guiding further treatment decisions. The decision to pursue endovascular treatment should be driven by a combination of the patient’s clinical and neurologic status as well as imaging findings including DSA.3 However, patients after aSAH are often sedated or comatose and cannot be evaluated clinically. In this patient cohort, the degree of vasospasm on multimodal CT imaging plays a crucial role in clinical decision-making. Although our study shows that CT screening alone for vasospasm should be used with caution, because interrater reliability for individual vessel segments is low, it may be helpful to focus on proximal vessels and to use perfusion imaging to identify hemodynamically relevant perfusion deficits.
Limitations
This study has some limitations. Patients were chosen on the basis of the completeness of imaging modalities, possibly introducing selection bias. Furthermore, the generalizability of our findings may be limited because for 25 patients, 2 data sets were included because these patients presented more than once with suspected vasospasm. In addition, our cohort of raters is all from the same institution, and we included only 1 junior rater, restricting the results concerning physicians with limited work experience. In 8 of 71 cases, the endovascular treatment was performed by raters participating in this study. Despite the anonymization of all cases, the raters could have recognized these cases, potentially introducing bias. The heterogeneity of the study cohort was reduced by including only patients who had undergone endovascular treatment, resulting in a scarcity of patients with mild vasospasm. Because the experimental setup of our study differs from the real-life assessment of patients, we can only speculate that responders devoted the necessary time and attention to respond as they would have when evaluating actual patients. Some patients who underwent CT examination were under general anesthesia. Because anesthetic drugs can impact cerebral hemodynamics, their use potentially confounds the interpretability of CTP. Of note, this study focused on the interrater reliability of CTA and CTP in diagnosing vasospasm and the resulting treatment decision. It was not the purpose of this study to investigate the diagnostic accuracy of these 2 modalities, which has been demonstrated previously.8,10,27 Therefore, we did not compare the CTA and CTP ratings with the DSA.
CONCLUSIONS
The grading of vasospasm on a per-segment analysis using CTA alone offers only low interrater reliability and is not a sufficient criterion to guide treatment decisions regarding endovascular therapy. However, perfusion imaging might help experienced neuroradiologists visualize perfusion deficits of presumed vasospastic origin and might improve interrater reliability regarding endovascular treatment decisions.
Footnotes
M. Bester and C. Thaler contributed equally to this work.
Disclosure forms provided by the authors are available with the full text and PDF of this article at www.ajnr.org.
References
- Received August 16, 2023.
- Accepted after revision December 2, 2023.
- © 2024 by American Journal of Neuroradiology
