Abstract
BACKGROUND AND PURPOSE: The best management of patients with persistent distal occlusion after mechanical thrombectomy with or without IV thrombolysis remains unknown. We sought to evaluate the variability and agreement in decision-making for persistent distal occlusions.
MATERIALS AND METHODS: A portfolio of 60 cases was sent to clinicians with varying backgrounds and experience. Responders were asked whether they considered conservative management or rescue therapy (stent retriever, aspiration, or intra-arterial thrombolytics) a treatment option as well as their willingness to enroll patients in a randomized trial. Agreement was assessed using κ statistics.
RESULTS: The electronic survey was answered by 31 physicians (8 vascular neurologists and 23 interventional neuroradiologists). Decisions for rescue therapies were more frequent (n = 1116/1860, 60%) than for conservative management (n = 744/1860, 40%; P < .001). Interrater agreement regarding the final management decision was “slight” (κ = 0.12; 95% CI, 0.09–0.14) and did not improve when subgroups of clinicians were studied according to background, experience, and specialty or when cases were grouped according to the level of occlusion. On delayed re-questioning, 23 of 29 respondents (79.3%) disagreed with themselves on at least 20% of cases. Respondents were willing to offer trial participation in 1295 of 1860 (69.6%) cases.
CONCLUSIONS: Individuals did not agree regarding the best management of patients with persistent distal occlusion after mechanical thrombectomy and IV thrombolysis. There is sufficient uncertainty to justify a dedicated randomized trial.
ABBREVIATIONS:
- CA
- contact aspiration
- IAT
- Intra-arterial thrombolysis
- IVT
- IV thrombolysis
- MT
- mechanical thrombectomy
- RCT
- randomized controlled trial
- SR
- stent retriever
Incomplete recanalization after mechanical thrombectomy (MT) and/or IV thrombolysis (IVT) can result in persistent distal occlusion of the middle, anterior, or posterior cerebral arteries or branches.1 Despite increased expertise and a multitude of new thrombectomy devices, incomplete recanalization remains a frequent clinical scenario, observed in almost 50% of patients in recent randomized trials.1,2
Better recanalization grades after thrombectomy are associated with improved clinical outcomes, but this post hoc observation does not mean that persistent distal occlusions should be treated.3⇓⇓⇓-7 Various interventions have been proposed for treating such occlusions, including small adjustable stent retrievers,8⇓⇓⇓-12 distal aspiration catheters,8,12 and intra-arterial thrombolytics, but there is no consensus on which management is best.12,13 Although these treatments can improve the reperfusion status of patients with reportedly little added risk,8⇓⇓⇓-12,14,15 there is currently no randomized evidence that these complementary treatments lead to better outcomes.
There have been several reliability studies on thrombolysis or thrombectomy decisions, but none have specifically addressed distal occlusions.16 Reliability and agreement studies on management decisions can measure the degree of clinical uncertainty and inform the design of randomized trials.17,18 With this aim in mind, we investigated the agreement of various experts in making decisions regarding the management of patients with persistent distal occlusion after MT or IVT.
MATERIALS AND METHODS
This study was prepared in accordance with the Guidelines for Reporting Reliability and Agreement Studies (GRRAS).19
Case Selection
We selected and assembled an electronic portfolio of catheter angiograms of 60 patients who underwent endovascular thrombectomy for acute ischemic stroke with large-vessel occlusion (internal carotid, proximal MCA, or basilar trunk occlusion) in a single comprehensive stroke center (Centre hospitalier de l’Université de Montréal [CHUM]) between December 2020 and July 2021. To minimize κ paradoxes,20 1 author (W.B.) selected the cases so that approximately one-third were cases for whom conservative management was expected to be a frequent choice, one-third of cases were those for whom rescue therapy was expected to be a frequent choice, and one-third of cases were “gray zone” cases for whom both options would likely be considered.
A wide variety of distal occlusions were included (middle, anterior, and posterior cerebral arteries) at various levels (such as M2-to-M4 segments of the MCA, A1-to-A5 segments of the anterior, and P1-to-P5 segments of the posterior cerebral arteries, as in previous reports).21⇓-23 In cases with multiple tandem occlusions, the most proximal was defined as the level of occlusion.
The number of cases in the present study (n = 60) was selected to cover a wide spectrum of distal occlusions and to ensure relatively small confidence intervals according to tables provided by Donner and Michael.24
Raters
Two hundred eleven clinicians were invited to participate, including 148 interventional neuroradiologists and 63 vascular neurologists from 6 different countries.
Agreement Study
Catheter angiograms were de-identified and uploaded onto a local secure server. Raters had no access to other imaging studies or clinical information other than sex, age, symptoms (ie, left or right motor deficit, aphasia), the initial NIHSS score at presentation, the ASPECTS or its variant for the posterior circulation on pretreatment imaging,25 pretreatment with IVT, the time from symptom onset to proximal recanalization (when available; or in case of unknown onset, the time from last seen well and from stroke discovery to proximal recanalization were provided), and the time from CT to proximal recanalization. Study data were collected and managed using the REDCap electronic data capture tools (https://www.project-redcap.org/) hosted at our institution.26,27
For each case, raters were asked to independently answer the 5 following questions:
1) Is rescue therapy (stent retriever [SR], contact aspiration [CA], or intra-arterial thrombolysis) an option for this patient (Yes/No)? 2) Is conservative management an option for this patient (Yes/No)? 3) What is your final best treatment choice? Choose one: rescue therapy; conservative management. If you choose rescue therapy, please choose one: SR, CA, combined use of SR and CA, and intra-arterial thrombolytic (ie, recombinant tPA or urokinase). 4) How confident are you regarding your final treatment choice? (answers in 10% increments, ranging from 0% to 100%) 5) Would you be willing to recruit this patient for a randomized controlled trial (RCT) that would give a 50% chance of rescue therapy and a 50% chance of conservative management? (Yes/No).
All raters were asked to perform a second reading session (with cases presented in a permuted order), at least 2 months later, to independently evaluate intrarater agreement.
Statistical Analysis
Descriptive statistics are presented using percentage for categoric variables and mean (SD) for continuous variables. The proportions of answers for each question among different groups of raters (background, specialty, years of experience) of the level of persistent occlusion (M2/A1-A2/P1-P2; M3/A3/P3; M4/A4-A5/P4-P5) were compared using a 1-way ANOVA. When applicable, we used pair-wise comparisons using a Bonferroni adjustment. Correlations between treatment choice and continuous variables (age, NIHSS score, ASPECTS, time between onset and proximal recanalization and between CT and recanalization, number of passes) were analyzed using the Pearson correlation coefficient. Confidence in decision-making (scale of 0–100) was analyzed using ANOVA. χ2 tests were used to compare the willingness to recruit patients in a RCT. Agreement between and within raters was measured using κ statistics and 95% bootstrap confidence intervals and was interpreted according to Landis and Koch.28 κ values ranged from −1 (perfect disagreement) to +1 (perfect agreement), with zero indicating no agreement among the raters other than that expected by chance alone. Analyses were performed using STATA (Version 16.1; StataCorp) and SPSS software (Version 26; IBM) with significance set at 5%.
RESULTS
Patients, treatment characteristics, and main outcomes of cases included in the portfolio are detailed in the Online Supplemental Data. Details regarding raters are provided in the Online Supplemental Data. Among the 211 clinicians who were invited to participate, 31 (14.7%) responded, including 8 vascular neurologists and 23 interventional neuroradiologists from 20 comprehensive stroke centers; 29 of 31 raters (93.6%) agreed to review the same cases in a permutated order at least 2 months later.
Rescue therapy was judged to be a treatment option by a mean of 20 (SD, 7) raters (minimum 6; maximum, 31). Interrater agreement regarding whether rescue therapy was an option was fair (κ = 0.22; 95% CI, 0.17–0.28) for all raters and subgroups (Online Supplemental Data).
Conservative management was judged to be an option by a mean of 25 (SD, 6) raters (minimum, 6; maximum, 31). Interrater agreement regarding conservative management was fair (κ = 0.21; 95%, CI, 0.13–0.30) for all raters and subgroups (Online Supplemental Data).
Overall, rescue therapy was more frequently selected as the final best management: 1116/1860 responses (60%; 95% CI, 59%–61%), with conservative management garnering 744/1860 votes (40%; 95% CI, 39%–41%) (P < .001) (Online Supplemental Data). Final treatment choices did not vary according to years of experience (P = .145), background (P = .153), or interventional practice (P = .897). The level of the occlusion influenced the final treatment choice: Rescue therapy was more frequently selected as the final management for M2/A1-A2/P1-P2 than for more distal occlusions (P < .001). The proportions of votes for each rescue therapy for all raters and subgroups and according to the level of occlusion are illustrated in Fig 1 and detailed in the Online Supplemental Data.
Proportions of final treatment choices for all cases and according to several levels of persistent distal occlusion. Note the strong correlation between the level of persistent distal occlusion and the choice of treatment.
The number of rescue therapy and conservative choices for each of the 60 cases is illustrated in Fig 2.
Proportions of votes for rescue therapy and conservative management for several levels of occlusion and the proportion of clinicians willing to include the patient in an RCT (line) for each case (n = 60). Red circles indicate cases with <50% of raters willing to include the patient in an RCT (4/29 for patients with M2/A1-A2/P1-P2 occlusion, none for M3/A3/P3 occlusions, and 2/15 for patients with M4/A4-A5/P4-P5).
The initial NIHSS score (0.279, P = .031) correlated with more rescue therapy choices. The previous use of IVT was associated with more conservative choices (P = .037).
Interrater agreement regarding the final management decision was “slight” (κ = 0.12; 95% CI, 0.09–0.14) and did not improve when answers were dichotomized (conservative management versus any rescue therapy); when subgroups of clinicians were studied according to background, experience, and specialty; or when cases were grouped according to the level of occlusion (Fig 3 and Online Supplemental Data).
Interrater agreement for final treatment choices (5 categories: conservative management, SR, CA, combined use of SR and CA, intra-arterial thrombolysis) was dichotomized as conservative management versus all others (2 categories) for all raters, according to background, practice specialty, years of experience, and level of occlusion. All κ values are well below the dashed “substantial” line (0.600). Y indicates years.
The proportion of final decisions for rescue therapy, intrarater agreement (ie, proportion of cases with the same judgment between both rating sessions), and the proportion of patients recruited in an RCT for each rater are illustrated in Fig 4.
Proportion of final decisions for rescue therapy (blue and orange bars, ordered by numbers and classified by specialty), intrarater agreement (ie, proportion of cases with same judgment between readings [red diamonds]), and proportion of patients recruited in an RCT (black circles) for each rater (n = 31). INR indicates interventional neuroradiology.
Raters changed their final decision (rescue therapy versus conservative treatment) between readings in a mean 25.5% of cases; 23/29 (79.3%) raters changed their final decision between readings in at least 20% of cases (Online Supplemental Data). Intrarater κ values regarding the best final management (conservative management versus any rescue therapy) reached a substantial level (ie, κ >0 .6) for 4/29 (13.8%) raters.
Clinicians were not very confident in their final decisions (mean confidence, 65%; minimum-maximum, 24%–68%). Recruitment of patients in an RCT comparing rescue therapy and conservative management is graphically displayed in Fig 2 (per patient) and Fig 4 (per rater). Trial participation was offered 69.6% of the time (1295/1860 Yes responses to question 5). In 54/60 (90%) cases, a majority (≥ 51%) of responders were willing to include the patient in a randomized trial. Willingness to be included in a trial did not vary according to years of experience (P = .624), background (P = .328), or specialty (P = .445) (Online Supplemental Data). Interrater agreement regarding recruitment in an RCT was slight (κ = 0.05; 95% CI, 0.03–0.09) (Online Supplemental Data).
DISCUSSION
The current study highlights the clinical uncertainty regarding the management of patients with persistent distal occlusions after MT or IVT. Overall, agreement on the management of persistent occlusions was barely above that expected by chance alone for all cases and all clinicians. The lack of agreement among clinicians has a more fundamental nature than a divergence in the opinions of experts from different specialties, for there was substantial variability at the level of individual clinicians themselves when they were asked on 2 different occasions to judge the management of the same patients. Only 4 raters reached substantial agreement with themselves, and 75% recommended different management for ≥20% of the cases.
A recent trend to actively intervene is perhaps reflected in this study by the 60%/40% proportions of answers in favor of intervention. A more proximal persistent occlusion was found to be the most influential factor in clinical decision-making; the presenting NIHSS score and the use of IVT were other factors that seemed to influence treatment decisions.
The uncertainty brought to light by this study should encourage our community to pursue the clinical research necessary to better define the optimal management of patients with persistent distal occlusions after MT and/or IVT. We believe that in the presence of such uncertainty, patients are best managed within the context of a care trial,17,29 in which they are given a 50% chance of receiving a promising treatment of yet-unknown benefit but an equal 50% chance of receiving more conservative alternative management.30
This survey also provides information regarding the feasibility of such a trial. Most clinicians (27 of 31 or 87%) claimed that they would recommend RCT participation to at least 50% of patients. In order to be eligible for the trial, both rescue therapy and conservative management should be considered reasonable treatment options, which was the case for nearly all patients. The survey also provides some indications regarding trial design. For example, patients with all sorts of persistent occlusions should be included because the uncertainty concerned all sites. Moreover, given the lack of agreement regarding the wide variety of interventional management strategies that were chosen by raters, we believe that the trial should not be limited to a specific device. The main question that needs to be addressed is whether rescue therapy (any selection or combination of SR, CA, and IAT) truly offers a safe and more effective alternative than conservative management.
There are several limitations to this study. The number of cases was relatively small (n = 60), to encourage participation of multiple raters. Creating a portfolio of artificially balanced cases was important to minimize κ paradoxes,15 but a different selection might have produced different results. Raters were not a random sample representative of a population of clinicians, a requirement that is not necessary when reliability cannot be shown within or between observers in a sufficiently wide range of clinicians.18 The survey did not include patients with a primary distal occlusion (without proximal occlusion). Thus, our study does not provide any insight into the management of such patients. All raters were informed that the cases included in this survey had at least 1 persistent distal occlusion. Therefore, our study did not evaluate the raters’ ability to detect the presence or absence of one or multiple distal occlusions. Finally, completing an electronic survey is different from caring for real patients. The degree to which responders imagined they were dealing with serious clinical decisions can only be surmised.
CONCLUSIONS
There is a lack of consensus among clinicians regarding the optimal management approach for patients with persistent distal occlusion following MT with or without IVT. This level of uncertainty justifies the need for a randomized trial to compare rescue therapy and conservative management in this context.
Footnotes
Disclosure forms provided by the authors are available with the full text and PDF of this article at www.ajnr.org.
References
- Received August 22, 2023.
- Accepted after revision October 24, 2023.
- © 2024 by American Journal of Neuroradiology
Jump to section
Related Articles
Cited By...
- No citing articles found.