Abstract
BACKGROUND AND PURPOSE: Intracranial hemorrhage (ICH) is an important event that is diagnosed on head NCCT. Increased NCCT utilization in busy hospitals may limit timely identification of ICH. RAPID ICH is an automated hybrid 2D–3D convolutional neural network application designed to detect ICH that may allow for expedited ICH diagnosis. We determined the accuracy of RAPID ICH for ICH detection and ICH volumetric quantification on NCCT.
MATERIALS AND METHODS: NCCT scans were evaluated for ICH by RAPID ICH. Consensus detection of ICH by 3 neuroradiology experts was used as the criterion standard for RAPID ICH comparison. ICH volume was also automatically determined by RAPID ICH in patients with intraparenchymal or intraventricular hemorrhage and compared with manually segmented ICH volumes by a single neuroradiology expert. ICH detection accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and positive and negative likelihood ratios by RAPID ICH were determined.
RESULTS: We included 308 studies. RAPID ICH correctly identified 151/158 ICH cases and 143/150 ICH-negative cases, which resulted in high sensitivity (0.956, CI: 0.911–0.978), specificity (0.953, CI: 0.907–0.977), positive predictive value (0.956, CI: 0.911-0.978), and negative predictive value (0.953, CI: 0.907–0.977) for ICH detection. The positive likelihood ratio (20.479, CI 9.928–42.245) and negative likelihood ratio (0.046, CI 0.023–0.096) for ICH detection were similarly favorable. RAPID ICH volumetric quantification for intraparenchymal and intraventricular hemorrhages strongly correlated with expert manual segmentation (correlation coefficient r = 0.983); the median absolute error was 3 mL.
CONCLUSIONS: RAPID ICH is highly accurate in the detection of ICH and in the volumetric quantification of intraparenchymal and intraventricular hemorrhages.
ABBREVIATIONS:
- CNN
- convolutional neural network
- ICH
- intracranial hemorrhage
- LR
- likelihood ratio
- NPV
- negative predictive value
- PPV
- positive predictive value
Intracranial hemorrhage (ICH) secondary to trauma, cerebrovascular disease, tumors, coagulation disorders, and other disorders results in significant morbidity and mortality.1⇓-3 The volume and severity of ICH at presentation correlate with neurologic status and likelihood of survival,4,5 and prompt medical and surgical intervention have been shown to reduce the mortality rate associated with ICH.6,7 Therefore, accurate and timely ICH diagnosis is essential for patient treatment, and prompt detection and interpretation of ICH on NCCT are necessary. Excluding acute ICH is also a critical component of the evaluation of patients with stroke for IV thrombolysis.
The presentation of ICH is often nonspecific, and ICH is most commonly diagnosed on head NCCT.2 Prompt NCCT interpretation in busy emergency departments and hospitals remains challenging, and interpretation delays may result in patient care delays, which can lead to poor outcomes.8 Triage software that identifies ICH, estimates ICH volume, and alerts radiologists and clinicians would streamline patient care and increase diagnostic confidence. Deep learning convolutional neural networks (CNNs) represent a technology for automated imaging interpretation that has shown promise in the detection of ICH and other cerebral emergencies.9⇓⇓⇓⇓-14 However, this technology has not yet been adopted in widespread clinical practice.
RAPID ICH (iSchemaView) is an artificial intelligence software program developed to identify acute ICH and determine ICH on NCCT studies. RAPID ICH builds on the widely used RAPID software platform for the detection of cerebral ischemia in patients with large-vessel occlusions15⇓⇓-18 and is readily adaptable to the RAPID mobile triage platform, which is currently used for triage of patients with ischemic stroke to thrombectomy.
In this study, we tested whether RAPID ICH can accurately detect and volumetrically quantify ICH.
MATERIALS AND METHODS
The data that support the findings of this study are available from the corresponding author upon reasonable request.
Patient Cohort and Image Acquisition
This retrospective cohort study complied with the Health Insurance Portability and Accountability Act. Institutional review board approval was obtained at each site, and the need for informed consent was waived. NCCTs were obtained from 6 institutions. Only NCCTs that were free of significant motion artifact were included for analysis.
NCCT studies from multiple vendors (Online Table 1) were acquired in the axial plane with section thickness that ranged from 1 to 5 mm. Radiation doses varied by vendor and location, and these variables were not controlled for in this study, which was intended to sample variations in standard radiology practices. Images were not tilt corrected or otherwise manipulated before interpretation by the neuroradiologists in the study. Studies with significant metal or motion artifact were excluded from the analysis.
RAPID Machine Learning ICH Detection
A deep CNN with a hybrid 2D–3D architecture was trained on a cohort of 805 NCCT examinations. All head CTs analyzed by RAPID ICH were postprocessed into 5-mm-thick axial slices, and 48 images per study were analyzed by the training dataset. If CT source data were acquired with a thickness of <5 mm, images were merged and averaged into 5-mm thickness before analysis. The training dataset included NCCT with intraparenchymal hemorrhage (245 cases; 30%), intraventricular hemorrhage (83 cases; 10%), extra-axial hemorrhage (70 cases, 9%), subarachnoid hemorrhage (67 cases, 8%), and no hemorrhage (457 cases, 57%). Regions of ICH were manually outlined by neuroradiology experts and stored as binary masks that were used as ground truth for the training analysis. Ground truth masks were randomly split into training (80%) and testing (20%) groups.
After training, the ICH detection module was prototyped in Python/Keras and implemented on a Linux server (4 CPUs) that was embedded on a larger imaging platform dedicated to ischemic stroke analysis (iSchemaView).
Additional details of the CNN are presented in Online Figure.
RAPID ICH Validation Study Design
The RAPID ICH module was validated in a different and independent dataset of 308 adult patients that included 158 patients with ICH (52%) and 150 patients without ICH (48%). The reference standard or truth was determined by consensus among 3 expert neuroradiologists who reviewed NCCT images via Horos (Horos Project, version 3.3) or Osirx (Pixmeo, version 11). Neuroradiologists had full control over window width and levels for all images and were blinded to RAPID ICH results.
The primary end point was the detection of any ICH, which included intraparenchymal, subdural, epidural, subarachnoid, and intraventricular hemorrhages, and ICH presence was recorded in a binary manner (present [ICH+] or absent [ICH−]). Most subdural hemorrhages were acute, but some had subacute or chronic components. Epidural and subdural hemorrhage are collectively referred to as extra-axial hemorrhage.
The secondary end point was ICH volume for isolated intraparenchymal and intraventricular hemorrhages. The reference standard ICH volume was determined by manual segmentation by a single neuroradiologist and verification by a second neuroradiologist. Manual segmentation was performed on axial images. Regions of confluent hyperattenuation that were consistent with intraparenchymal or intraventricular hemorrhage were outlined on all images and volumetrically quantified in Osirix (Pixmeo, version 11).
Statistical Analysis
Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (LR), and negative LR were calculated by comparison of RAPID identification of ICH+ studies compared with the reference standard ICH+. All statistical analyses were performed using JMP Pro 15 (SAS Institute).
RESULTS
A total of 308 NCCTs were included in the validation study. NCCT was acquired from 5 CT vendors (Online Table 1), distributed across the ICH- and ICH+ studies. After training, the processing time for RAPID’s ICH detection module was <3 minutes per study.
NCCT was derived from 117 female patients (38.0%) and 164 male patients (53.3%), and sex was unknown in 27 patients (8.8%). Most patients were 60 years or older (63.3%), but young patients (ages 20–39 years) were well represented (33 patients, 10.9%). These demographic data are summarized in Table 1.
Patient demographic details
ICH was detected (ICH+) in 158 NCCT studies (51.2%) by neuroradiology experts, and the distribution of ICH is denoted in Online Table 2. Intraparenchymal hemorrhage was part of most ICH+ NCCT studies (79 cases, 50%), and extra-axial hemorrhage (39 cases, 25%), subarachnoid hemorrhage (25 cases, 16%), intraventricular hemorrhage (10 cases, 6%), and other hemorrhage (5 cases, 3%) were less common.
We compared ICH detection by RAPID with the consensus of 3 expert neuroradiologists (Table 2). RAPID correctly identified 151/158 ICH+ cases and 143/150 ICH–cases (Figs 1 and 2). Therefore, RAPID had a high sensitivity (0.956, CI: 0.911–0.978), specificity (0.953, CI: 0.907–0.977), PPV (0.956, CI: 0.911–0.978), and NPV (0.953, CI: 0.907–0.977) for ICH detection (Table 3). The positive LR (20.479, CI 9.928–42.245) and negative LR (0.046, CI 0.023–0.096) for ICH detection were similarly favorable. NCCT in which RAPID did not detect hemorrhage (false-negative cases) involved cases with small volumes of ICH (<1.5 mL in all instances) that were intraparenchymal, intraventricular, and subdural in location (Fig 2).
Performance of RAPID for ICH detection
RAPID ICH performance
Representative imaging examples of ICH correctly detected by RAPID. A. Primary intraparenchymal hemorrhage within the right thalamus and posterior limb of the right internal capsule (red outline by RAPID). B. Small extra-axial subdural hematoma overlying the right cerebral hemisphere (red outline by RAPID). C. Subarachnoid hemorrhage (red outline by RAPID). D. Subarachnoid and intraventricular (right and left lateral, third, and fourth ventricles) hemorrhage (red outline by RAPID).
Representative false-positive and false-negative ICH examples. A. RAPID incorrectly detected ICH (false-positive result) in a patient with volume averaging in the anterior cranial fossa and beam-hardening artifact in the posterior fossa (red outlines by RAPID). B. RAPID failed to detect ICH (false-negative result) in a patient intraparenchymal hemorrhage in the left thalamus (yellow arrow). C. RAPID failed to detect ICH (false-negative result) in a patient with a small amount of subarachnoid and intraventricular hemorrhage (yellow arrows).
We then determined the accuracy of RAPID for ICH volumetric quantification. ICH volume was automatically quantified from the segmentation using Osirix. RAPID quantification of ICH volume demonstrated a strong correlation with the neuroradiology experts (correlation coefficient r = 0.983, Fig 3). The mean volumes of ICH in this analysis were 12 mL (RAPID) and 8 mL (expert), and the median absolute error was 3 mL.
Intraparenchymal and intraventricular hemorrhage volumetric agreement between RAPID and expert interpretation. A and B, Representative example of an intraparenchymal hemorrhage in the right basal ganglia. Manual segmentation by a neuroradiologist (A, green outline) is well correlated with automated segmentation by RAPID ICH (B, red outline). C. Scatterplot denotes volumetric agreement between RAPID and expert evaluation for NCCT with isolated intraparenchymal or intraventricular hemorrhage.
DISCUSSION
In this study, we found that the automated artificial intelligence RAPID ICH module is highly accurate for the detection of ICH on NCCT. Moreover, RAPID ICH volumetric quantification of ICH for intraparenchymal and intraventricular hemorrhage was highly accurate. These findings have important implications for the more widespread adoption of artificial intelligence ICH detection into clinical practice.
Other studies have used CNNs to detect the presence of ICH on NCCT.9⇓-11,14 The high sensitivity (96%) and specificity (95%) of RAPID ICH compare favorably with these prior studies that found a sensitivity of 70%–98% and a specificity of 87%–95% for their individual CNN.9⇓-11,14 All 7 false-negative RAPID ICH evaluations consisted of small ICHs that measured <1.5 mL in volume, and 6 of the 7 false-positive cases were attributable to volume averaging. The remaining false-positive study was read as a dural fold by an expert reader. Future iterations of this platform might improve the detection of these small volumes of ICH and the exclusion of false-positive ICHs with artifact reduction approaches. The use of studies from multiple centers and the use of neuroradiology expert interpretations as the criterion standard rather than radiology reports likely increase the generalizability and accuracy of our study compared with prior studies.
Computer-assisted detection of findings on imaging studies has been previously used for cancer detection in mammography, pulmonary nodule detection on chest CT, and colonic polyp detection,19⇓-21 but these technologies are not based on CNNs and have limited utility in routine clinical practice. By contrast, CNN technology has been successfully applied to nonradiology studies, such as the detection of diabetic retinopathy22 and skin cancer detection,23 which has prompted efforts to move this technology into routine medical practice.
The progressive and substantial increase in the amount of diagnostic imaging studies24,25 places particular stress on the timely interpretation of NCCT. Delays in interpretation may lead to delayed identification of ICH, resulting in nonexpeditious patient care that could lead to poor patient outcomes.8 We speculate that the adoption of CNN technology for ICH detection into routine clinical practice will speed NCCT interpretations, which must still be performed by a board-certified radiologist, and patient treatment decisions. Future studies should be designed to test how this technology changes radiology workflows and patient care.
The use of automated image processing in acute ischemic stroke, which is another cerebrovascular emergency that relies on prompt diagnosis and treatment, is already in widespread clinical practice after several randomized studies demonstrated the effectiveness of this approach in selecting patients for endovascular thrombectomy treatment.15⇓⇓-18 We expect that the adoption of ICH detection in a manner similar to cerebral ischemia detection will lead to timely detection, resulting in improved care of patients with hemorrhagic stroke. In addition, the detection of ICH may also impact the treatment of patients with concomitant ischemic stroke because the presence of ICH is a contraindication to treatment with IV thrombolysis and, in some instances, endovascular thrombectomy.
Further studies are required to determine how automated ICH volume quantification may be best used in clinical practice. Other studies have found similar accuracy for CNN methods of ICH volume measurement9,13 as in our study, which suggests that application of these methods to the care of patients with intraparenchymal, intraventricular, and even subarachnoid hemorrhage may have a role in patient treatment and prognostication.
Our study has several limitations. The retrospective design may introduce bias, and the inclusion of studies from a limited number of CT vendors and locations may limit the generalizability of our findings. Future prospective studies that include a larger number of sites are required for further validation of our findings. It is also possible that beam-hardening artifact, particularly within the posterior fossa, may further limit the sensitivity of RAPID ICH. We also note that although RAPID ICH was highly accurate in this study, automated ICH detection should not preclude interpretation by a trained radiologist because even small undetected hemorrhages might impact outcomes in patients, such as those with ischemic strokes who are being considered for IV thrombolysis.
CONCLUSIONS
RAPID ICH is highly accurate in the detection of ICH and in the volumetric quantification of intraparenchymal and intraventricular hemorrhages. The overall robustness of this CNN approach suggests that automated ICH detection is sufficiently developed for introduction into routine clinical practice.
Footnotes
Disclosures: Jeremy Heit—RELATED: Consulting Fee or Honorarium: iSchemaView, Medtronic, Microvention, Comments: iSchemaView Medical and Scientific Advisory Board, Medtronic-minimal consulting fees, MicroVention-modest consulting fees. Amin Aghaebrahim—UNRELATED: Consultancy: IschemaView advisor. Ricardo Hanel—UNRELATED: Board Membership: Elum, Three Rivers Medical; Consultancy: Medtronic, Stryker, Microvention, Balt, Phenox, Cerenovus; Stock/Stock Options: Corindus. Keith Kwok—RELATED: Consulting Fee or Honorarium: Consultant for iSchemaView. Hafez Haerian—RELATED: Consulting Fee or Honorarium: iSchemaView. Carlo Cereda—UNRELATED: Consultancy: Member of the iSchemaView Scientific Board, Comments: Minor.* Chitra Venkatasubramanian—UNRELATED: Expert Testimony: Law firms; Grants/Grants Pending: NIH and industry grants as part of multi center trials; Stock/Stock Options: Ceribell. Seena Dehkharghani—UNRELATED: Grants/Grants Pending: Grant support from anonymous donor paid to institution, Comments: Grant on development of a device for collecting and analyzing brain electrical properties*; Patents (Planned, Pending or Issued): Patent pending for device to collect and analyze brain dielectric properties; Travel/Accommodations/Meeting Expenses Unrelated to Activities Listed: iSchemaView, Comments: Travel (2019) and research support for unrelated scientific collaboration. Leonardo Carbonera—UNRELATED: Consultancy: Allm Inc; Payment for Lectures Including Service on Speakers Bureaus: Boehringer Ingelheim, IschemaView. Karen Copeland—RELATED: Fees for Participation in Review Activities Such as Data Monitoring Boards, Statistical Analysis, Endpoint Committees, and the Like: Boulder Statistics LLC, Comments: I (Karen Copeland, Boulder Statistics) was paid as a consultant to iSchemaView for statistical analysis and manuscript review; Payment for Writing or Reviewing the Manuscript: Boulder Statistics LLC, Comments: I (Karen Copeland, Boulder Statistics) was paid as a consultant to iSchemaView for statistical analysis and manuscript review; UNRELATED: Consultancy: Boulder Statistics, Comments: I (Karen Copeland, Boulder Statistics) am a paid consultant to iSchemaView. Francisco Mont’Alverne—UNRELATED: Consultancy: Medtronic; Payment for Lectures Including Service on Speakers Bureaus: Medtronic, Penumbra, Stryker. *Money paid to institution.
References
- Received May 29, 2020.
- Accepted after revision September 13, 2020.
- © 2021 by American Journal of Neuroradiology
