Critical Appraisal of Artificial Intelligence–Enabled Imaging Tools Using the Levels of Evidence System

References

1. 1.↵
   1. Zaharchuk G,
   2. Gong E,
   3. Wintermark M, et al

   . Deep learning in neuroradiology. AJNR Am J Neuroradiol 2018;39:1776–84 doi:10.3174/ajnr.A5543 pmid:29419402

   CrossRefPubMed
2. 2.↵
   1. Lui YW,
   2. Chang PD,
   3. Zaharchuk G, et al

   . Artificial intelligence in neuroradiology: current status and future directions. AJNR Am J Neuroradiol 2020;41:E52–59 doi:10.3174/ajnr.A6681 pmid:32732276

   CrossRefPubMed
3. 3.↵
   1. Bohr A,
   2. Memarzadeh K

   . The rise of artificial intelligence in healthcare applications. Artificial Intelligence in Healthcare. Cambridge, Massachusetts; Academic Press; 2020:25–60
4. 4.↵
   1. van Leeuwen KG,
   2. de Rooij M,
   3. Schalekamp S, et al

   . How does artificial intelligence in radiology improve efficiency and health outcomes? Pediatr Radiol 2022;52:2087–93 doi:10.1007/s00247-021-05114-8 pmid:34117522

   CrossRefPubMed
5. 5.↵
   Artificial intelligence and machine learning (AI/ML)-enabled medical devices. October 5, 2022. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices. Accessed March 5, 2023
6. 6.↵
   1. van Leeuwen KG,
   2. Schalekamp S,
   3. Rutten MJ, et al

   . Artificial intelligence in radiology: 100 commercially available products and their scientific evidence. Eur Radiol 2021;31:3 797–804 doi:10.1007/s00330-021-07892-z pmid:33856519

   CrossRefPubMed
7. 7.↵
   1. Goergen SK,
   2. Frazer HM,
   3. Reddy S

   . Quality use of artificial intelligence in medical imaging: what do radiologists need to know? J Med Imaging Radiat Oncol 2022;66:225–32 doi:10.1111/1754-9485.13379 pmid:35243782

   CrossRefPubMed
8. 8.↵
   1. Letourneau-Guillon L,
   2. Camirand D,
   3. Guilbert F, et al

   . Artificial intelligence applications for workflow, process optimization and predictive analytics. Neuroimaging Clin N Am 2020;30:e1–15 doi:10.1016/j.nic.2020.08.008 pmid:33039002

   CrossRefPubMed
9. 9.↵
   1. Kitamura FC,
   2. Pan I,
   3. Ferraciolli SF, et al

   . Clinical artificial intelligence applications in radiology: Neuro. Radiol Clin North Am 2021;59:1003–12 doi:10.1016/j.rcl.2021.07.002 pmid:34689869

   CrossRefPubMed
10. 10.↵
    1. Kaka H,
    2. Zhang E,
    3. Khan N

    . Artificial intelligence and deep learning in neuroradiology: exploring the new frontier. Can Assoc Radiol J 2021;72:35–44 doi:10.1177/0846537120954293 pmid:32946272

    CrossRefPubMed
11. 11.↵
    1. Scott I,
    2. Carter S,
    3. Coiera E

    . Clinician checklist for assessing suitability of machine learning applications in healthcare. BMJ Health Care Inform 2021;28:e100251 doi:10.1136/bmjhci-2020-100251 pmid:33547086

    CrossRefPubMed
12. 12.↵
    1. Burns PB,
    2. Rohrich RJ,
    3. Chung KC

    . The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg 2011;128:305–10 doi:10.1097/PRS.0b013e318219c171 pmid:21701348

    CrossRefPubMed
13. 13.↵
    1. Willemink MJ,
    2. Koszek WA,
    3. Hardell C, et al

    . Preparing medical imaging data for machine learning. Radiology 2020;295:4–15 doi:10.1148/radiol.2020192224 pmid:32068507

    CrossRefPubMed
14. 14.↵
    1. Mongan J,
    2. Moy L,
    3. Kahn CE

    . Checklist for artificial intelligence in medical imaging (CLAIM): a guide for authors and reviewers. Radiol Artif Intell 2020;2:e200029 doi:10.1148/ryai.2020200029 pmid:33937821

    CrossRefPubMed
15. 15.↵
    1. Bluemke DA,
    2. Moy L,
    3. Bredella MA, et al

    . Assessing radiology research on artificial intelligence: a brief guide for authors, reviewers, and readers-from the radiology editorial board. Radiology 2020;294:487–89 doi:10.1148/radiol.2019192515 pmid:31891322

    CrossRefPubMed
16. 16.↵
    1. Yu AC,
    2. Eng J

    . One algorithm may not fit all: how selection bias affects machine learning performance. Radiographics 2020;40:1932–39 doi:10.1148/rg.2020200040 pmid:32976062

    CrossRefPubMed
17. 17.↵
    1. Hernandez-Boussard T,
    2. Bozkurt S,
    3. Ioannidis JP, et al

    . MINIMAR (MINimum information for medical AI reporting): developing reporting standards for artificial intelligence in health care. J Am Med Inform Assoc 2020;27:2011–15 doi:10.1093/jamia/ocaa088 pmid:32594179

    CrossRefPubMed
18. 18.↵
    Radiology: Artificial intelligence. Checklist for Artificial Intelligence in Medical Imaging (CLAIM). https://pubs.rsna.org/page/ai/claim?doi=10.1148%2Fryai&publicationCode=ai. Accessed March 2, 2023
19. 19.↵
    1. Liu X,
    2. Rivera SC,
    3. Moher D, et al

    . Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. BMJ (Online) 2020;370:m3164 doi:10.1136/bmj.m3164

    CrossRef
20. 20.↵
    1. Park Y,
    2. Jackson GP,
    3. Foreman MA, et al

    . Evaluating artificial intelligence in medicine: phases of clinical research. JAMIA Open 2020;3:326–31 doi:10.1093/jamiaopen/ooaa033 pmid:33215066

    CrossRefPubMed
21. 21.↵
    1. Jha A,
    2. Bradshaw T,
    3. Buvat I, et al

    . Best practices for evaluation of artificial intelligence-based algorithms for nuclear medicine: the RELIANCE guidelines. J Nucl Med 2022;63(Suppl 2):1725
22. 22.↵
    1. Filice RW,
    2. Mongan J,
    3. Kohli MD

    . Evaluating artificial intelligence systems to guide purchasing decisions. J Am Coll Radiol 2020;17:1405–09 doi:10.1016/j.jacr.2020.09.045 pmid:33035503

    CrossRefPubMed
23. 23.↵
    1. Krause J,
    2. Gulshan V,
    3. Rahimy E, et al

    . Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 2018;125:1264–72 doi:10.1016/j.ophtha.2018.01.034 pmid:29548646

    CrossRefPubMed
24. 24.↵
    1. Duggan GE,
    2. Reicher JJ,
    3. Liu Y, et al

    . Improving reference standards for validation of AI-based radiography. Br J Radiol 2021;94:20210435 doi:10.1259/bjr.20210435 pmid:34142868

    CrossRefPubMed
25. 25.↵
    1. Yu AC,
    2. Mohajer B,
    3. Eng J

    . External validation of deep learning algorithms for radiologic diagnosis: a systematic review. Radiol Artif Intell 2022;4:e210064 doi:10.1148/ryai.210064 pmid:35652114

    CrossRefPubMed
26. 26.↵
    1. Kim DW,
    2. Jang HY,
    3. Kim KW, et al

    . Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: results from recently published papers. Korean J Radiol 2019;20:405–10 doi:10.3348/kjr.2019.0025 pmid:30799571

    CrossRefPubMed
27. 27.↵
    1. Dunnmon J

    . Separating hope from hype: artificial intelligence pitfalls and challenges in radiology. Radiol Clin North Am 2021;59:1063–74 doi:10.1016/j.rcl.2021.07.006 pmid:34689874

    CrossRefPubMed
28. 28.↵
    1. Vokinger KN,
    2. Feuerriegel S,
    3. Kesselheim AS

    . Mitigating bias in machine learning for medicine. Commun Med (Lond) 2021;1:25 doi:10.1038/s43856-021-00028-w pmid:34522916

    CrossRefPubMed
29. 29.↵
    1. Kelly CJ,
    2. Karthikesalingam A,
    3. Suleyman M, et al

    . Key challenges for delivering clinical impact with artificial intelligence. BMC Med 2019;17:195 doi:10.1186/s12916-019-1426-2 pmid:31665002

    CrossRefPubMed
30. 30.↵
    1. He J,
    2. Baxter SL,
    3. Xu J, et al

    . The practical implementation of artificial intelligence technologies in medicine. Nat Med 2019;25:30–36 doi:10.1038/s41591-018-0307-0 pmid:30617336

    CrossRefPubMed
31. 31.↵
    1. Park SH,
    2. Han K

    . Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 2018;286:800–09 doi:10.1148/radiol.2017171920 pmid:29309734

    CrossRefPubMed
32. 32.↵
    1. Wolff J,
    2. Pauling J,
    3. Keck A, et al

    . Success factors of artificial intelligence implementation in healthcare. Front Digit Health 2021;3:594971 doi:10.3389/fdgth.2021.594971 pmid:34713083

    CrossRefPubMed
33. 33.↵
    1. Omoumi P,
    2. Ducarouge A,
    3. Tournier A, et al

    . To buy or not to buy: evaluating commercial AI solutions in radiology (the ECLAIR guidelines). Eur Radiol 2021;31:3786–96 doi:10.1007/s00330-020-07684-x pmid:33666696

    CrossRefPubMed
34. 34.↵
    1. DeCamp M,
    2. Lindvall C

    . Latent bias and the implementation of artificial intelligence in medicine. J Am Med Inform Asso. 2020;27:2020–23 doi:10.1093/jamia/ocaa094 pmid:32574353

    CrossRefPubMed
35. 35.↵
    1. Finlayson SG,
    2. Subbaswamy A,
    3. Singh K, et al

    . The clinician and dataset shift in artificial intelligence. N Engl J Med 2021;385:283–86 doi:10.1056/NEJMc2104626 pmid:34260843

    CrossRefPubMed
36. 36.↵
    1. Geis JR,
    2. Brady AP,
    3. Wu CC, et al

    . Ethics of artificial intelligence in radiology: summary of the Joint European and North American Multisociety Statement. Radiology 2019;293:436–40 doi:10.1148/radiol.2019191586

    CrossRefPubMed
37. 37.↵
    1. Liu X,
    2. Glocker B,
    3. McCradden MM, et al

    . The medical algorithmic audit. Lancet Digit Health .2022;4:e384–97 doi:10.1016/S2589-7500(22)00003-6 pmid:35396183

    CrossRefPubMed
38. 38.↵
    1. Vollmer S,
    2. Mateen BA,
    3. Bohner G, et al

    . Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness. BMJ 2020;368:l6927 doi:10.1136/bmj.l6927 pmid:32198138

    CrossRefPubMed
39. 39.↵
    1. Spilseth B,
    2. McKnight CD,
    3. Li MD, et al

    . AUR-RRA review: logistics of academic-industry partnerships in artificial intelligence. Acad Radiol 2022;29:119–28 doi:10.1016/j.acra.2021.08.002 pmid:34561163

    CrossRefPubMed
40. 40.↵
    FDA-regulated AI algorithms: trends, strengths, and gaps of validation studies. Acad Radiol 2022;29:559–66 doi:10.1016/j.acra.2021.09.002 pmid:34969610

    CrossRefPubMed
41. 41.↵
    1. Pianykh OS,
    2. Langs G,
    3. Dewey M, et al

    . Continuous learning AI in radiology: implementation principles and early applications. Radiology 2020;297:6–14 doi:10.1148/radiol.2020200038 pmid:32840473

    CrossRefPubMed
42. 42.↵
    1. Genereaux B,
    2. O'Donnell K,
    3. Bialecki B, et al

    . IHE radiology white paper: AI interoperability in imaging. Integrating the Healthcare Enterprise 2021;1:????? https://www.ihe.net/uploadedFiles/Documents/Radiology/IHE_RAD_White_Paper_AI_Interoperability_in_Imaging.pdf. Accessed March 3, 2023
43. 43.↵
    1. Wiggins WF,
    2. Magudia K,
    3. Schmidt TMS, et al

    . Imaging AI in practice: a demonstration of future workflow using integration standards. Radiol Artif Intell 2021;3:e210152 doi:10.1148/ryai.2021210152 pmid:34870224

    CrossRefPubMed
44. 44.↵
    1. Kirchner JE,
    2. Smith JL,
    3. Powell BJ, et al

    . Getting a clinical innovation into practice: an introduction to implementation strategies. Psychiatry Res 2020;283:112467 doi:10.1016/j.psychres.2019.06.042 pmid:31488332

    CrossRefPubMed
45. 45.↵
    1. Porter ME

    . What is value in health care? N Engl J Med 2010;363:2477–81 doi:10.1056/NEJMp1011024

    CrossRefPubMed
46. 46.↵
    1. Brady AP,
    2. Visser J,
    3. Frija G, et al

    . Value-based radiology: what is the ESR doing, and what should we do in the future? Insights Imaging 2021;12:108 doi:10.1186/s13244-021-01056-9

    CrossRefPubMed
47. 47.↵
    1. Teisberg E,
    2. Wallace S,
    3. O’Hara S

    . Defining and implementing value-based health care: a strategic framework. Acad Med 2020;95:682–85 doi:10.1097/ACM.0000000000003122 pmid:31833857

    CrossRefPubMed
48. 48.↵
    1. Plana D,
    2. Shung DL,
    3. Grimshaw AA, et al

    . Randomized clinical trials of machine learning interventions in health care: a systematic review. JAMA Netw Open 2022;5:e2233946 doi:10.1001/jamanetworkopen.2022.33946 pmid:36173632

    CrossRefPubMed
49. 49.↵
    1. Wilkinson J,
    2. Arnold KF,
    3. Murray EJ, et al

    . Time to reality check the promises of machine learning-powered precision medicine. Lancet Digit Health 2020;2:e677–80 doi:10.1016/S2589-7500(20)30200-4 pmid:33328030

    CrossRefPubMed
50. 50.↵
    1. Campbell BC,
    2. Mitchell PJ,
    3. Kleinig TJ, et al

    ; EXTEND-IA Investigators. Endovascular therapy for ischemic stroke with perfusion-imaging selection. N Engl J Med 2015;372:1009–18 doi:10.1056/NEJMoa1414792 pmid:25671797

    CrossRefPubMed
51. 51.↵
    1. Soun JE,
    2. Chow DS,
    3. Nagamine M, et al

    . Artificial intelligence and acute stroke imaging. AJNR Am J Neuroradiol 2021;42:2–11 doi:10.3174/ajnr.A6883 pmid:33243898

    CrossRefPubMed
52. 52.↵
    1. Albers GW,
    2. Marks MP,
    3. Kemp S, et al

    ; DEFUSE 3 Investigators. Thrombectomy for stroke at 6 to 16 hours with selection by perfusion imaging. N Engl J Med 2018;378:708–18 doi:10.1056/NEJMoa1713973 pmid:29364767

    CrossRefPubMed
53. 53.↵
    1. Ma H,
    2. Campbell BC,
    3. Parsons MW, et al

    ; EXTEND Investigators. Thrombolysis guided by perfusion imaging up to 9 hours after onset of stroke. N Engl J Med 2019;380:1795–803 doi:10.1056/NEJMoa1813046 pmid:31067369

    CrossRefPubMed
54. 54.↵
    1. van Leeuwen KG,
    2. Meijer FJ,
    3. Schalekamp S, et al

    . Cost-effectiveness of artificial intelligence aided vessel occlusion detection in acute stroke: an early health technology assessment. Insights Imaging 2021;12:133 doi:10.1186/s13244-021-01077-4 pmid:34564764

    CrossRefPubMed
55. 55.
    1. Heit JJ,
    2. Honce JM,
    3. Yedavalli VS, et al

    . RAPID aneurysm: artificial intelligence for unruptured cerebral aneurysm detection on CT angiography. J Stroke Cerebrovasc Dis 2022;31:106690 doi:10.1016/j.jstrokecerebrovasdis.2022.106690 pmid:35933764

    CrossRefPubMed
56. 56.
    1. Sahlein DH,
    2. Gibson D,
    3. Scott JA, et al

    . Artificial intelligence aneurysm measurement tool finds growth in all aneurysms that ruptured during conservative management. J Neurointerv Surg 2022 Sep 30. [Epub ahead of print] doi:10.1136/jnis-2022-019339 pmid:36180207

    CrossRefPubMed