Convolutional Neural Network to Stratify the Malignancy Risk of Thyroid Nodules: Diagnostic Performance Compared with the American College of Radiology Thyroid Imaging Reporting and Data System Implemented by Experienced Radiologists

References

1. 1.↵
   2. Haugen BR,
   3. Alexander EK,
   4. Bible KC, et al

   . 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid 2016;26:1–133 doi:10.1089/thy.2015.0020 pmid:26462967

   CrossRefPubMed
2. 2.↵
   2. Kwak JY,
   3. Han KH,
   4. Yoon JH, et al

   . Thyroid imaging reporting and data system for US features of nodules: a step in establishing better stratification of cancer risk. Radiology 2011;260:892–99 doi:10.1148/radiol.11110206 pmid:21771959

   CrossRefPubMedWeb of Science
3. 3.↵
   2. Choi SH,
   3. Kim EK,
   4. Kwak JY, et al

   . Interobserver and intraobserver variations in ultrasound assessment of thyroid nodules. Thyroid 2010;20:167–72 doi:10.1089/thy.2008.0354 pmid:19725777

   CrossRefPubMedWeb of Science
4. 4.↵
   2. Park SH,
   3. Kim SJ,
   4. Kim EK, et al

   . Interobserver agreement in assessing the sonographic and elastographic features of malignant thyroid nodules. AJR Am J Roentgenol 2009;193:W416–23 doi:10.2214/AJR.09.2541 pmid:19843721

   CrossRefPubMedWeb of Science
5. 5.↵
   2. Chang Y,
   3. Paul AK,
   4. Kim N, et al

   . Computer-aided diagnosis for classifying benign versus malignant thyroid nodules based on ultrasound images: a comparison with radiologist-based assessments. Med Phys 2016;43:554 doi:10.1118/1.4939060 pmid:26745948

   CrossRefPubMed
6. 6.↵
   2. Choi YJ,
   3. Baek JH,
   4. Park HS, et al

   . A computer-aided diagnosis system using artificial intelligence for the diagnosis and characterization of thyroid nodules on ultrasound: initial clinical assessment. Thyroid 2017;27:546–52 doi:10.1089/thy.2016.0372 pmid:28071987

   CrossRefPubMed
7. 7.↵
   2. Acharya UR,
   3. Swapna G,
   4. Sree SV, et al

   . A review on ultrasound-based thyroid cancer tissue characterization and automated classification. Technol Cancer Res Treat 2014;13:289–301 doi:10.7785/tcrt.2012.500381 pmid:24206204

   CrossRefPubMed
8. 8.↵
   2. Gopinath B,
   3. Shanthi N

   . Computer-aided diagnosis system for classifying benign and malignant thyroid nodules in multi-stained FNAB cytological images. Australas Phys Eng Sci Med 2013;36:219–30 doi:10.1007/s13246-013-0199-8 pmid:23690210

   CrossRefPubMed
9. 9.↵
   2. Shin HC,
   3. Roth HR,
   4. Gao M, et al

   . Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging 2016;35:1285–98 doi:10.1109/TMI.2016.2528162 pmid:26886976

   CrossRefPubMed
10. 10.↵
    2. Ko SY,
    3. Lee JH,
    4. Yoon JH, et al

    . Deep convolutional neural network for the diagnosis of thyroid nodules on ultrasound. Head Neck 2019;41:885–91 doi:10.1002/hed.25415 pmid:30715773

    CrossRefPubMed
11. 11.↵
    2. Li X,
    3. Zhang S,
    4. Zhang Q, et al

    . Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. Lancet Oncol 2019;20:193–201 doi:10.1016/S1470-2045(18)30762-9 pmid:30583848

    CrossRefPubMed
12. 12.↵
    2. Kim HL,
    3. Ha EJ,
    4. Han M

    . Real-world performance of computer-aided diagnosis system for thyroid nodules using ultrasonography. Ultrasound Med Biol 2019;45:2672–78 doi:10.1016/j.ultrasmedbio.2019.05.032 pmid:31262524

    CrossRefPubMed
13. 13.↵
    2. Ma J,
    3. Wu F,
    4. Zhu J, et al

    . A pre-trained convolutional neural network based method for thyroid nodule diagnosis. Ultrasonics 2017;73:221–30 doi:10.1016/j.ultras.2016.09.011 pmid:27668999

    CrossRefPubMed
14. 14.↵
    2. Ma J,
    3. Wu F,
    4. Jiang T, et al

    . Cascade convolutional neural networks for automatic detection of thyroid nodules in ultrasound images. Med Phys 2017;44:1678–91 doi:10.1002/mp.12134 pmid:28186630

    CrossRefPubMed
15. 15.↵
    2. Buda M,
    3. Wildman-Tobriner B,
    4. Hoang JK, et al

    . Management of thyroid nodules seen on US images: deep learning may match performance of radiologists. Radiology 2019;292:695–701 doi:10.1148/radiol.2019181343 pmid:31287391

    CrossRefPubMed
16. 16.↵
    2. Koh J,
    3. Lee E,
    4. Han K, et al

    . Diagnosis of thyroid nodules on ultrasonography by a deep convolutional neural network. Sci Rep 2020;10:15245 doi:10.1038/s41598-020-72270-6 pmid:30583848

    CrossRefPubMed
17. 17.↵
    2. Jin Z,
    3. Zhu Y,
    4. Zhang S, et al

    . Ultrasound computer-aided diagnosis (CAD) based on the thyroid imaging reporting and data system (TI-RADS) to distinguish benign from malignant thyroid nodules and the diagnostic performance of radiologists with different diagnostic experience. Med Sci Monit 2020;26:e918452 doi:10.12659/MSM.918452 pmid:31929498

    CrossRefPubMed
18. 18.↵
    2. Park S,
    3. Oh CM,
    4. Cho H, et al

    . Association between screening and the thyroid cancer “epidemic” in South Korea: evidence from a nationwide study. BMJ 2016;355:i5745 doi:10.1136/bmj.i5745 pmid:27903497

    Abstract/FREE Full Text
19. 19.↵
    2. Jegerlehner S,
    3. Bulliard JL,
    4. Aujesky D, et al

    . NICER Working Group. Overdiagnosis and overtreatment of thyroid cancer: a population-based temporal trend study. PLoS One 2017;12:e0179387 doi:10.1371/journal.pone.0179387 pmid:28614405

    CrossRefPubMed
20. 20.↵
    2. Wu XL,
    3. Du JR,
    4. Wang H, et al

    . Comparison and preliminary discussion of the reasons for the differences in diagnostic performance and unnecessary FNA biopsies between the ACR TIRADS and 2015 ATA guidelines. Endocrine 2019;65:121–31 doi:10.1007/s12020-019-01886-0 pmid:30830584

    CrossRefPubMed
21. 21.↵
    2. Yoon JH,
    3. Lee HS,
    4. Kim EK, et al

    . Pattern-based vs. score-based guidelines using ultrasound features have different strengths in risk stratification of thyroid nodules. Eur Radiol 2020;30:3793–3802 doi:10.1007/s00330-020-06722-y pmid:32088739

    CrossRefPubMed
22. 22.↵
    2. Tappouni RR,
    3. Itri JN,
    4. McQueen TS, et al

    . ACR TI-RADS: pitfalls, solutions, and future directions. Radiographics 2019;39:2040–52 doi:10.1148/rg.2019190026 pmid:31603734

    CrossRefPubMed
23. 23.↵
    2. Tessler FN,
    3. Middleton WD,
    4. Grant EG, et al

    . ACR thyroid imaging, reporting and data system (TI-RADS): White Paper of the ACR TI-RADS Committee. J Am Coll Radiol 2017;14:587–95 doi:10.1016/j.jacr.2017.01.046 pmid:28372962

    CrossRefPubMed
24. 24.↵
    2. Kim EK,
    3. Park CS,
    4. Chung WY, et al

    . New sonographic criteria for recommending fine-needle aspiration biopsy of nonpalpable solid nodules of the thyroid. AJR Am J Roentgenol 2002;178:687–91 doi:10.2214/ajr.178.3.1780687 pmid:11856699

    CrossRefPubMedWeb of Science
25. 25.↵
    2. Lee E,
    3. Ha H,
    4. Kim HJ, et al

    . Differentiation of thyroid nodules on US using features learned and extracted from various convolutional neural networks. Sci Rep 2019;9:19854 doi:10.1038/s41598-019-56395-x pmid:31882683

    CrossRefPubMed
26. 26.↵
    2. Middleton WD,
    3. Teefey SA,
    4. Reading CC, et al

    . Comparison of performance characteristics of American College of Radiology TI-RADS, Korean Society of Thyroid Radiology TIRADS, and American Thyroid Association Guidelines. AJR Am J Roentgenol 2018;210:1148–54 doi:10.2214/AJR.17.18822 pmid:29629797

    CrossRefPubMed
27. 27.↵
    2. Youden WJ

    . Index for rating diagnostic tests. Cancer 1950;3:32–35 doi:10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3 pmid:15405679

    CrossRefPubMedWeb of Science
28. 28.↵
    2. DeLong ER,
    3. DeLong DM,
    4. Clarke-Pearson DL

    . Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988;44:837–45 pmid:3203132

    CrossRefPubMedWeb of Science
29. 29.↵
    2. Jeong EY,
    3. Kim HL,
    4. Ha EJ, et al

    . Computer-aided diagnosis system for thyroid nodules on ultrasonography: diagnostic performance and reproducibility based on the experience level of operators. Eur Radiol 2019;29:1978–85 doi:10.1007/s00330-018-5772-9 pmid:30350161

    CrossRefPubMed
30. 30.↵
    2. LeCun Y,
    3. Bengio Y,
    4. Hinton G

    . Deep learning. Nature 2015;521:436–44 doi:10.1038/nature14539 pmid:26017442

    CrossRefPubMed
31. 31.↵
    2. Park SH,
    3. Han K

    . Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 2018;286:800–09 doi:10.1148/radiol.2017171920 pmid:29309734

    CrossRefPubMed
32. 32.↵
    2. Ruan JL,
    3. Yang HY,
    4. Liu RB, et al

    . Fine needle aspiration biopsy indications for thyroid nodules: compare a point-based risk stratification system with a pattern-based risk stratification system. Eur Radiol 2019;29:4871–78 doi:10.1007/s00330-018-5992-z pmid:30715590

    CrossRefPubMed
33. 33.↵
    2. Ha EJ,
    3. Na DG,
    4. Baek JH, et al

    . US fine-needle aspiration biopsy for thyroid malignancy: diagnostic performance of seven society guidelines applied to 2000 thyroid nodules. Radiology 2018;287:893–900 doi:10.1148/radiol.2018171074 pmid:29465333

    CrossRefPubMed
34. 34.↵
    2. Ha EJ,
    3. Baek JH,
    4. Na DG

    . Deep convolutional neural network models for the diagnosis of thyroid cancer. Lancet Oncol 2019;20:e130 e130 doi:10.1016/S1470-2045(19)30086-5 pmid:30842054

    CrossRefPubMed
35. 35.↵
    2. Li X,
    3. Zhang S,
    4. Zhang Q, et al

    . Deep convolutional neural network models for the diagnosis of thyroid cancer: authors’ reply. Lancet Oncol 2019;20:e131 doi:10.1016/S1470-2045(19)30083-X pmid:30842055

    CrossRefPubMed
36. 36.↵
    2. Wildman-Tobriner B,
    3. Buda M,
    4. Hoang JK, et al

    . Using artificial intelligence to revise ACR TI-RADS risk stratification of thyroid nodules: diagnostic accuracy and utility. Radiology 2019;292:112–19 doi:10.1148/radiol.2019182128 pmid:31112088

    CrossRefPubMed
37. 37.↵
    2. Akkus Z,
    3. Cai J,
    4. Boonrod A, et al

    . A survey of deep-learning applications in ultrasound: artificial intelligence-powered ultrasound for improving clinical workflow. J Am Coll Radiol 2019;16:1318–28 doi:10.1016/j.jacr.2019.06.004 pmid:31492410

    CrossRefPubMed
38. 38.↵
    2. Cibas ES,
    3. Ali SZ

    . The 2017 Bethesda System for reporting thyroid cytopathology. Thyroid 2017;27:1341–46 doi:10.1089/thy.2017.0500 pmid:29091573

    CrossRefPubMed