External and Internal Validation of a Computer Assisted Diagnostic Model for Detecting Multi-Organ Mass Lesions in CT images

Xu Lianyan; Yan Ke; Lu Le; Zhang Weihong; Chen Xu; Huo Xiaofei; Lu Jingjing

doi:10.24920/003968

Your Location：

Home >

Browse articles >

External and Internal Validation of a Computer Assisted Diagnostic Model for Detecting Multi-Organ Mass Lesions in CT images

Chinese Medical Sciences Journal Vol. 36, Issue 3, Pages: 210-217(2021)

ORIGINAL ARTICLE | Updated：2024-04-10

- External and Internal Validation of a Computer Assisted Diagnostic Model for Detecting Multi-Organ Mass Lesions in CT images
- Affiliations：
  
  1. 1Department of Radiology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing 100730, China
  2. 2PAII Inc., Bethesda, MD 20817, USA
  3. 3Department of Radiology, Beijing United Family Hospital, Beijing 100015, China
- Author bio：
- Funds：
- DOI：10.24920/003968
  CLC：
- Received：10 July 2021，
  
  Accepted：2021-8-30，
  
  Published：21 September 2021
- Accepted：
Scan QR Code
Xu Lianyan, Yan Ke, Lu Le, et al. External and Internal Validation of a Computer Assisted Diagnostic Model for Detecting Multi-Organ Mass Lesions in CT images[J]. Chinese medical sciences journal, 2021, 36(3): 210-217.
DOI：

Xu Lianyan, Yan Ke, Lu Le, et al. External and Internal Validation of a Computer Assisted Diagnostic Model for Detecting Multi-Organ Mass Lesions in CT images[J]. Chinese medical sciences journal, 2021, 36(3): 210-217. DOI： 10.24920/003968.

摘要

目的

我们开发了一种在实验室测试中具有较佳表现的通用病变检测模型ULDor。本研究旨在通过外部数据集和内部数据集对其检测性能进行测试并评估其在临床上的应用价值。

方法

通过卷积神经网络（convolutional neural network

CNN）构建通用病变检测模型（ULDor）。该模型经过DeepLesion数据集和其他5个特定器官的公共数据集对模型进行训练

其中DeepLesion数据集包括12

000多组CT扫描图像及其中80

000多个病变注释。验证测试集包括外部和内部验证数据集。外部验证数据集由一家综合医院回顾性收集的164组胸部（含上腹部）CT平扫检查图像组成

内部验证数据集由来自美国国家肺筛查试验（NLST）的187组胸部低剂量螺旋CT扫描图像组成。我们运行ULDor对这两个测试集的图像进行病变检测

记录并测量模型所检测出的所有肺外器官（包括肝、肾、胰腺、甲状腺、淋巴结、体壁、胸椎

等）的占位性病变;另由三名经过资格认证的放射科医生对两个测试集进行人工阅片

以此为标准对ULDor的检测结果进行验证分析

采用阳性预测值和灵敏度来评价模型的检测性能。

结果

在外部验证中

模型对所有病变的整体阳性预测值和敏感性分别为57.9%和67.0%

平均每组图像检测出2.1个病变

其中0.9个是假阳性。ULDor检出肝脏病变的能力最佳

阳性预测值为78.9%

敏感性为92.7%

其次是肾脏

阳性预测值为70.0%

敏感性为58.3%。在内部验证中

尽管图像的软组织噪声水平较高

ULDor仍实现了75.3%的阳性预测值和52.0%的灵敏度。

结论

ULDor在外部真实数据的验证显示模型在多用途计算机辅助诊断方面对于某些器官占位病变具有较好的检测效能。通过进一步优化和迭代升级

ULDor或许可以很好地推广应用到外部数据。

Abstract

Objective

We developed a universal lesion detector (ULDor) which showed good performance in in-lab experiments. The study aims to evaluate the performance and its ability to generalize in clinical setting via both external and internal validation.

Methods

The ULDor system consists of a convolutional neural network (CNN) trained on around 80K lesion annotations from about 12K CT studies in the DeepLesion dataset and 5 other public organ-specific datasets. During the validation process

the test sets include two parts: the external validation dataset which was comprised of 164 sets of non-contrasted chest and upper abdomen CT scans from a comprehensive hospital

and the internal validation dataset which was comprised of 187 sets of low-dose helical CT scans from the National Lung Screening Trial (NLST). We ran the model on the two test sets to output lesion detection. Three board-certified radiologists read the CT scans and verified the detection results of ULDor. We used positive predictive value (PPV) and sensitivity to evaluate the performance of the model in detecting space-occupying lesions at all extra-pulmonary organs visualized on CT images

including liver

kidney

pancreas

adrenal

spleen

esophagus

thyroid

lymph nodes

body wall

thoracic spine

etc

Results

In the external validation

the lesion-level PPV and sensitivity of the model were 57.9% and 67.0%

respectively. On average

the model detected 2.1 findings per set

and among them

0.9 were false positives. ULDor worked well for detecting liver lesions

with a PPV of 78.9% and a sensitivity of 92.7%

followed by kidney

with a PPV of 70.0% and a sensitivity of 58.3%. In internal validation with NLST test set

ULDor obtained a PPV of 75.3% and a sensitivity of 52.0% despite the relatively high noise level of soft tissue on images.

Conclusions

The performance tests of ULDor with the external real-world data have shown its high effectiveness in multiple-purposed detection for lesions in certain organs. With further optimisation and iterative upgrades

ULDor may be well suited for extensive application to external data.

关键词

Keywords

references

Dreyer KJ , Geis JR . When machines think: radiology’s next frontier . Radiology 2017 ; 285 ( 3 ): 713 - 8 . doi: 10.1148/radiol.2017171183 1001-9294-36-3-210/radiol.2017171183 . DOI: 10.1148/radiol.2017171183 http://doi.org/10.1148/radiol.2017171183

Cheng CT , Wang YR , Chen HW , et al . A scalable physician-level deep learning algorithm detects universal trauma on pelvic radiographs . Nat Commun 2021 ; 12 ( 1 ): 1066 . doi: 10.1038/s41467-021-21311-3 1001-9294-36-3-210/s41467-021-21311-3 . DOI: 10.1038/s41467-021-21311-3 http://doi.org/10.1038/s41467-021-21311-3 https://doi.org/10.1038/s41467-021-21311-3 https://doi.org/10.1038/s41467-021-21311-3

Korfiatis P , Kline TL , Lachance DH , et al . Residual deep convolutional neural network predicts MGMT methylation status . J Digit Imaging 2017 ; 30 ( 5 ): 622 - 8 . doi: 10.1007/s10278-017-0009-z 1001-9294-36-3-210/s10278-017-0009-z . DOI: 10.1007/s10278-017-0009-z http://doi.org/10.1007/s10278-017-0009-z https://doi.org/10.1007/s10278-017-0009-z https://doi.org/10.1007/s10278-017-0009-z

Nam JG , Kim M , Park JC , et al . Development and validation of a deep learning algorithm detecting 10 common abnormalities on chest radiographs . Eur Respirat J 2021 ; 57 ( 5 ): 2003061 . doi: 10.1183/13993003.03061-2020 1001-9294-36-3-210/13993003.03061-2020 . DOI: 10.1183/13993003.03061-2020 http://doi.org/10.1183/13993003.03061-2020 http://erj.ersjournals.com/lookup/doi/10.1183/13993003.03061-2020 http://erj.ersjournals.com/lookup/doi/10.1183/13993003.03061-2020

Rajpurkar P , Irvin J , Ball RL , et al . Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists . PLOS Med 2018 ; 15 ( 11 ): e1002686 . doi: 10.1371/journal.pmed.1002686 1001-9294-36-3-210/journal.pmed.1002686 . DOI: 10.1371/journal.pmed.1002686 http://doi.org/10.1371/journal.pmed.1002686 https://dx.plos.org/10.1371/journal.pmed.1002686 https://dx.plos.org/10.1371/journal.pmed.1002686

Monteiro M , Newcombe VFJ , Mathieu F , et al . Multiclass semantic segmentation and quantification of traumatic brain injury lesions on head CT using deep learning: an algorithm development and multicentre validation study . Lancet Digit Health 2020 ; 2 ( 6 ): e314 - e322 . doi: 10.1016/S2589-7500(20)30085-6 1001-9294-36-3-210/S2589-7500(20)30085-6 . DOI: 10.1016/S2589-7500(20)30085-6 http://doi.org/10.1016/S2589-7500(20)30085-6

Benndorf M , Burnside ES , Herda C , et al . External validation of a publicly available computer assisted diagnostic tool for mammographic mass lesions with two high prevalence research datasets . Med Physic 2015 ; 42 ( 8 ): 4987 - 96 . doi: 10.1118/1.4927260 1001-9294-36-3-210/1.4927260 . DOI: 10.1118/1.4927260 http://doi.org/10.1118/1.4927260

Si K , Xue Y , Yu XZ , et al . Fully end-to-end deep-learning-based diagnosis of pancreatic tumors . Theranostics 2021 ; 11 ( 4 ): 1982 - 90 . doi: 10.7150/thno.52508 1001-9294-36-3-210/thno.52508 . DOI: 10.7150/thno.52508 http://doi.org/10.7150/thno.52508 https://www.thno.org/v11p1982.htm https://www.thno.org/v11p1982.htm

Masood A , Sheng B , Li P , et al . Computer-assisted decision support system in pulmonary cancer detection and stage classification on CT images . J Biomed Inform 2018 ; 79 : 117 - 28 . doi: 10.1016/j.jbi.2018.01.005 1001-9294-36-3-210/j.jbi.2018.01.005 . DOI: S1532-0464(18)30007-8 http://doi.org/S1532-0464(18)30007-8

Yan K , Wang X , Lu L , et al . DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning . J Med Imaging (Bellingham) 2018 ; 5 ( 3 ): 036501 . doi: 10.1117/1.JMI.5.3.036501 1001-9294-36-3-210/1.JMI.5.3.036501 . DOI: 10.1117/1.JMI.5.3.036501 http://doi.org/10.1117/1.JMI.5.3.036501

Yan K , Cai JZ , Zheng YJ , et al . Learning from multiple datasets with heterogeneous and partial labels for universal lesion detection in CT . IEEE Trans Med Imaging 2020 ; Dec 28, 2020; E-pub ahead of print. doi: 10.1109/TMI.2020.3047598 1001-9294-36-3-210/TMI.2020.3047598 . DOI: 10.1109/TMI.2020.3047598 http://doi.org/10.1109/TMI.2020.3047598

Cai JZ , Yan K , Cheng CT , et al. Deep volumetric universal lesion detection using light-weight pseudo 3D convolution and surface point regression. In: Martel A.L. et al. (eds) Medical Image Computing and Computer Assisted Intervention - MICCAI 2020. MICCAI 2020 . Lecture Notes in Computer Science , vol 12264 . p. 3 - 13 . Springer, Cham. doi: 10.1007/978-3-030-59719-1_1 1001-9294-36-3-210/978-3-030-59719-1_1 . DOI: 10.1007/978-3-030-59719-1_1 http://doi.org/10.1007/978-3-030-59719-1_1

Yan K , Tang YB , Peng YF , et al. Multitask universal lesion analysis network for joint lesion detection, tagging, and segmentation, In: Shen D. et al. (eds) Medical Image Computing and Computer Assisted Intervention - MICCAI 2019. MICCAI 2019 . Lecture Notes in Computer Science , vol 11769 . p. 194 - 202 . Springer, Cham. doi: 10.1007/978-3-030-32226-7_22 1001-9294-36-3-210/978-3-030-32226-7_22 . DOI: 10.1007/978-3-030-32226-7_22 http://doi.org/10.1007/978-3-030-32226-7_22

Tang YB , Yan K , Tang YX , et al . ULDor: A universal lesion detector for CT scans with pseudo masks and hard negative example mining . 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) , 2019 . p. 833 - 6 . doi: 10.1109/ISBI.2019.8759478 1001-9294-36-3-210/ISBI.2019.8759478 . DOI: 10.1109/ISBI.2019.8759478 http://doi.org/10.1109/ISBI.2019.8759478

Setio AA , Traverso A , de Bel T , et al . Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge . Med Image Anal 2017 ; 42 : 1 - 13 . doi: 10.1016/j.media.2017.06.015 1001-9294-36-3-210/j.media.2017.06.015 . DOI: 10.1016/j.media.2017.06.015 http://doi.org/10.1016/j.media.2017.06.015 https://linkinghub.elsevier.com/retrieve/pii/S1361841517301020 https://linkinghub.elsevier.com/retrieve/pii/S1361841517301020

16. Bilic P , Christ FP , Vorontsov E , et al . The Liver Tumor Segmentation Benchmark (LiTS). arXiv:1901.04056vi [preprint], 2019. Available from . The Liver Tumor Segmentation Benchmark (LiTS). arXiv:1901.04056vi [preprint], 2019 . Available from http://arxiv.org/abs/1901.04056. Accessed June 20, 2021. http://arxiv.org/abs/1901.04056 http://arxiv.org/abs/1901.04056

17. Roth H , Lu L , Seff A , et al . ( 2015 ). A new 2.5 D representation for lymph node detection in CT [Data set] . The Cancer Imaging Archive . doi: 10.7937/K9/TCIA.2015.AQIIDCNM 1001-9294-36-3-210/K9/TCIA.2015.AQIIDCNM . DOI: 10.7937/K9/TCIA.2015.AQIIDCNM http://doi.org/10.7937/K9/TCIA.2015.AQIIDCNM

18. Antonelli M , Reinke A , Bakas S , et al . The Medical Segmentation Decathlon . arXiv: 2106. 05735v1 [preprint], 2021 . Available from https://arxiv.org/abs/2106.05735. Accessed June 20, 2021. https://arxiv.org/abs/2106.05735 https://arxiv.org/abs/2106.05735

National Cancer Institute . Cancer Data Access System . The National Lung Screening Trial (NLST) . Available from https://cdas.cancer.gov/nlst/. Accessed June 20, 2021. https://cdas.cancer.gov/nlst/ https://cdas.cancer.gov/nlst/

Nin CS , de Souza VVS , Do Amaral RH , et al . Thoracic lymphadenopathy in benign diseases: a state of the art review . Respir Med 2016 ; 112 : 10 - 17 . doi: 10.1016/j.rmed.2016.01.021 1001-9294-36-3-210/j.rmed.2016.01.021 . DOI: 10.1016/j.rmed.2016.01.021 http://doi.org/10.1016/j.rmed.2016.01.021 https://linkinghub.elsevier.com/retrieve/pii/S0954611116300178 https://linkinghub.elsevier.com/retrieve/pii/S0954611116300178

Wu ZJ , Wang ZJ , Zheng ZY , et al . Risk factors for lymph node metastasis and survival outcomes in colorectal neuroendocrine tumors . Cancer Manag Res 2020 ; 12 : 7151 - 64 . doi: 10.2147/CMAR.S256723 1001-9294-36-3-210/CMAR.S256723 . DOI: 10.2147/CMAR.S256723 http://doi.org/10.2147/CMAR.S256723 https://www.dovepress.com/risk-factors-for-lymph-node-metastasis-and-survival-outcomes-in-colore-peer-reviewed-article-CMAR https://www.dovepress.com/risk-factors-for-lymph-node-metastasis-and-survival-outcomes-in-colore-peer-reviewed-article-CMAR

Li ZH , Zhang S , Zhang JG , et al. MVP-Net: multi-view FPN with position-aware attention for deep universal lesion detection. In: Shen D et al. (eds) Medical Image Computing and Computer Assisted Intervention - MICCAI 2019 MICCAI 2019 . MICCAI 2019 . Lecture Notes in Computer Science , vol 11769 . p. 13 - 21 . Springer, Cham. doi: 10.1007/978-3-030-32226-7_2 1001-9294-36-3-210/978-3-030-32226-7_2 . DOI: 10.1007/978-3-030-32226-7_2 http://doi.org/10.1007/978-3-030-32226-7_2

Tao QY , Ge ZY , Cai JF , et al. Improving deep lesion detection using 3D contextual and spatial attention. In: Shen D et al. (eds) Medical Image Computing and Computer Assisted Intervention - MICCAI 2019. MICCAI 2019 . Lecture Notes in Computer Science , vol 11769 . p. 185 - 93 . Springer, Cham. doi: 10.1007/978-3-030-32226-7_21 1001-9294-36-3-210/978-3-030-32226-7_21 . DOI: 10.1007/978-3-030-32226-7_21 http://doi.org/10.1007/978-3-030-32226-7_21

Zhang S , Xu JC , Chen YC , et al. Revisiting 3D context modeling with supervised pre-training for universal lesion detection in CT slices, In: Martel AL et al. (eds) Medical Image Computing and Computer Assisted Intervention - MICCAI 2020. MICCAI 2020 . Lecture Notes in Computer Science , vol 12264 . p. 542 - 51 . Springer, Cham. doi: 10.1007/978-3-030-59719-1_53 1001-9294-36-3-210/978-3-030-59719-1_53 . DOI: 10.1007/978-3-030-59719-1_53 http://doi.org/10.1007/978-3-030-59719-1_53

Views

564

Downloads

CSCD

Alert me when the article has been cited

Submit

Publicity Resources

Prostate Cancer Risk Prediction and Online Calculation Based on Machine Learning Algorithm

Radiomics in Antineoplastic Agents Development: Application and Challenge in Response Evaluation

Histopathological Diagnosis System for Gastritis Using Deep Learning Algorithm

An Insight of the First Community Infected COVID-19 Patient in Beijing by Imported Case: Role of Deep Learning-Assisted CT Diagnosis

Medical Knowledge Extraction and Analysis from Electronic Medical Records Using Deep Learning

Related Author

Qinxue Chang

Xiaomeng Wang

Keyun Wang

He Wang

Zhuang Cui

Changping Li

Chun Wang

Li Jiazheng

Related Institution

Department of Health Statistics, School of Public Health, Tianjin Medical University

Department of Medical Imaging, Peking University First Hospital

Department of Radiology, Peking University Cancer Hospital & Institute, Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education)

Department of Pathology, Chinese PLA General Hospital & Medical School

Artificial Intelligence Lab, Thorough Images

⁰