Chinese Medical Sciences Journal ›› 2021, Vol. 36 ›› Issue (3): 210-217.doi: 10.24920/003968

Special Issue: 人工智能与精准肿瘤学

• Original Article • Previous Articles     Next Articles

External and Internal Validation of a Computer Assisted Diagnostic Model for Detecting Multi-Organ Mass Lesions in CT images

Lianyan Xu1, Ke Yan2, Le Lu2, Weihong Zhang1, Xu Chen3, Xiaofei Huo3, Jingjing Lu3()   

  1. 1Department of Radiology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing 100730, China
    2PAII Inc., Bethesda, MD 20817, USA
    3Department of Radiology, Beijing United Family Hospital, Beijing 100015, China
  • Received:2021-07-10 Accepted:2021-08-30 Published:2021-09-30 Online:2021-09-23
A universal lesion detector (ULDor) using deep learning was evaluated for its  ability to generalize in clinical setting via both external and internal validation. The performance was organ dependant and need further optimisation and iterative upgrades.

Objective We developed a universal lesion detector (ULDor) which showed good performance in in-lab experiments. The study aims to evaluate the performance and its ability to generalize in clinical setting via both external and internal validation.
Methods The ULDor system consists of a convolutional neural network (CNN) trained on around 80K lesion annotations from about 12K CT studies in the DeepLesion dataset and 5 other public organ-specific datasets. During the validation process, the test sets include two parts: the external validation dataset which was comprised of 164 sets of non-contrasted chest and upper abdomen CT scans from a comprehensive hospital, and the internal validation dataset which was comprised of 187 sets of low-dose helical CT scans from the National Lung Screening Trial (NLST). We ran the model on the two test sets to output lesion detection. Three board-certified radiologists read the CT scans and verified the detection results of ULDor. We used positive predictive value (PPV) and sensitivity to evaluate the performance of the model in detecting space-occupying lesions at all extra-pulmonary organs visualized on CT images, including liver, kidney, pancreas, adrenal, spleen, esophagus, thyroid, lymph nodes, body wall, thoracic spine, etc.
Results In the external validation, the lesion-level PPV and sensitivity of the model were 57.9% and 67.0%, respectively. On average, the model detected 2.1 findings per set, and among them, 0.9 were false positives. ULDor worked well for detecting liver lesions, with a PPV of 78.9% and a sensitivity of 92.7%, followed by kidney, with a PPV of 70.0% and a sensitivity of 58.3%. In internal validation with NLST test set, ULDor obtained a PPV of 75.3% and a sensitivity of 52.0% despite the relatively high noise level of soft tissue on images.
Conclusions The performance tests of ULDor with the external real-world data have shown its high effectiveness in multiple-purposed detection for lesions in certain organs. With further optimisation and iterative upgrades, ULDor may be well suited for extensive application to external data.

Key words: lesion detection, computer-aided diagnosis, convolutional neural network, deep learning

Copyright © 2021 Chinese Academy of Medical Sciences.  京公安备110402430088  京ICP备06002729号-1  Powered by Magtech.

Supervised by National Health Commission of the People's Republic of China

9 Dongdan Santiao, Dongcheng district, Beijing, 100730 China

Tel: 86-10-65105897  Fax:86-10-65133074 


Copyright © 2018 Chinese Academy of Medical Sciences

All right reserved.

京公安备110402430088  京ICP备06002729号-1