Semi-supervised Long-tail Endoscopic Image Classification

Runnan Cao; Mengjie Fang; Hailing Li; Jie Tian; Di Dong

doi:10.24920/004135

Your Location：

Home >

Browse articles >

Semi-supervised Long-tail Endoscopic Image Classification

Chinese Medical Sciences Journal Vol. 37, Issue 3, Pages: 171-180(2022)

Scientific Data Sharing and Reuse:Original Article | Updated：2024-04-10

- Semi-supervised Long-tail Endoscopic Image Classification
- Affiliations：
  
  1. 1School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
  2. 2CAS Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
  3. 3Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing 100191, China
  4. 4Engineering Research Center of Molecular and Neuro Imaging of Ministry of Education, School of Life Science and Technology, Xidian University, Xi’an 710126, China
- Author bio：
- Funds：
- DOI：10.24920/004135
  CLC：
- Received：30 June 2022，
  
  Accepted：2022-9-9，
  
  Published Online：27 September 2022，
  
  Published：30 September 2022
- Accepted：
Scan QR Code
Runnan Cao, Mengjie Fang, Hailing Li, et al. Semi-supervised Long-tail Endoscopic Image Classification[J]. Chinese medical sciences journal, 2022, 37(3): 171-180.
DOI：

Runnan Cao, Mengjie Fang, Hailing Li, et al. Semi-supervised Long-tail Endoscopic Image Classification[J]. Chinese medical sciences journal, 2022, 37(3): 171-180. DOI： 10.24920/004135.

摘要

目的

探索半监督学习算法在内镜图像长尾分类中的应用。

方法

我们在HyperKvasir数据集上探索了半监督的内镜图像长尾分类

该数据集是最大的胃肠道公共数据集

有23个不同的类别。使用基于一致性正则化和伪标签的半监督学习算法FixMatch

在将训练数据集和测试数据集按4:1的比例进行划分后

按照20%、50%和100%的比例抽取有标签的训练样本

以测试在有标签数据有限下的分类性能。

结果

通过微观平均、宏观平均评价指标和马修斯相关系数（Mathews correlation coefficient

MCC）作为总体评价指标来评估分类性能。半监督学习算法在有标签训练数据比例为20%、50%和100%的情况下

MCC分别从0.8761提高到0.8850、0.8983提高到0.8994、0.9075提高到0.9095。在有标签训练数据比例为20%的情况下

半监督学习算法可以提高微观平均和宏观平均的分类性能。对于50%和100%的情况

半监督学习算法可以提高微观平均下的分类性能

但会损害宏观平均的分类性能。通过分析每个类的混淆矩阵和标注偏差

我们发现基于伪标签的半监督学习算法加剧了分类器对头类的偏好

导致头类的性能提高而尾类的性能下降。

结论

半监督学习算法可以提高内镜图像长尾分类的性能

特别是在标签极其有限的情况下

这可能有利于为小医院建立辅助诊断系统。然而

伪标签策略可能会放大类不平衡的影响

从而损害尾部类的分类性能。

Abstract

Objective

To explore the semi-supervised learning (SSL) algorithm for long-tail endoscopic image classification with limited annotations.

Method

We explored semi-supervised long-tail endoscopic image classification in HyperKvasir

the largest gastrointestinal public dataset with 23 diverse classes. Semi-supervised learning algorithm FixMatch was applied based on consistency regularization and pseudo-labeling. After splitting the training dataset and the test dataset at a ratio of 4:1

we sampled 20%

50%

and 100% labeled training data to test the classification with limited annotations.

Results

The classification performance was evaluated by micro-average and macro-average evaluation metrics

with the Mathews correlation coefficient (MCC) as the overall evaluation. SSL algorithm improved the classification performance

with MCC increasing from 0.8761 to 0.8850

from 0.8983 to 0.8994

and from 0.9075 to 0.9095 with 20%

50%

and 100% ratio of labeled training data

respectively. With a 20% ratio of labeled training data

SSL improved both the micro-average and macro-average classification performance; while for the ratio of 50% and 100%

SSL improved the micro-average performance but hurt macro-average performance. Through analyzing the confusion matrix and labeling bias in each class

we found that the pseudo-based SSL algorithm exacerbated the classifier’s preference for the head class

resulting in improved performance in the head class and degenerated performance in the tail class.

Conclusion

SSL can improve the classification performance for semi-supervised long-tail endoscopic image classification

especially when the labeled data is extremely limited

which may benefit the building of assisted diagnosis systems for low-volume hospitals. However

the pseudo-labeling strategy may amplify the effect of class imbalance

which hurts the classification performance for the tail class.

关键词

Keywords

references

Sung H , Ferlay J , Siegel RL , et al. Global Cancer Statistics 2020: globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries . CA Cancer J Clin 2021 ; 71 ( 3 ): 209 - 49 . doi: 10.3322/caac.21660 https://dx.doi.org/10.3322/caac.21660 . DOI: 10.3322/caac.21660 http://doi.org/10.3322/caac.21660

Asplund J , Kauppila JH , Mattsson F , et al. Survival trends in gastric adenocarcinoma: a population-based study in Sweden . Ann Surg Oncol 2018 ; 25 ( 9 ): 2693 - 702 . doi: 10.1245/s10434-018-6627-y https://dx.doi.org/10.1245/s10434-018-6627-y . DOI: 10.1245/s10434-018-6627-y http://doi.org/10.1245/s10434-018-6627-y

Hosokawa O , Hattori M , Douden K , et al. Difference in accuracy between gastroscopy and colonoscopy for detection of cancer . Hepatogastroenterology 2007 ; 54 ( 74 ): 442 - 4 . doi: 10.1136/gut.2006.115394 https://dx.doi.org/10.1136/gut.2006.115394 . DOI: 10.1136/gut.2006.115394 http://doi.org/10.1136/gut.2006.115394

Sivak M . Gastrointestinal endoscopy: past and future . Gut 2006 ; 55 ( 8 ): 1061 - 4 . doi: 10.1136/gut.2005.086371 https://dx.doi.org/10.1136/gut.2005.086371 . DOI: 10.1136/gut.2005.086371 http://doi.org/10.1136/gut.2005.086371

Litjens G , Kooi T , Bejnordi BE , et al. A survey on deep learning in medical image analysis . Med Image Anal 2017 ; 42 : 60 - 88 . doi: 10.1016/j.media.2017.07.005 https://dx.doi.org/10.1016/j.media.2017.07.005 . DOI: S1361-8415(17)30113-5 http://doi.org/S1361-8415(17)30113-5

Dong D , Tang Z , Wang S , et al. The role of imaging in the detection and management of covid-19: a review . IEEE Rev Biomed Eng 2020 ; 14 : 16 - 29 . doi: 10.1109/RBME.2020.2990959 https://dx.doi.org/10.1109/RBME.2020.2990959 . DOI: 10.1109/RBME.2020.2990959 http://doi.org/10.1109/RBME.2020.2990959 https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=4664312 https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=4664312

Liu Z , Wang S , Di Dong JW , et al. The applications of radiomics in precision diagnosis and treatment of oncology: opportunities and challenges . Theranostics 2019 ; 9 ( 5 ): 1303 . doi: 10.7150/thno.30309 https://dx.doi.org/10.7150/thno.30309 . DOI: 10.7150/thno.30309 http://doi.org/10.7150/thno.30309

Poon CC , Jiang Y , Zhang R , et al. Ai-Doscopist: a real-time deep-learning-based algorithm for localising polyps in colonoscopy videos with edge computing devices . NPJ Digit Med 2020 ; 3 ( 1 ): 1 - 8 . doi: 10.1038/s41746-020-0281-z https://dx.doi.org/10.1038/s41746-020-0281-z . DOI: 10.1038/s41746-020-0281-z http://doi.org/10.1038/s41746-020-0281-z https://doi.org/10.1038/s41746-020-0281-z https://doi.org/10.1038/s41746-020-0281-z

Jha D , Smedsrud PH , Johansen D , et al. A comprehensive study on colorectal polyp segmentation with resunet++, conditional random field and test-time augmentation . IEEE JBHI 2021 ; 25 ( 6 ): 2029 - 40 . doi: 10.1109/JBHI.2021.3049304 https://dx.doi.org/10.1109/JBHI.2021.3049304 . DOI: 10.1109/JBHI.2021.3049304 http://doi.org/10.1109/JBHI.2021.3049304

Hsu CC , Ma HT , Lee JY SSSNet: Small-scale-aware siamese network for gastric cancer detection . 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS): IEEE , 2019 , 1 - 5 . doi: 10.1109/AVSS.2019.8909849 https://dx.doi.org/10.1109/AVSS.2019.8909849 . DOI: 10.1109/AVSS.2019.8909849 http://doi.org/10.1109/AVSS.2019.8909849

Hu H , Gong L , Dong D , et al . Identifying early gastric cancer under magnifying narrow-band images with deep learning: a multicenter study . Gastrointest Endosc 2021 ; 93 ( 6 ): 1333 - 1341 . e3 . doi: 10.1016/j.gie.2020.11.014 https://dx.doi.org/10.1016/j.gie.2020.11.014 . DOI: 10.1016/j.gie.2020.11.014 http://doi.org/10.1016/j.gie.2020.11.014

Dong D , Tang L , Li ZY , et al. Development and validation of an individualized nomogram to identify occult peritoneal metastasis in patients with advanced gastric cancer . Ann Oncol 2019 ; 30 ( 3 ): 431 - 8 . doi: 10.1093/annonc/mdz001 https://dx.doi.org/10.1093/annonc/mdz001 . DOI: S0923-7534(19)31081-6 http://doi.org/S0923-7534(19)31081-6

Gong L , Wang M , Shu L , et al . Automatic captioning of early gastric cancer via magnification endoscopy with narrow band imaging . Gastrointest Endosc 2022 . S0016-5107 ( 22 ) 01836 - 3 . doi: 10.1016/j.gie.2022.07.019 https://dx.doi.org/10.1016/j.gie.2022.07.019 . DOI: 10.1016/j.gie.2022.07.019 http://doi.org/10.1016/j.gie.2022.07.019

Hirasawa T , Aoyama K , Tanimoto T , et al. Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images . Gastric Cancer 2018 ; 21 ( 4 ): 653 - 60 . doi: 10.1007/s10120-018-0793-2 https://dx.doi.org/10.1007/s10120-018-0793-2 . DOI: 10.1007/s10120-018-0793-2 http://doi.org/10.1007/s10120-018-0793-2

Luo H , Xu G , Li C , et al. Real-time artificial intelligence for detection of upper gastrointestinal cancer by endoscopy: a multicentre, case-control, diagnostic study . Lancet Oncol 2019 ; 20 ( 12 ): 1645 - 54 . doi: 10.1016/S1470-2045(19)30637-0 https://dx.doi.org/10.1016/S1470-2045(19)30637-0 . DOI: S1470-2045(19)30637-0 http://doi.org/S1470-2045(19)30637-0

Yoon HJ , Kim S , Kim JH , et al. A lesion-based convolutional neural network improves endoscopic detection and depth prediction of early gastric cancer . J Clin Med 2019 ; 8 ( 9 ): 1310 . doi: 10.3390/jcm8091310 https://dx.doi.org/10.3390/jcm8091310 . DOI: 10.3390/jcm8091310 http://doi.org/10.3390/jcm8091310 https://www.mdpi.com/2077-0383/8/9/1310 https://www.mdpi.com/2077-0383/8/9/1310

Ikenoyama Y , Hirasawa T , Ishioka M , et al. Detecting Early Gastric Cancer: Comparison between the diagnostic ability of convolutional neural networks and endoscopists . Dig Endosc 2021 ; 33 ( 1 ): 141 - 50 . doi: 10.1111/den.13688 https://dx.doi.org/10.1111/den.13688 . DOI: 10.1111/den.13688 http://doi.org/10.1111/den.13688 https://onlinelibrary.wiley.com/toc/14431661/33/1 https://onlinelibrary.wiley.com/toc/14431661/33/1

Bernal J , Tajkbaksh N , Sanchez FJ , et al. Comparative validation of polyp detection methods in video colonoscopy: results from the Miccai 2015 Endoscopic Vision Challenge . IEEE T Med Imaging 2017 ; 36 ( 6 ): 1231 - 49 . doi: 10.1109/TMI.2017.2664042 https://dx.doi.org/10.1109/TMI.2017.2664042 . DOI: 10.1109/TMI.2017.2664042 http://doi.org/10.1109/TMI.2017.2664042

Angermann Q , Bernal J , Sánchez-Montes C , et al. Towards real-time polyp detection in colonoscopy videos: Adapting still frame-based methodologies for video sequences analysis . // Computer Assisted and Robotic Endoscopy and Clinical Image-Based Procedures 2017 ; 29 - 41 . doi: 10.1007/978-3-319-67543-5_3 https://dx.doi.org/10.1007/978-3-319-67543-5_3 . DOI: 10.1007/978-3-319-67543-5_3 http://doi.org/10.1007/978-3-319-67543-5_3

Pogorelov K , Randel KR , Griwodz C , et al. Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection . // Proceedings of the 8th ACM on Multimedia Systems Conference 2017 ; 164 - 9 . doi: 10.1145/3193289 https://dx.doi.org/10.1145/3193289 . DOI: 10.1145/3193289 http://doi.org/10.1145/3193289

Hicks SA , Thambawita V , Hammer HL , et al. Acm Multimedia Biomedia 2020 Grand Challenge overview . // Proceedings of the 28th ACM International Conference on Multimedia 2020 ; 4655 - 8 . doi: 10.1145/3394171.3416287 https://dx.doi.org/10.1145/3394171.3416287 . DOI: 10.1145/3394171.3416287 http://doi.org/10.1145/3394171.3416287

Zhu XJ . Semi-supervised learning literature survey . University of Wisconsin-Madison 2006 . doi: 10.1.1.103.1693 https://dx.doi.org/10.1.1.103.1693 . DOI: 10.1.1.103.1693 http://doi.org/10.1.1.103.1693

Borgli H , Thambawita V , Smedsrud PH , et al. Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy . Sci Data 2020 ; 7 ( 1 ): 1 - 14 . doi: 10.1038/s41597-020-00622-y https://dx.doi.org/10.1038/s41597-020-00622-y . DOI: 10.1038/s41597-020-00622-y http://doi.org/10.1038/s41597-020-00622-y https://doi.org/10.1038/s41597-020-00622-y https://doi.org/10.1038/s41597-020-00622-y

Pogorelov K , Riegler M , Halvorsen P , et al. Medico Multimedia Task at Mediaeval 2018 . // CEUR Workshop Proceedings: Technical University of Aachen 2018 ; 1 - 4 . doi: 10.48550/arXiv.2012.15244 https://dx.doi.org/10.48550/arXiv.2012.15244 . DOI: 10.48550/arXiv.2012.15244 http://doi.org/10.48550/arXiv.2012.15244

Harzig P , Einfalt M , Lienhart R . Automatic disease detection and report generation for gastrointestinal tract examination // Proceedings of the 27th ACM International Conference on Multimedia 2019 ; 2573 - 7 . doi: 10.1145/3343031.3356066 https://dx.doi.org/10.1145/3343031.3356066 . DOI: 10.1145/3343031.3356066 http://doi.org/10.1145/3343031.3356066

Weese J , Lorenz C . Four challenges in medical image analysis from an industrial perspective . Med Image Anal 2016 ; 44 - 49 . doi: 10.1016/j.media.2016.06.023 https://dx.doi.org/10.1016/j.media.2016.06.023 . DOI: 10.1016/j.media.2016.06.023 http://doi.org/10.1016/j.media.2016.06.023

Chapelle O , Scholkopf B , Zien A . (Chapelle, O. Et Al., Eds.; 2006) . IEEE Trans Neural Netwlearn Syst 2009 ; 20 ( 3 ): 542 . doi: 10.1109/TNN.2009.2015974 https://dx.doi.org/10.1109/TNN.2009.2015974 . DOI: 10.1109/TNN.2009.2015974 http://doi.org/10.1109/TNN.2009.2015974

Madani A , Ong JR , Tibrewal A , et al. Deep echocardiography: data-efficient supervised and semi-supervised deep learning towards automated diagnosis of cardiac disease . NPJ Digit Med 2018 ; 1 ( 1 ): 1 - 11 . doi: 10.1038/s41746-018-0065-x https://dx.doi.org/10.1038/s41746-018-0065-x . DOI: 10.1038/s41746-018-0065-x http://doi.org/10.1038/s41746-018-0065-x https://doi.org/10.1038/s41746-018-0065-x https://doi.org/10.1038/s41746-018-0065-x

Su H , Shi X , Cai J , et al. Local and global consistency regularized mean teacher for semi-supervised nuclei classification . IN: International Conference on Medical Image Computing and Computer-Assisted Intervention: Springer 2019 ; 559 - 67 . doi: 10.1007/978-3-030-32239-7_62 https://dx.doi.org/10.1007/978-3-030-32239-7_62 . DOI: 10.1007/978-3-030-32239-7_62 http://doi.org/10.1007/978-3-030-32239-7_62

Lee D-H . Pseudo-Label:The simple and efficient semi-supervised learning method for deep neural networks . IN: Workshop on challenges in representation learning, International Conference on Machine Learning 2013 ; 896 . doi: 10.1.1.664.354 https://dx.doi.org/10.1.1.664.354 . DOI: 10.1.1.664.354 http://doi.org/10.1.1.664.354

Sajjadi M , Javanmardi M , Tasdizen T . Regularization with stochastic transformations and perturbations for deep semi-supervised learning . Adv Neural Inf Process Syst 2016 ; 29 . doi: 10.48550/arXiv.1606.04586 https://dx.doi.org/10.48550/arXiv.1606.04586 . DOI: 10.48550/arXiv.1606.04586 http://doi.org/10.48550/arXiv.1606.04586

Sohn K , Berthelot D , Carlini N , et al. Fixmatch: Simplifying semi-supervised learning with consistency and confidence . Adv Neural Inf Process Syst 2020 ; 33 : 596 - 608 . doi: 10.48550/arXiv.2001.07685 https://dx.doi.org/10.48550/arXiv.2001.07685 . DOI: 10.48550/arXiv.2001.07685 http://doi.org/10.48550/arXiv.2001.07685

Rahman MM , Davis DN . Addressing the class imbalance problem in medical datasets . Int Mach Learn 2013 ; 3 ( 2 ): 224 . doi: 10.7763/IJMLC.2013.V3.307 https://dx.doi.org/10.7763/IJMLC.2013.V3.307 . DOI: 10.7763/IJMLC.2013.V3.307 http://doi.org/10.7763/IJMLC.2013.V3.307

Zhou B , Cui Q , Wei X-S , et al. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition . In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020 ; 9719 - 28 . doi: 10.1109/CVPR42600.2020.00974 https://dx.doi.org/10.1109/CVPR42600.2020.00974 . DOI: 10.1109/CVPR42600.2020.00974 http://doi.org/10.1109/CVPR42600.2020.00974

Cao KD , Wei C , Gaidon A , et al. Learning imbalanced datasets with label-distribution-aware margin loss . Adv Neural Inf Process Syst 2019 ; 32 . doi: 10.48550/arXiv.1906.07413 https://dx.doi.org/10.48550/arXiv.1906.07413 . DOI: 10.48550/arXiv.1906.07413 http://doi.org/10.48550/arXiv.1906.07413

Kang B , Xie S , Rohrbach M , et al. Decoupling representation and classifier for long-tailed recognition . arXiv preprint 2019 . doi: 10.48550/arXiv.1910.09217 https://dx.doi.org/10.48550/arXiv.1910.09217 . DOI: 10.48550/arXiv.1910.09217 http://doi.org/10.48550/arXiv.1910.09217

Kim J , Hur Y , Park S , et al. Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning . Adv in Neural Inf Process Syst 2020 ; 33 : 14567 - 79 . doi: 10.48550/arXiv:2007 https://dx.doi.org/10.48550/arXiv:2007 .08844. DOI: 10.48550/arXiv:2007 http://doi.org/10.48550/arXiv:2007

Sandler M , Howard A , Zhu M , et al. Mobilenetv2: Inverted residuals and linear bottlenecks . // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018 ; 4510 - 4520 . doi: 10.1109/CVPR.2018.00474 https://dx.doi.org/10.1109/CVPR.2018.00474 . DOI: 10.1109/CVPR.2018.00474 http://doi.org/10.1109/CVPR.2018.00474

Wang Y , Yao Q , Kwok JT , et al. Generalizing from a few examples: a survey on few-shot learning . ACM Comput Surv 2020 ;53 ( 3 ): 1 - 34 . doi: 10.1145/3386252 https://dx.doi.org/10.1145/3386252 . DOI: 10.1145/3386252 http://doi.org/10.1145/3386252

Views

330

Downloads

CSCD

Alert me when the article has been cited

Submit

Publicity Resources

Histopathological Diagnosis System for Gastritis Using Deep Learning Algorithm

Advances of Artificial Intelligence Application in Medical Imaging of Ovarian Cancers

Ontology: Footstone for Strong Artificial Intelligence

Collaborations of Industry, Academia, Research and Application Improve the Healthy Development of Medical Imaging Artificial Intelligence Industry in China

The Artificial Intelligence-Enabled Medical Imaging: Today and Its Future

Related Author

Song Zhigang

Shi Huaiyin

Wang Yuefeng

Ba Wei

Wang Shuhao

Liu Cancheng

Xu Chen

Xiaofei Huo

Related Institution

Department of Pathology, Chinese PLA General Hospital & Medical School

Artificial Intelligence Lab, Thorough Images

Department of Radiology, Beijing United Family Hospital

Department of Radiology, Fushun Central Hospital

Department of Biomedical Engineering, Institute of Basic Medical Sciences, Chinese Academy of Medical Sciences

⁰