ERKT-Net: Implementing Efficient and Robust Knowledge Distillation for Remote Sensing Image Classification

Huaxiang Song; Yafang Li; Xiaowen Li; Yuxuan Zhang; Yangyan Zhu; Yong Zhou

doi:10.4108/eetinis.v11i3.4748

Authors

Huaxiang Song Hunan University of Arts and Science
Yafang Li Hunan University of Arts and Science
Xiaowen Li Hunan University of Arts and Science
Yuxuan Zhang Hunan University of Arts and Science
Yangyan Zhu Hunan University of Arts and Science
Yong Zhou

DOI:

https://doi.org/10.4108/eetinis.v11i3.4748

Keywords:

ERKT-Net, Variance-Suppression Strategy, Knowledge Distillation, Remote Sensing Image Classification, Deep Learning

Abstract

The classification of Remote Sensing Images (RSIs) poses a significant challenge due to the presence of clustered ground objects and noisy backgrounds. While many approaches rely on scaling models to enhance accuracy, the deployment of RSI classifiers often requires substantial computational and storage resources, thus necessitating the use of lightweight algorithms. In this paper, we present an efficient and robust knowledge transfer network named ERKT-Net, which is designed to provide a lightweight yet accurate Convolutional Neural Network (CNN) classifier. This method utilizes innovative yet simple concepts to better accommodate the inherent nature of RSIs, thereby significantly improving the efficiency and robustness of traditional Knowledge Distillation (KD) techniques developed on ImageNet-1K. We evaluated ERKT-Net on three benchmark RSI datasets and found that it demonstrated superior accuracy and a very compact volume compared to 40 other advanced methods published between 2020 and 2023. On the most challenging NWPU45 dataset, ERKT-Net outperformed other KD-based methods with a maximum Overall Accuracy (OA) value of 22.4%. Using the same criterion, it also surpassed the first-ranked multi-model method with a minimum OA value of 0.7 but presented at least an 82% reduction in parameters. Furthermore, ablation experiments indicated that our training approach has significantly improved the efficiency and robustness of classic DA techniques. Notably, it can reduce the time expenditure in the distillation phase by at least 80%, with a slight sacrifice in accuracy. This study confirmed that a logit-based KD technique can be more efficient and effective in developing lightweight yet accurate classifiers, especially when the method is tailored to the inherent characteristics of RSIs.

Downloads

Download data is not yet available.

References

Xu C, Du X, Fan X, Giuliani G, Hu Z, Wang W, et al. Cloud-based storage and computing for remote sensing big data: a technical review. International Journal of Digital Earth 2022;15:1417–45. https://doi.org/10.1080/17538947.2022.2115567. DOI: https://doi.org/10.1080/17538947.2022.2115567

Mountrakis G, Heydari SS. Harvesting the Landsat archive for land cover land use classification using deep neural networks: Comparison with traditional classifiers and multi-sensor benefits. ISPRS Journal of Photogrammetry and Remote Sensing 2023;200:106–19. https://doi.org/10.1016/j.isprsjprs.2023.05.005. DOI: https://doi.org/10.1016/j.isprsjprs.2023.05.005

Dimitrovski I, Kitanovski I, Kocev D, Simidjievski N. Current trends in deep learning for Earth Observation: An open-source benchmark arena for image classification. ISPRS Journal of Photogrammetry and Remote Sensing 2023;197:18–35. https://doi.org/10.1016/j.isprsjprs.2023.01.014. DOI: https://doi.org/10.1016/j.isprsjprs.2023.01.014

Song H, Zhou Y. Simple is best: A single-CNN method for classifying remote sensing images. NHM 2023;18:1600–29. https://doi.org/10.3934/nhm.2023070. DOI: https://doi.org/10.3934/nhm.2023070

Song H. MBC-Net: long-range enhanced feature fusion for classifying remote sensing images. IJICC 2024;17:181–209. https://doi.org/10.1108/IJICC-07-2023-0198. DOI: https://doi.org/10.1108/IJICC-07-2023-0198

Jamali A, Mahdianpari M, Mohammadimanesh F, Homayouni S. A deep learning framework based on generative adversarial networks and vision transformer for complex wetland classification using limited training samples. International Journal of Applied Earth Observation and Geoinformation 2022;115:103095. https://doi.org/10.1016/j.jag.2022.103095. DOI: https://doi.org/10.1016/j.jag.2022.103095

Song H, Yuan Y, Ouyang Z, Yang Y, Xiang H. Quantitative regularization in robust vision transformer for remote sensing image classification. The Photogrammetric Record. 2024: Online First. https://doi.org/10.1111/phor.12489. DOI: https://doi.org/10.1111/phor.12489

Yue J, Fang L, Ghamisi P, Xie W, Li J, Chanussot J, et al. Optical Remote Sensing Image Understanding With Weak Supervision: Concepts, methods, and perspectives. IEEE Geosci Remote Sens Mag 2022;10:250–69. https://doi.org/10.1109/MGRS.2022.3161377. DOI: https://doi.org/10.1109/MGRS.2022.3161377

Thoreau R, Achard V, Risser L, Berthelot B, Briottet X. Active Learning for Hyperspectral Image Classification: A comparative review. IEEE Geosci Remote Sens Mag 2022;10:256–78. https://doi.org/10.1109/MGRS.2022.3169947. DOI: https://doi.org/10.1109/MGRS.2022.3169947

Song H. A Leading but Simple Classification Method for Remote Sensing Images. AETiC 2023;7:1–20. https://doi.org/10.33166/AETiC.2023.03.001. DOI: https://doi.org/10.33166/AETiC.2023.03.001

Chen J, Di X, Xu R, Luo H, Qi H, Zhan P, et al. An efficient scheme for in-orbit remote sensing image data retrieval. Future Generation Computer Systems 2024;150:103–14. https://doi.org/10.1016/j.future.2023.08.017. DOI: https://doi.org/10.1016/j.future.2023.08.017

Wang Y, Zhao C, Dong D, Wang K. Real-time monitoring of insects based on laser remote sensing. Ecological Indicators 2023;151:110302. https://doi.org/10.1016/j.ecolind.2023.110302. DOI: https://doi.org/10.1016/j.ecolind.2023.110302

Zhang Z, Liu Q, Liu X, Zhang Y, Du Z, Cao X. PMNet: a multi-branch and multi-scale semantic segmentation approach to water extraction from high-resolution remote sensing images with edge-cloud computing. J Cloud Comp 2024;13:76. https://doi.org/10.1186/s13677-024-00637-5. DOI: https://doi.org/10.1186/s13677-024-00637-5

Yu D, Xu Q, Guo H, Zhao C, Lin Y, Li D. An Efficient and Lightweight Convolutional Neural Network for Remote Sensing Image Scene Classification. Sensors 2020;20:1999. https://doi.org/10.3390/s20071999. DOI: https://doi.org/10.3390/s20071999

Chen Z, Yang J, Feng Z, Chen L. RSCNet: An Efficient Remote Sensing Scene Classification Model Based on Lightweight Convolution Neural Networks. Electronics 2022;11:3727. https://doi.org/10.3390/electronics11223727. DOI: https://doi.org/10.3390/electronics11223727

Liang L, Wang G. Efficient recurrent attention network for remote sensing scene classification. IET Image Processing 2021;15:1712–21. https://doi.org/10.1049/ipr2.12139. DOI: https://doi.org/10.1049/ipr2.12139

Zheng F, Lin S, Zhou W, Huang H. A Lightweight Dual-Branch Swin Transformer for Remote Sensing Scene Classification. Remote Sensing 2023;15:2865. https://doi.org/10.3390/rs15112865. DOI: https://doi.org/10.3390/rs15112865

Alhichri H, Alswayed AS, Bazi Y, Ammour N, Alajlan NA. Classification of Remote Sensing Images Using EfficientNet-B3 CNN Model With Attention. IEEE Access 2021;9:14078–94. https://doi.org/10.1109/ACCESS.2021.3051085. DOI: https://doi.org/10.1109/ACCESS.2021.3051085

Chen S-B, Wei Q-S, Wang W-Z, Tang J, Luo B, Wang Z-Y. Remote Sensing Scene Classification via Multi-Branch Local Attention Network. IEEE Trans on Image Process 2022;31:99–109. https://doi.org/10.1109/TIP.2021.3127851. DOI: https://doi.org/10.1109/TIP.2021.3127851

Zhao Z, Li J, Luo Z, Li J, Chen C. Remote Sensing Image Scene Classification Based on an Enhanced Attention Module. IEEE Geosci Remote Sensing Lett 2021;18:1926–30. https://doi.org/10.1109/LGRS.2020.3011405. DOI: https://doi.org/10.1109/LGRS.2020.3011405

Wan H, Chen J, Huang Z, Feng Y, Zhou Z, Liu X, et al. Lightweight Channel Attention and Multiscale Feature Fusion Discrimination for Remote Sensing Scene Classification. IEEE Access 2021;9:94586–600. https://doi.org/10.1109/ACCESS.2021.3093308. DOI: https://doi.org/10.1109/ACCESS.2021.3093308

Huang X, Liu F, Cui Y, Chen P, Li L, Li P. Faster and Better: A Lightweight Transformer Network for Remote Sensing Scene Classification. Remote Sensing 2023;15:3645. https://doi.org/10.3390/rs15143645. DOI: https://doi.org/10.3390/rs15143645

Xu C, Zhu G, Shu J. A Lightweight and Robust Lie Group-Convolutional Neural Networks Joint Representation for Remote Sensing Scene Classification. IEEE Trans Geosci Remote Sensing 2022;60:1–15. https://doi.org/10.1109/TGRS.2020.3048024. DOI: https://doi.org/10.1109/TGRS.2020.3048024

Wang X, Xu H, Yuan L, Wen X. A lightweight and stochastic depth residual attention network for remote sensing scene classification. IET Image Processing 2023;17:3106–26. https://doi.org/10.1049/ipr2.12836. DOI: https://doi.org/10.1049/ipr2.12836

Shi C, Zhang X, Sun J, Wang L. Remote Sensing Scene Image Classification Based on Self-Compensating Convolution Neural Network. Remote Sensing 2022;14:545. https://doi.org/10.3390/rs14030545. DOI: https://doi.org/10.3390/rs14030545

Bai L, Liu Q, Li C, Ye Z, Hui M, Jia X. Remote Sensing Image Scene Classification Using Multiscale Feature Fusion Covariance Network With Octave Convolution. IEEE Trans Geosci Remote Sensing 2022;60:1–14. https://doi.org/10.1109/TGRS.2022.3160492. DOI: https://doi.org/10.1109/TGRS.2022.3160492

Zhang W, Jiao L, Liu F, Liu J, Cui Z. LHNet: Laplacian Convolutional Block for Remote Sensing Image Scene Classification. IEEE Trans Geosci Remote Sensing 2022;60:1–13. https://doi.org/10.1109/TGRS.2022.3192321. DOI: https://doi.org/10.1109/TGRS.2022.3192321

Bi Q, Zhou B, Qin K, Ye Q, Xia G-S. All Grains, One Scheme (AGOS): Learning Multigrain Instance Representation for Aerial Scene Classification. IEEE Trans Geosci Remote Sensing 2022;60:1–17. https://doi.org/10.1109/TGRS.2022.3201755. DOI: https://doi.org/10.1109/TGRS.2022.3201755

Guo W, Li S, Yang J, Zhou Z, Liu Y, Lu J, et al. Remote Sensing Image Scene Classification by Multiple Granularity Semantic Learning. IEEE J Sel Top Appl Earth Observations Remote Sensing 2022;15:2546–62. https://doi.org/10.1109/JSTARS.2022.3158703. DOI: https://doi.org/10.1109/JSTARS.2022.3158703

Shi A, Li Z, Wang X. A lightweight skip-connected expansion inception network for remote sensing scene classification. Remote Sensing Letters 2023;14:1098–108. https://doi.org/10.1080/2150704X.2023.2266118. DOI: https://doi.org/10.1080/2150704X.2023.2266118

Ao L, Feng K, Sheng K, Zhao H, He X, Chen Z. TPENAS: A Two-Phase Evolutionary Neural Architecture Search for Remote Sensing Image Classification. Remote Sensing 2023;15:2212. https://doi.org/10.3390/rs15082212. DOI: https://doi.org/10.3390/rs15082212

Broni-Bediako C, Murata Y, Mormille LHB, Atsumi M. Searching for CNN Architectures for Remote Sensing Scene Classification. IEEE Trans Geosci Remote Sensing 2022;60:1–13. https://doi.org/10.1109/TGRS.2021.3097938. DOI: https://doi.org/10.1109/TGRS.2021.3097938

Shen J, Cao B, Zhang C, Wang R, Wang Q. Remote Sensing Scene Classification Based on Attention-Enabled Progressively Searching. IEEE Trans Geosci Remote Sensing 2022;60:1–13. https://doi.org/10.1109/TGRS.2022.3186588. DOI: https://doi.org/10.1109/TGRS.2022.3186588

Cristian Buciluǎ, Rich Caruana, Alexandru Niculescu-Mizil. Model Compression, Philadelphia, Pennsylvania, USA: Association for Computing Machinery; 2006, p. Pages 535-541. https://doi.org/10.1145/1150402.1150464. DOI: https://doi.org/10.1145/1150402.1150464

Hinton G, Vinyals O, Dean J. Distilling the Knowledge in a Neural Network. arXiv, 2015. Available at: https://doi.org/10.48550/arXiv.1503.02531. Accessed on: May 01, 2024.

Romero A, Ballas N, Kahou SE, Chassang A, Gatta C, Bengio Y. FitNets: Hints for Thin Deep Nets, arXiv,2015. Available at: https://doi.org/10.48550/arXiv.1412.6550. Accessed on: May 01, 2024.

Park W, Kim D, Lu Y, Cho M. Relational Knowledge Distillation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA: IEEE; 2019, p. 3962–71. https://doi.org/10.1109/CVPR.2019.00409. DOI: https://doi.org/10.1109/CVPR.2019.00409

Zhao B, Cui Q, Song R, Qiu Y, Liang J. Decoupled Knowledge Distillation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA: IEEE; 2022, p. 11943–52. https://doi.org/10.1109/CVPR52688.2022.01165. DOI: https://doi.org/10.1109/CVPR52688.2022.01165

Huang T, You S, Wang F, Qian C, Xu C. Knowledge Distillation from A Stronger Teacher. In: Koyejo S, Mohamed S, Agarwal A, Belgrave D, Cho K, Oh A, editors. Advances in Neural Information Processing Systems, vol. 35, Curran Associates, Inc.; 2022, p. 33716–27. Available at: https://proceedings.neurips.cc/paper_files/paper/2022/file/da669dfd3c36c93905a17ddba01eef06-Paper-Conference.pdf. Accessed on: May 01, 2024.

Yim J, Joo D, Bae J, Kim J. A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI: IEEE; 2017, p. 7130–8. https://doi.org/10.1109/CVPR.2017.754. DOI: https://doi.org/10.1109/CVPR.2017.754

Stanton S, Izmailov P, Kirichenko P, Alemi AA, Wilson AG. Does Knowledge Distillation Really Work? In: Ranzato M, Beygelzimer A, Dauphin Y, Liang PS, Vaughan JW, editors. Advances in Neural Information Processing Systems, vol. 34, Curran Associates, Inc.; 2021, p. 6906–19. Available at: https://proceedings.neurips.cc/paper_files/paper/2021/file/376c6b9ff3bedbbea56751a84fffc10c-Paper.pdf. Accessed on: May 01, 2024.

Beyer L, Zhai X, Royer A, Markeeva L, Anil R, Kolesnikov A. Knowledge distillation: A good teacher is patient and consistent. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA: IEEE; 2022, p. 10915–24. https://doi.org/10.1109/CVPR52688.2022.01065. DOI: https://doi.org/10.1109/CVPR52688.2022.01065

Chen G, Zhang X, Tan X, Cheng Y, Dai F, Zhu K, et al. Training Small Networks for Scene Classification of Remote Sensing Images via Knowledge Distillation. Remote Sensing 2018;10:719. https://doi.org/10.3390/rs10050719. DOI: https://doi.org/10.3390/rs10050719

Xu K, Deng P, Huang H. Vision Transformer: An Excellent Teacher for Guiding Small Networks in Remote Sensing Image Scene Classification. IEEE Trans Geosci Remote Sensing 2022;60:1–15. https://doi.org/10.1109/TGRS.2022.3152566. DOI: https://doi.org/10.1109/TGRS.2022.3152566

Wang X, Zhu J, Yan Z, Zhang Z, Zhang Y, Chen Y, et al. LaST: Label-Free Self-Distillation Contrastive Learning With Transformer Architecture for Remote Sensing Image Scene Classification. IEEE Geosci Remote Sensing Lett 2022;19:1–5. https://doi.org/10.1109/LGRS.2022.3185088. DOI: https://doi.org/10.1109/LGRS.2022.3185088

Li D, Nan Y, Liu Y. Remote Sensing Image Scene Classification Model Based on Dual Knowledge Distillation. IEEE Geosci Remote Sensing Lett 2022;19:1–5. https://doi.org/10.1109/LGRS.2022.3208904. DOI: https://doi.org/10.1109/LGRS.2022.3208904

Hu Y, Huang X, Luo X, Han J, Cao X, Zhang J. Variational Self-Distillation for Remote Sensing Scene Classification. IEEE Trans Geosci Remote Sensing 2022;60:1–13. https://doi.org/10.1109/TGRS.2022.3194549. DOI: https://doi.org/10.1109/TGRS.2022.3194549

Xing S, Xing J, Ju J, Hou Q, Ding X. Collaborative Consistent Knowledge Distillation Framework for Remote Sensing Image Scene Classification Network. Remote Sensing 2022;14:5186. https://doi.org/10.3390/rs14205186. DOI: https://doi.org/10.3390/rs14205186

Zhao Q, Ma Y, Lyu S, Chen L. Embedded Self-Distillation in Compact Multibranch Ensemble Network for Remote Sensing Scene Classification. IEEE Trans Geosci Remote Sensing 2022;60:1–15. https://doi.org/10.1109/TGRS.2021.3126770. DOI: https://doi.org/10.1109/TGRS.2021.3126770

Song H. A Consistent Mistake in Remote Sensing Images’ Classification Literature. Intelligent Automation & Soft Computing 2023;37:1381–98. https://doi.org/10.32604/iasc.2023.039315. DOI: https://doi.org/10.32604/iasc.2023.039315

Zhang J, Zhao H, Li J. TRS: Transformers for Remote Sensing Scene Classification. Remote Sensing 2021;13:4143. https://doi.org/10.3390/rs13204143. DOI: https://doi.org/10.3390/rs13204143

Lv P, Wu W, Zhong Y, Du F, Zhang L. SCViT: A Spatial-Channel Feature Preserving Vision Transformer for Remote Sensing Image Scene Classification. IEEE Trans Geosci Remote Sensing 2022;60:1–12. https://doi.org/10.1109/TGRS.2022.3157671. DOI: https://doi.org/10.1109/TGRS.2022.3157671

Wang D, Zhang J, Du B, Xia G-S, Tao D. An Empirical Study of Remote Sensing Pretraining. IEEE Trans Geosci Remote Sensing 2023;61:1–20. https://doi.org/10.1109/TGRS.2022.3176603. DOI: https://doi.org/10.1109/TGRS.2022.3176603

Li B, Guo Y, Yang J, Wang L, Wang Y, An W. Gated Recurrent Multiattention Network for VHR Remote Sensing Image Classification. IEEE Trans Geosci Remote Sensing 2022;60:1–13. https://doi.org/10.1109/TGRS.2021.3093914. DOI: https://doi.org/10.1109/TGRS.2021.3093914

Shen J, Yu T, Yang H, Wang R, Wang Q. An Attention Cascade Global–Local Network for Remote Sensing Scene Classification. Remote Sensing 2022;14:2042. https://doi.org/10.3390/rs14092042. DOI: https://doi.org/10.3390/rs14092042

Tang X, Ma Q, Zhang X, Liu F, Ma J, Jiao L. Attention Consistent Network for Remote Sensing Scene Classification. IEEE J Sel Top Appl Earth Observations Remote Sensing 2021;14:2030–45. https://doi.org/10.1109/JSTARS.2021.3051569. DOI: https://doi.org/10.1109/JSTARS.2021.3051569

Wang W, Chen Y, Ghamisi P. Transferring CNN With Adaptive Learning for Remote Sensing Scene Classification. IEEE Trans Geosci Remote Sensing 2022;60:1–18. https://doi.org/10.1109/TGRS.2022.3190934. DOI: https://doi.org/10.1109/TGRS.2022.3190934

Xu K, Huang H, Deng P. Remote Sensing Image Scene Classification Based on Global–Local Dual-Branch Structure Model. IEEE Geosci Remote Sensing Lett 2022;19:1–5. https://doi.org/10.1109/LGRS.2021.3075712. DOI: https://doi.org/10.1109/LGRS.2021.3075712

Deng P, Xu K, Huang H. When CNNs Meet Vision Transformer: A Joint Framework for Remote Sensing Scene Classification. IEEE Geosci Remote Sensing Lett 2022;19:1–5. https://doi.org/10.1109/LGRS.2021.3109061. DOI: https://doi.org/10.1109/LGRS.2021.3109061

Zhao M, Meng Q, Zhang L, Hu X, Bruzzone L. Local and Long-Range Collaborative Learning for Remote Sensing Scene Classification. IEEE Trans Geosci Remote Sensing 2023;61:1–15. https://doi.org/10.1109/TGRS.2023.3265346. DOI: https://doi.org/10.1109/TGRS.2023.3265346

Ma J, Li M, Tang X, Zhang X, Liu F, Jiao L. Homo–Heterogenous Transformer Learning Framework for RS Scene Classification. IEEE J Sel Top Appl Earth Observations Remote Sensing 2022;15:2223–39. https://doi.org/10.1109/JSTARS.2022.3155665. DOI: https://doi.org/10.1109/JSTARS.2022.3155665

Wang G, Chen H, Chen L, Zhuang Y, Zhang S, Zhang T, et al. P2FEViT: Plug-and-Play CNN Feature Embedded Hybrid Vision Transformer for Remote Sensing Image Classification. Remote Sensing 2023;15:1773. https://doi.org/10.3390/rs15071773. DOI: https://doi.org/10.3390/rs15071773

Cheng X, Lei H. Remote Sensing Scene Image Classification Based on mmsCNN–HMM with Stacking Ensemble Model. Remote Sensing 2022;14:4423. https://doi.org/10.3390/rs14174423. DOI: https://doi.org/10.3390/rs14174423

Sesmero MP, Ledezma AI, Sanchis A. Generating ensembles of heterogeneous classifiers using Stacked Generalization. WIREs Data Min & Knowl 2015;5:21–34. https://doi.org/10.1002/widm.1143. DOI: https://doi.org/10.1002/widm.1143

Yun S, Han D, Chun S, Oh SJ, Yoo Y, Choe J. CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, p. 6022–31. https://doi.org/10.1109/ICCV.2019.00612. DOI: https://doi.org/10.1109/ICCV.2019.00612

Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Int J Comput Vis 2020;128:336–59. https://doi.org/10.1007/s11263-019-01228-7. DOI: https://doi.org/10.1007/s11263-019-01228-7

Maaten L van der, Hinton G. Visualizing Data using t-SNE. Journal of Machine Learning Research 2008;9:2579–605. Available at: http://jmlr.org/papers/v9/vandermaaten08a.html. Accessed on: May 01, 2024.

Radosavovic I, Kosaraju RP, Girshick R, He K, Dollar P. Designing Network Design Spaces. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA: IEEE; 2020, p. 10425–33. https://doi.org/10.1109/CVPR42600.2020.01044. DOI: https://doi.org/10.1109/CVPR42600.2020.01044