Facial Emotion Recognition by CNN Combined Ensemble Model

Abdurahmon Kurbanov

doi:10.4108/airo.9720

Authors

Abdurahmon Kurbanov Jizzakh branch of the National University of Uzbekistan named after Mirzo Ulugbek

DOI:

https://doi.org/10.4108/airo.9720

Keywords:

CNN, ResNet, VGGNet, DenseNet, transfer learning, ensemble model, ESPCN, hash_compare

Abstract

Studying emotions can provide important information about a person's mental state. According to research, more than 50% of a person's current emotions can be identified from the human face. In this research, we propose an ensemble model for emotion recognition from facial images, which is obtained by combining the results obtained by retraining previously trained convolutional neural networks on a new and high-quality FaceEmocDS dataset. The methodological advantage of the ensemble model we propose is that the combination of VGG19, ResNet50, and DenseNet121 models allows us to take advantage of the strengths of each architecture: the ability to extract detailed features of VGG19, the stable learning process of ResNet50 through residual connections, and the efficiency of feature reuse of DenseNet121. This approach improves the results of individual models, increasing the accuracy to 85.66% . The FaceEmocDS dataset consists of 72,412 images and includes eight emotion classes, including a unique “contempt” class. The results show significant superiority when compared to other datasets (FER2013, AffectNet, CK+) and studies.

Downloads

Download data is not yet available.

References

[1] G. Perveen, S. F. Ali, J. Ahmad, S. Shahab, M. Adnan and Mohd Anjum, "Multi-Stream Deep Convolution Neural Network With Ensemble Learning for Facial Micro-Expression Recognition," IEEE Access, vol. 11, pp. 118474-118489, 2023.

[2] M. Almubarak and F. Alsulaiman, "An Ensemble Learning Approach for Facial Emotion Recognition Based on Deep Learning Techniques," Electronics, vol. 14, p. 3415, 2025.

[3] G. Zhou, Y. Xie and W. Tian., "Multi loss-based feature fusion and top two voting ensemble decision strategy for facial expression recognition in the wild.," 2023. [Online]. Available: arXiv:2311.03478.

[4] E. Younis, S. Zaki, E. Kanjo and E. Houssein, "Evaluating Ensemble Learning Methods for Multi-Modal Emotion Recognition Using Sensor Data Fusion.," Sensor, vol. 22, p. 5611, 2022.

[5] A. Renda, M. Barsacchi, A. Bechini and F. Marcelloni, "Comparing ensemble strategies for deep learning: An application to facial expression recognition," Expert Systems with Applications, vol. 136, pp. 1-11, 2019.

[6] D. Pandit and S. Jadhav, "Fusion of ensemble technique with CNN model to equilibrate prediction of face emotions in real-time," Traitement du Signal, vol. 42, no. 1, pp. 519-530, 2024.

[7] M. Ganaie, M. Hu, A. Malik, M. Tanveer and P.N. Suganthan, "Ensemble deep learning: A review," Engineering Applications of Artificial Intelligence, vol. 115, p. 105151, 2022.

[8] E. A. Sağbaş, A. Uğur and S. Korukoğlu, "Performance Evaluation of Ensemble Learning Methods for Facial Expression Recognition,," in Innovations in Intelligent Systems and Applications Conference (ASYU), Izmir, Turkey, 2019.

[9] J. Y. Choi and B. Lee, "Combining Deep Convolutional Neural Networks With Stochastic Ensemble Weight Optimization for Facial Expression Recognition in the Wild," IEEE Transactions on Multimedia, vol. 25, pp. 100-111, 2023.

[10] A. M. Esfar-E-Alam, M. Hossain, M. Gomes, R. Islam and R. Raihana, "Multimodal Emotion Recognition Using Heterogeneous Ensemble Techniques,," in International Conference on Computer and Information Technology (ICCIT), Cox's Bazar, Bangladesh, 2022.

[11] W. Li, M. Luo, P. Zhang and W. Huang, "A Novel Multi-Feature Joint Learning Ensemble Framework for Multi-Label Facial Expression Recognition," IEEE Access, vol. 9, pp. 119766-119777, 2021.

[12] H. D. Nguyen, S. H. Kim, G. S. Lee, H. J. Yang, I. S. Na and S. H. Kim, "Facial Expression Recognition Using a Temporal Ensemble of Multi-Level Convolutional Neural Networks," IEEE Transactions on Affective Computing, vol. 13, no. 1, pp. 226-237, 2019.

[13] E. S. Smitha, S. Sendhilkumar and G. S. Mahalakshmi, "Ensemble Convolution Neural Network for Robust Video Emotion Recognition Using Deep Semantics," Scientific Programming, p. 21, 2023.

[14] M. Muhajir, K. Muchtar, M. Oktiana and A. Bintang, "Students’ emotion classification system through an ensemble approach," SINERGI, vol. 28, no. 2, pp. 413-424, 2024.

[15] S. Gupta, P. Kumar and RajKumar Tekchandani, "EDFA: Ensemble deep CNN for assessing student's cognitive state in adaptive online learning environments," International Journal of Cognitive Computing in Engineering, vol. 4, pp. 373-387, 2023.

[16] H. Dương, S. Yeom, G.-S. Lee, H.-J. Yang, I. Na and S. Kim, "Facial Emotion Recognition Using an Ensemble of Multi-Level Convolutional Neural Networks.," International Journal of Pattern Recognition and Artificial Intelligence., vol. 33, 2019.

[17] E. G. Moung, C. C. Wooi, M. M. Sufian, J. A. Dargham and J. Khoo, "A Robust Ensemble Approach to Face Expression Recognition and Image Sentiment Analysis.," in Internet of Things and Artificial Intelligence for Smart Environments., Singapore, Springer, 2024, p. 203–223.

[18] V. Shtino and M. Muca, "Improving Facial Expression Classification through Ensemble Deep Learning Models. Nanotechnology Perceptions," Nanotechnology and the Applications in Engineering and Emerging Technologies, vol. 20, 2024.

[19] J. X. Chang, C. P. Lee, K. M. Lim and J. Y. Lim, "Facial Expression Recognition with Machine Learning," in 11th International Conference on Information and Communication Technology (ICoICT), Malaysia, 2023.

[20] A. Das, M. S. Jalal, A. Bari and M. Huda, "Facial Emotion Recognition by Ensemble-DenseNet Networks," in IEEE 11th Region 10 Humanitarian Technology Conference (R10-HTC), Rajkot, India,, 2023.

[21] J. X. Yu, K. M. Lim and C. P. Lee, "MoVE-CNNs: Model aVeraging Ensemble of Convolutional Neural Networks for Facial Expression Recognition.," IAENG International Journal of Computer Science,, vol. 48, no. 3, pp. 1-5, 2021.

[22] R. Lawpanom, W. Songpan and J. Kaewyotha, "Advancing Facial Expression Recognition in Online Learning Education Using a Homogeneous Ensemble Convolutional Neural Network Approach.," Applied Sciences, vol. 14, no. 3, p. 1156, 2024.

[23] M. Ashraf, M. Zaman and M. Ahmed, "An intelligent prediction system for educational data mining based on ensemble and filtering approaches.," Procedia Comput. Sci., vol. 167, pp. 1471-1483, 2020.

[24] scikit-learn.org, Available: https://scikit-learn.org/stable/modules/ensemble.html.

[25] F. H and S. Weijs, "Entropy Ensemble Filter: A Modified Bootstrap Aggregating (Bagging) Procedure to Improve Efficiency in Ensemble Model Simulation.," Entropy, vol. 19, no. 10, p. 520, 2017.

[26] N. Mungoli, "Adaptive ensemble learning: Boosting model performance through intelligent feature fusion in deep neural networks.," arXiv preprint, 2023.

[27] J. Brownlee, "https://machinelearningmastery.com/stacking-ensemble-for-deep-learning-neural-networks/," machinelearningmastery.com, August 2020. [Online]. Available: https://machinelearningmastery.com/stacking-ensemble-for-deep-learning-neural-networks/ .

[28] J. Montgomery, F. Hollenbach, M. Ward and R. Alvarez, "Improving Predictions Using Ensemble Bayesian Model Averaging.," Political Analysis., pp. 271-291, 2012.

[29] K. Hara, D. Saitoh and H. Shouno, "Analysis of Dropout Learning Regarded as Ensemble Learning," in Artificial Neural Networks and Machine Learning – ICANN 2016. ICANN 2016, 2019.

[30] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar and I. Matthews, "The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression," in IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, San Francisco, 2010.

[31] A. Mollahosseini, B. Hasani and M. H. Mahoor, "Affectnet: A database for facial expression, valence, and arousal computing in the wild.," IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 18-31, 2017.

[32] Y. Khaireddin and Z. Chen, "Facial emotion recognition: State of the art performance on FER2013.," arxiv.org, 2021. [Online]. Available: https://arxiv.org/pdf/2105.03588.

[33] C. Dong, C. C. Loy and X. Tang, "Accelerating the Super-Resolution Convolutional Neural Network.," in In European conference on computer vision , 2016.

[34] H. Ruan, Z. Tan, L. Chen, W. Wan and J. Cao, "Efficient sub-pixel convolutional neural network for terahertz image super-resolution.," Optics letters, vol. 47, no. 12, p. 3115–3118., 2022.

[35] M. N. Kmuar and S. G. Winster, "Emotion identification in human faces through ensemble of deep learning models," Journal of Intelligent & Fuzzy Systems, vol. 45, no. 6, pp. 9729-9752, 2023.

[36] M. J. Lyons, "Excavating AI" re-excavated: debunking a fallacious account of the JAFFE dataset," arXiv, 2021. [Online]. Available: https://arxiv.org/pdf/2107.13998.

[37] J. Brownlee, "Stacking Ensemble for Deep Learning Neural Networks in Python," 28 07 2020. [Online]. Available: https://machinelearningmastery.com/stacking-ensemble-for-deep-learning-neural-networks/.