Enhancing the Potential of Machine Learning for Immersive Emotion Recognition in Virtual Environment

Abinaya M; Vadivu G

doi:10.4108/eetsis.5036

Authors

Abinaya M SRM Institute of Science and Technology
Vadivu G SRM Institute of Science and Technology

DOI:

https://doi.org/10.4108/eetsis.5036

Keywords:

Emotion Recognition, Immersive Technology, Machine Learning, Virtual Environments

Abstract

Emotion recognition is an immense challenge for immersive technology. In order to detect the emotions of the user, we use machine learning methods and techniques to use the potential of the Virtual Environment and to improve the user Experience. Emotion recognition plays an important role in developing realistic and emotionally immersive experiences in augmented reality (AR) and virtual reality (VR) settings by instantly adjusting interactions, content, and visuals based on the accurate detection and interpretation of users’ emotions. Immersive systems can enhance user experience through various machine learning algorithms and methods used for emotion recognition, which are examined in this article. Upon novel idea, challenges and potential applications of incorporating emotion recognition in immersive virtual environments with Machine Learning (ML) Techniques and the benefits of tailoring powerful immersive experiences with ML methods were highlighted, and also the study discusses potential advancements in identifying the user’s emotion recognition in the future by modeling an Architecture, as well as how the ML techniques were enhanced for virtual environment is discussed.

References

Zhan, T., Yin, K., Xiong, J., He, Z., Wu, S-T. Augmented Reality and Virtual Reality Displays: Perspectives and Challenges. iScience. 2020; Vol. 23, No. 8: pp. 101397.

Shu, L., Marín-Morales, J., Llinares, C., Guixeres, J., Alcaniz, M. A Review of Emotion Recognition Using Physiological Signals. Sensors. 2018; Vol. 18, No. 7: pp. 2074

Marín-Morales, J., Llinares, C., Guixeres, J., Alcaniz, M. Affective Computing in Virtual Reality: Emotion Recognition from Brain and Heartbeat Dynamics Using Wearable Sensors. Scientific Reports. 2018; Vol. 8, No. 1: pp. 23.

Balan, O., Moise, G., Petrescu, L., Moldoveanu, A., Leordeanu, M., Moldoveanu, F. Emotion Classification Based on Biophysical Signals and Machine Learning Techniques. Symmetry. 2019; Vol. 12, No. 1: pp. 21.

Mehta, D., Siddiqui, M. F. H., Javaid, A. Y. Recognition of Emotion Intensities Using Machine Learning Algorithms: A Comparative Study. Sensors. 2019; Vol. 19, No. 8: pp. 1897.

Marín-Morales, J., Llinares, C., Guixeres, J., Alcaniz, M. Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing. Sensors. 2020; Vol. 20, No. 18: pp. 5163.

Kamruzzaman, M., Hossain, M., Imran, Md. R. I., Bakchy, S. C. A Comparative Analysis of Sentiment Classification Based on Deep and Traditional Ensemble Machine Learning Models. IEEE Xplore. 2021; pp. 631–638.

Tao, Y., Donato, G., Bartlett, M. S, Hager, J. C., Ekman, P., Sejnowski, T. J. Confidence Estimation Using Machine Learning in Immersive Learning Environments. IEEE Xplore. 2020; pp. 145–150.

Alsaleh, S., Tepljakov, A., Tamre, M., Petlenkov, E. Towards Artificial Intelligence Driven Immersive Environments in Virtual Reality for Industrial Applications. IEEE Xplore. 2021; pp. 172–182.

Maxim, B. R., Patel, N. V., Martineau, N. D., Schwartz, M. Work in progress - learning via gaming: An immersive environment for teaching kids handwriting. IEEE Xplore. 2007; Vol. 11, No. 2: pp. 145–150.

Gombolay, M. C., Jensen, R. E., Son, S-H. Machine Learning Techniques for Analyzing Training Behavior in Serious Gaming.IEEE Transactions on Games. 2019; Vol. 11, No. 2: pp. 109–120.

Kasun, L. L. C., Yang, Y., Huang, G-B., Zhang, Z. Dimension Reduction With Extreme Learning Machine. IEEE Transactions on Image Processing. 2016; Vol. 25, No. 8: pp. 3906–3918.

Kuś, S., Szmurło, R. CNN-based character recognition for a contextless text input system in immersive VR. IEEE Xplore. 2021; pp. 191–197.

Zhu, K., Marín-Morales, J., Llinares, C., Guixeres, J., Alcaniz, M. Interface and Experience Design with AI for VR/AR (DAIVAR'18) and AI/ML for Immersive Simulations (AMISIM’18). IEEE Xplore. 2018; pp. 187–194.

Pfeiffer, J., Pfeiffer, T., Meißner, M., Weiß, E. Eye-Tracking-Based Classification of Information Search Behavior Using Machine Learning: Evidence from Experiments in Physical Shops and Virtual Reality Shopping Environments. Information Systems Research. 2020; Vol. 31, No. 3: pp. 675–691.

Jingyu, D., Hongyu, D., Jianxing, W., Xuanzi, W. Children’s Clothing Virtual Simulation Immersive Design and Show Based on Machine Learning. Mobile Information Systems. 2021; Vol. 20: pp. 1–9.

Donato, G., Bartlett, M. S, Hager, J. C., Ekman, P., Sejnowski, T. J. "Classifying Facial Actions." IEEE Transactions on Pattern Analysis and Machine Intelligence. 1999; Vol. 21, No. 10: pp. 974.

Gavrilescu, Mihai. Proposed architecture of a fully integrated modular neural network-based automatic facial emotion recognition system based on Facial Action Coding System. 2014; Vol. 11, No.1: pp. 145–150.

Kotsia, I., Pitas, I. Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines. IEEE Transactions on Image Processing. 2007; Vol. 16, No. 1: pp. 172–187.

Wu, Chung-Hsien, Liang, Wei-Bin. Emotion Recognition of Affective Speech Based on Multiple Classifiers Using Acoustic-Prosodic Information and Semantic Labels.IEEE Transactions on Affective Computing. 2011; Vol. 2, No. 1: pp. 10–21.

Wu, C-H., Lin, J-C., Wei, W-L. Two-Level Hierarchical Alignment for Semi-Coupled HMM-Based Audiovisual Emotion Recognition with Temporal Course. IEEE Transactions on Multimedia. 2013; Vol. 15, No. 8: pp. 1880–1895.

Li, Y., Zeng, J., Shan, S., Chen, X. Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism. IEEE Transactions on Image Processing. 2019; Vol. 28, No. 5: pp. 2439–2450.

Devanne, M., Wannous, H., Berretti, S., Pala, P., Daoudi, M., Del Bimbo, A. 3-D Human Action Recognition by Shape Analysis of Motion Trajectories on Riemannian Manifold.IEEE Transactions on Cybernetics. 2015; Vol. 45, No. 7: pp. 1340–1352.

Ahmad, Z., Khan, N. A Survey on Physiological Signal-Based Emotion Recognition. Bioengineering. 2022; Vol. 9, No. 11: pp. 681-688.

Guo, G., Li, S. Z., Chan, K. L. Support Vector Machines for Face Recognition. Image and Vision Computing. 2001; Vol. 19, No. 9–10, pp: 631–638.

Bendjoudi, I., Vanderhaegen, F., Hamad, D., Dornaika, F. Multi-label, multi-task CNN approach for context-based emotion recognition. Information Fusion. 2020; pp. 751–756.

Huang, C-W., Narayanan, S. S. Deep Convolutional Recurrent Neural Network with Attention Mechanism for Robust Speech Emotion Recognition. IEEE Xplore. 2017; pp. 691–698.

Ma, F., Li, Y., Ni, S., Huang, S-L., Zhang, L. Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN. Applied Sciences. 2022; Vol. 12, No. 1: pp. 527.

Yang, J. Ensemble deep learning with HuBERT for speech emotion recognition. 2023 IEEE 17th International Conference on Semantic Computing (ICSC). 2023; pp. 175–183.

Jayachitra, S., Prasanth, A., Rafi, S. M., Beevi, S. Z. Hierarchical-Based Binary Moth Flame Optimization for Feature Extraction in Biomedical Application.Proceedings in 4th International Conference on Machine Learning, Image Processing, Network Security and Data Sciences. 2023; pp. 27-38.

Anbumozhi, A., Shanthini, A. Leaf Diseases Identification and Classification of Self-Collected Dataset on Groundnut Crop using Progressive Convolutional Neural Network (PGCNN). International Journal of Advanced Computer Science and Applications. 2023; Vol. 14, No. 2:pp.128.

Jayachitra, S., Prasanth, A., Hariprasath, S., Benazir Begam, R., Madiajagan, M. AI Enabled Internet of Medical Things in Smart Healthcare. In AI Models for Blockchain-Based Intelligent Networks in IoT Systems: Concepts, Methodologies, Tools, and Applications. 2023; pp. 141-161.

Kavitha, M., Roobini, S., Prasanth, A., Sujaritha, M. Systematic View and Impact of Artificial Intelligence in Smart Healthcare Systems, Principles, Challenges and Applications. In Machine Learning and Artificial Intelligence in Healthcare Systems. 2023; pp. 25-56.