Gesture Recognition Based on Deep Learning: A Review
DOI:
https://doi.org/10.4108/eetel.5191Keywords:
Gesture recognition, Deep learning, CNNs, LSTMAbstract
Gesture recognition is an important and inevitable technology in modern times, its appearance and improvement greatly improve the convenience of people's lives, but also enrich people's lives. It has a wide range of applications in various fields. In daily life, it can carry out human-computer interaction and the use of smart home. In terms of medical treatment, it can help patients to recover and assist doctors to carry out experiments. In terms of entertainment, it allows users to interact with the game in an immersive manner. This paper chooses three technologies that deep learning plays a more prominent role in gesture recognition, namely CNNs, LSTM and transfer learning based on deep learning. They each have their own advantages and disadvantages. Because of the different principles of use, different techniques have different roles, such as CNNs can carry out feature extraction, LSTM can deal with long time series, transfer learning can transfer what is learned from another task to this task. Select different practical technologies according to different application scenarios, and make improvements in real time in practical applications. Gesture recognition based on deep learning has the advantages of good accuracy, robustness and real-time implementation, but it also bears the disadvantages of huge economic and time costs and high hardware requirements. Despite some challenges, researchers continue to optimize and improve the technology, and believe that in the future, gesture recognition technology will be more mature and valuable.
References
W. H. Yeo, et al., "Machine-learned wearable sensors forreal-time hand-motion recognition: toward practicalapplications," National science review, vol. 11, pp. nwad298-nwad298, 2024.
N. Fadel and E. I. A. Kareem, "Detecting HandGestures Using Machine Learning Techniques," Ingénieriedes Systèmes d’Information, vol. 27, 2022.
D. Fu, "Human-computer Interaction Gesture ControlRecognition for High-performance Embedded AI Computing Platform," Advances in Computer and Communication, vol.4, 2023.
T. Min, "Gesture Detection and Recognition Based onPyramid Frequency Feature Fusion Module and MultiscaleAttention in Human-Computer Interaction," MathematicalProblems in Engineering, vol. 2021, 2021.
L. Zhihan, P. Fabio, D. Qi, L. Jaime, and S. Houbing,"Deep Learning for Intelligent Human–ComputerInteraction," Applied Sciences, vol. 12, pp. 11457-11457,2022.
J. Lesong, Z. Xiaozhou, and X. Chengqi, "Non-trajectory-based gesture recognition in human-computerinteraction based on hand skeleton data," Multimedia Toolsand Applications, vol. 81, pp. 20509-20539, 2022.
A. Mohammad, A. Abdullah, A. Gharbi, A. T. S, A.Saud, A. Dhahi, et al., "Human-computer Interaction Gesture Control Recognition for High-performance Embedded AIComputing Platform," Advances in Computer andCommunication, vol. 4, 2023.
N. Prasanth, K. Shrivastava, A. Sharma, A. Basu, R. A.Sinha, and S. P. Raja, "Gesture-based mouse control systembased on MPU6050 and Kalman filter technique,"International Journal of Intelligent Systems Technologiesand Applications, vol. 21, pp. 56-71, 2023.
C. Alexandre, M. Luciano, R. Miriam, and B. Gilberto,"Musical Control Gestures in Mobile Handheld Devices:Design Guidelines Informed by Daily User Experience,"Multimodal Technologies and Interaction, vol. 5, pp. 32-32,2021.
Y. Chen, Q. Li, C. Huang, C. Ye, Y. Li, and R. Lu,"Research on the Virtuality-Reality Interaction Based onDynamic Gesture Recognition in Augmented RealitySystem," Advanced Science Letters, vol. 7 2012, pp. 468-472(5), 2012.
F. Zhao, W. Jinlong, and N. Taile, "Research andApplication of Multifeature Gesture Recognition in Human-Computer Interaction Based on Virtual Reality Technology,"WIRELESS COMMUNICATIONS & MOBILECOMPUTING, vol. 2021, 2021.
L. Jiang, X. Yu, and L. Wang, "A brief analysis ofgesture recognition in VR," SID Symposium Digest ofTechnical Papers, vol. 51, pp. 190-195, 2020.
H. Asim, M. Sebastian, and P. Andrew, "How good arevirtual hands? Influences of input modality on motor tasks invirtual reality," Journal of Environmental Psychology, vol.92, 2023.
P. Anastasia, G. Geoffrey, B. Domna, and W. Sebastian, "Using virtual reality to assess gesture performance deficitsin schizophrenia patients," Frontiers in psychiatry, vol. 14,pp. 1191601-1191601, 2023.
W. Peng, W. Yue, B. Mark, Y. Huizhen, X. Peng, andL.Yanhong, "BeHere: a VR/SAR remote collaborationsystem based on virtual replicas sharing gesture and avatar ina procedural task," Virtual reality, vol. 27, pp. 21-22, 2023.
B. Ganguly, P. Vishwakarma, Rahul, and S. Biswas,"Gesture based Virtual Reality Implementation for CarRacing Game employing an Improved Neural Network,"International Journal of Engineering Research and, vol. V8,2019.
L. Wen, W. Shunxin, L. Simou, Z. Xiyang, Z. Xiaobo,Q.Hao, et al., "Gesture Recognition System Using ReducedGraphene Oxide-Enhanced Hydrogel Strain Sensors forRehabilitation Training," ACS applied materials & interfaces,vol. 15, 2023.
G. Kai, O. Mostafa, L. Jingxin, S. A. Maged, Y. Hongbo, and E. Mahmoud, "Empowering Hand Rehabilitation withAI-Powered Gesture Recognition: A Study of an sEMG-Based System," Bioengineering (Basel, Switzerland), vol. 10,2023.
D. M. Doriana, D. S. Elisa, and V. Giovanni,"Embodying Language through Gestures: Residuals ofMotor Memories Modulate Motor Cortex Excitability during Abstract Words Comprehension," Sensors, vol. 22, pp. 7734-7734, 2022.
P.Vogiatzidakis and P.Koutsabasis, "Mid-air gesturecontrol of multiple home devices in spatial augmented realityprototype," Multimodal Technologies and Interaction, vol. 4,pp. 1-22, 2020.
Z. Sheng, S. Zening, Z. Wenjie, S. Xu, X. Linghui, J.Bo, et al., "An EMG-based wearable multifunctional Eye-control glass to control home appliances and communicateby voluntary blinks," Biomedical Signal Processing andControl, vol. 86, 2023.
J. Dai, "Gesture Recognition Based Smart HomeControl System," Journal of Electronic Research andApplication, vol. 3, pp. 7-8, 2019.
Y.-D. Zhang, "Advances in Multimodality Data Fusionin Neuroimaging," Information Fusion, vol. 76, pp. 87-88,2021.
S. Wang, "Advances in data preprocessing forbiomedical data fusion: an overview of the methods,challenges, and prospects," Information Fusion, vol. 76, pp.376-421, 2021.
J. Wang, "A Review of Deep Learning on MedicalImage Analysis," Mobile Networks & Applications, vol. 26,pp. 351-380, 2021.
S. Arooj, S. Altaf, S. Ahmad, H. Mahmoud, and A. S.N.Mohamed, "Enhancing sign language recognition usingCNN and SIFT: A case study on Pakistan sign language,"Journal of King Saud University - Computer and InformationSciences, vol. 36, pp. 101934-, 2024.
A. H. Victoria and G.Maragatham, "Gesturerecognition of radar micro doppler signatures using separableconvolutional neural networks," Materials Today:Proceedings, vol. 80, pp. 1961-1964, 2023.
B. H. Lee, T. Kim, and D. H. Oh, "3D Virtual RealityGame with Deep Learning-based Hand GestureRecognition," Journal of the Korea Computer GraphicsSociety, vol. 24, pp. 41-48, 2018.
C. C. d. Santos, J. L. A. Samatelo, and R. F. Vassallo,"Dynamic gesture recognition by using CNNs and star RGB:A temporal information condensation," Neurocomputing, vol.400, pp. 238-254, 2020.
A. A. Barbhuiya, R. K. Karsh, and R. Jain, "CNN basedfeature extraction and classification for sign language,"Multimedia Tools and Applications, vol. 80, pp. 1-19, 2020.
J. P. Sahoo, S. P. Sahoo, S. Ari, and S. K. Patra, "RBI-2RCNN: Residual Block Intensity Feature using a Two-stageResidual Convolutional Neural Network for Static HandGesture Recognition," Signal, Image and Video Processing,vol. 16, pp. 1-9, 2022.
A. O. Tolulope, A. Adewale, K. Faiq, and H. S. Rafay,"FM-ModComp: Feature Map Modification and Hardware–Software Co-Comparison for secure hardware accelerator-based CNN inference," Microprocessors and Microsystems,vol. 100, 2023.
S.-H. Wang and M. A. Khan, "VISPNN: VGG-inspiredstochastic pooling neural network," Computers, Materials &Continua, vol. 70, pp. 3081-3097, 2022.
S.-H. Wang, K. M. Attique, and G. Vishnuvarthanan,"Deep rank-based average pooling network for COVID-19recognition," Computers, Materials, & Continua, vol. 70, pp.2797-2813, 2022.
X. Guo, "A Survey on Machine Learning in COVID-19Diagnosis," Computer Modeling in Engineering & Sciences,vol. 130, pp. 23-71, 2022.
S. Ziyi, L. Handong, Q. Jinwu, Z. Zhen, and Z. Lunwei,"Hand Gesture Recognition Based on sEMG Signal andConvolutional Neural Network," International Journal ofPattern Recognition and Artificial Intelligence, vol. 35, 2021.
N. Cui, "Applying Gradient Descent in ConvolutionalNeural Networks," Journal of Physics: Conference Series,vol. 1004, pp. 012027-012027, 2018.
O.S.Kayhan and J. C. v. Gemert, "On translationinvariance in CNNs: Convolutional layers can exploitabsolute spatial location," Proceedings of the IEEEComputer Society Conference on Computer Vision andPattern Recognition, pp. 14262-14273, 2020.
A. B. Ibrahimm, A. Hira, M. N. Al, A. Abdulwahab, A.Abdullah, A. S. S, et al., "Smart Home Automation-BasedHand Gesture Recognition Using Feature Fusion andRecurrent Neural Network," Sensors (Basel, Switzerland),vol. 23, 2023.
T. Alejandro, J. Juan, C. T. Juan, P. Alejandro, L.Alexandro, and C. R. Alexandre, "LSTM Recurrent NeuralNetwork for Hand Gesture Recognition Using EMGSignals," Applied Sciences, vol. 12, pp. 9700-9700, 2022.
R. Zahra, M. Malik, and D. Frédéric, "Autonomousgesture recognition using multi-layer LSTM networks andlaban movement analysis," International Journal ofKnowledge-based and Intelligent Engineering Systems, vol.26, pp. 289-297, 2023.
M. U. Rehman, F. Ahmed, M. A. Khan, U. Tariq, F. A.Alfouzan, N. M. Alzahrani, et al., "Dynamic Hand GestureRecognition Using 3D-CNN and LSTM Networks,"Computers, Materials & Continua, vol. 70, pp. 4675-4690,2022.
H. Giannis, R. Gerasimos, and A. Ioannis, "A ModifiedLong Short-Term Memory Cell," International journal ofneural systems, vol. 33, pp. 2350039-2350039, 2023.
A. N. Moustafa and W. Gomaa, "Gate and commonpathway detection in crowd scenes and anomaly detectionusing motion units and LSTM predictive models,"Multimedia Tools and Applications, vol. 79, pp. 1-40, 2020.
W. Weina, S. Jiapeng, and J. Huxidan, "Fuzzyinference-based LSTM for long-term time series prediction,"Scientific Reports, vol. 13, pp. 20359-20359, 2023.
F. Liu, L. Zhang, and Z. Jin, "Modeling programshierarchically with stack-augmented LSTM," The Journal ofSystems & Software, vol. 164, pp. 110547-110547, 2020.
Z. Zhen, L. Shilong, W. Yanyu, S. Wei, and Z. Yuhui,"Online cross session electromyographic hand gesturerecognition using deep learning and transfer learning,"Engineering Applications of Artificial Intelligence, vol. 127,2024.
G. Alfonso, M. Delfina, Z. Rocco, C. Carmine, and L.Nicola, "Touchscreen gestures as images. A transfer learningapproach for soft biometric traits recognition," ExpertSystems With Applications, vol. 219, 2023.
K. Peiqi, L. Jinxuan, F. Bingfei, J. Shuo, and B. S. Peter,"Wrist-worn Hand Gesture Recognition while Walking viaTransfer Learning," IEEE journal of biomedical and healthinformatics, vol. PP, 2021.
Z. Shenyilang, F. Yinfeng, W. Jiacheng, J. Guozhang,and L. Gongfa, "Transfer Learning Enhanced Cross-Subject Hand Gesture Recognition with sEMG," Journal of Medical and Biological Engineering, vol. 43, pp. 672-688, 2023.
R. Yu and G. Guoying, "Deep transfer learning-basedadaptive gesture recognition of a soft e-skin patch withreduced training data and time," Sensors and Actuators: A.Physical, vol. 363, 2023.
Q. Bu, G. Yang, X. Ming, T. Zhang, J. Feng, and J.Zhang, "Deep transfer learning for gesture recognition withWiFi signals," Personal and Ubiquitous Computing, vol. 26,pp. 1-12, 2020.
W. Fei, H. Ronglin, and J. Ying, "Research on TransferLearning of Vision-based Gesture Recognition,"International Journal of Automation and Computing, vol. 18,pp. 1-10, 2021.
A. R.Patil and S.Subbaraman, "Performance analysis ofstatic hand gesture recognition approaches using artificialneural network, support vector machine and two streambased transfer learning approach," International Journal ofInformation Technology, vol. 14, pp. 1-12, 2021.
K. Yang, M. Kim, Y. Jung, and S. Lee, "Hand GestureRecognition Using FSK Radar Sensors," Sensors, vol. 24,2024.
Z. Yinger, W. Zhouyi, L. Peiying, P. Yang, W. Yuting,Z.Liufang, et al., "Hand gestures recognition in videos takenwith a lensless camera," Optics express, vol. 30, pp. 39520-39533, 2022.
R. Dmitry, I. Denis, and R. Elena, "Audio-VisualSpeech and Gesture Recognition by Sensors of MobileDevices," Sensors, vol. 23, pp. 2284-2284, 2023.
H. Pingao, W. Hui, W. Yuan, G. Yanjuan, Y. Wenlong,G.Chao, et al., "In-Situ Measuring sEMG and Muscle ShapeChange with a Flexible and Stretchable Hybrid Sensor forHand Gesture Recognition," IEEE transactions on neuralsystems and rehabilitation engineering : a publication of theIEEE Engineering in Medicine and Biology Society, vol. PP,2022.
E. N. Rubén and E. B. Marco, "Analysis and Evaluationof Feature Selection and Feature Extraction Methods,"International Journal of Computational Intelligence Systems,vol. 16, 2023.
H. Ashraf, A. Waris, S. O. Gilani, U. Shafiq, J. Iqbal, E.N.Kamavuako, et al., "Optimizing the performance ofconvolutional neural network for enhanced gesturerecognition using sEMG," Scientific reports, vol. 14, pp.2020-2020, 2024.
W. S. Josiah, T. Shiva, W. Richard, M. Yiorgos, and T.Murat, "Improved Static Hand Gesture Classification onDeep Convolutional Neural Networks Using Novel SterileTraining Technique," IEEE ACCESS, vol. 9, pp. 10893-10902, 2021.
Y. D. Zhang and S. C. Satapathy, "Fruit categoryclassification by fractional Fourier entropy with rotationangle vector grid and stacked sparse autoencoder," ExpertSystems, vol. 39, 2022.
L. Yang, "EDNC: ensemble deep neural network forCovid-19 recognition," Tomography, vol. 8, pp. 869-890,2022.
S.Rajko, G.Qian, T.Ingalls, and J.James, "Real-timegesture recognition with minimal training requirements andon-line learning," Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition,2007.
R. Georg, S. Moritz, F. Tim, and B. Luca, "7 [formulaomitted]J/inference end-to-end gesture recognition fromdynamic vision sensor data using ternarized hybridconvolutional neural networks," Future GenerationComputer Systems, vol. 149, pp. 717-731, 2023.
B. Anton, H. Andreas, O. Emilia, K. Kimmo, and P. Kai,"Explaining any black box model using real data ,"Frontiers in Computer Science, vol. 5, 2023.
H. Zhou, D. Wang, Y. Yu, and Z. Zhang, "ResearchProgress of Human–Computer Interaction Technology Based on Gesture Recognition," Electronics, vol. 12, 2023.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 EAI Endorsed Transactions on e-Learning
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
This is an open-access article distributed under the terms of the Creative Commons Attribution CC BY 3.0 license, which permits unlimited use, distribution, and reproduction in any medium so long as the original work is properly cited.
Funding data
-
State Key Laboratory of Millimeter Waves
Grant numbers K202218
