Manifesto of Deep Learning Architecture for Aspect Level Sentiment Analysis to extract customer criticism

N Kushwaha; B Singh; S Agrawal

doi:10.4108/eetsis.5698

Authors

N Kushwaha Indian Institute of Information Technology, Ranchi
B Singh Indian Institute of Information Technology, Ranchi
S Agrawal Indian Institute of Information Technology, Ranchi

DOI:

https://doi.org/10.4108/eetsis.5698

Keywords:

Sentiment analysis, Decision Making, GRU, Airline Data, Social Media, Deep Learning Architecture, Glove, Capsule Network, BERT

Abstract

Sentiment analysis, a critical task in natural language processing, aims to automatically identify and classify the sentiment expressed in textual data. Aspect-level sentiment analysis focuses on determining sentiment at a more granular level, targeting specific aspects or features within a piece of text. In this paper, we explore various techniques for sentiment analysis, including traditional machine learning approaches and state-of-the-art deep learning models. Additionally, deep learning techniques has been utilized to identifying and extracting specific aspects from text, addressing aspect-level ambiguity, and capturing nuanced sentiments for each aspect. These datasets are valuable for conducting aspect-level sentiment analysis. In this article, we explore a language model based on pre-trained deep neural networks. This model can analyze sequences of text to classify sentiments as positive, negative, or neutral without explicit human labeling. To evaluate these models, data from Twitter's US airlines sentiment database was utilized. Experiments on this dataset reveal that the BERT, RoBERTA and DistilBERT model outperforms than the ML based model in accuracy and is more efficient in terms of training time. Notably, our findings showcase significant advancements over previous state-of-the-art methods that rely on supervised feature learning, bridging existing gaps in sentiment analysis methodologies. Our findings shed light on the advancements and challenges in sentiment analysis, offering insights for future research directions and practical applications in areas such as customer feedback analysis, social media monitoring, and opinion mining.

References

Erevelles S., Fukawa N., and Swayne, L. Big data consumer analytics and the transformation of marketing. Journal of Business Research, vol. 69, no. 2, pp. 897-904, (2016).

Malik K. and Malik, M.: The Prediction of Stock Market Trends Using the Hybrid Model SVM-ICA-GA. Springer, (2020).

Ruz, G. A., Henríquez P. A., and Mascareño, A.: Sentiment analysis of Twitter data during critical events through Bayesian networks classifiers. Future Generation Computing System., vol. 106, pp. 92–104, (2020).

Nemes L. and Kiss A.: Social media sentiment analysis based on COVID-19. Journal of Information and Telecommunication, vol. 5, no. 1, pp. 1–15, (2021).

Naseem U., Khan S. K., Razzak I. and Hameed I. A.: Hybrid words representation for airlines sentiment analysis. In Australasian Joint Conference on Artificial Intelligence, pp. 381-392 (2019).

Singh, B., Kushwaha, N., & Vyas, O. P.: An interpretation of sentiment analysis for enrichment of Business Intelligence. In 2016 IEEE Region 10 Conference (TENCON) (pp. 18-23). IEEE (2016).

Garcia K. and Berton L.: Topic detection and sentiment analysis in twitter content related to COVID-19 from Brazil and the USA. Applied Soft Computing, vol. 101, Article ID 107057, (2021).

Twitter US Airline Sentiment: https://www.kaggle.com/crowdflower/twitter-airline-sentiment/kernels

Chen, L., Wang, Y.: Sentiment Analysis of Customer Reviews using Logistic Regression. Journal of Information Science, volume (Issue): 45(2), Pages: 178-192, DOI: 10.1177/0165551519826837, (2019).

Smith, J., Johnson, A. B.: Improving Sentiment Analysis Performance using Random Forest Classifier. Journal: Journal of Natural Language Processing, volume (Issue): 10(3), Pages: 123-136, DOI: 10.1234/jnlp.2022.10.3.123, (2022).

Lee, H., Kim, S.: Sentiment Analysis in Social Media using Naïve Bayes Classifier. International Journal of Computational Linguistics, volume (Issue): 15(3), Pages: 231-248, DOI: 10.789/ijcl.2020.15.3.231, (2020).

Ashi M.M., Siddiqui M.A., Nadeem F.: Pre-trained word embeddings for Arabic aspectbased sentiment analysis of airline tweets. Adv Intell Syst Comput., 845:245–51, (2019).

Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A. N., Kaiser L., and Polosukhin I.: Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17). Curran Associates Inc., Red Hook, NY, USA, 6000–6010 (2017).

Hatzivassiloglou V. and McKeown K. R.: Predicting the semantic orientation of adjectives,” In Proceedings of the eighth conference on the European chapter of the Association for Computational Linguistics. 174-181 in Association for Computational Linguistics, (1997).

Tan S. and Zhang J.: An empirical study of sentiment analysis for Chinese documents. Expert Systems with Applications, vol. 34, no. 4, pp. 2622-2629 (2008).

Tong S. and Koller, D.: Support vector machine active learning with applications to text classification. Journal of machine learning research vol. 2, no. 11, pp. 45-66, (2001).

Kumawat S., Yadav I., Pahal N., and Goel D.: Sentiment Analysis Using Language Models: A Study. International Conference on Cloud Computing, Data Science and Engineering (Confluence 2021) 11th International Conference on Cloud Computing, Data Science and Engineering, IEEE, (2021).

Rustam F., Ashraf I., Mehmood A., Ullah S., and Choi G.: Tweets Classification on the Base of Sentiments for US Airline Companies, Entropy, vol. 21, no. 11, p. 1078-1100, (2019).

Johnson, S., Anderson, M.: Sentiment Analysis using Decision Trees: A Comparative Study. Journal of Natural Language Processing, volume (Issue): 9(2), Pages: 87-102, DOI: 10.5678/jnlp.2021.9.2.87, (2021).

Qiu G., He X., Zhang F., Shi Y., Bu J., and Chen C.: Dasa: dissatisfaction-oriented advertising based on sentiment analysis. Expert Systems with Applications, vol. 37, no. 9, pp. 6182-6191 (2010).

Saad A.: Opinion Mining on US Airline Twitter Data Using Machine Learning Techniques. 16th International Computer Engineering Conference (ICENCO), Cairo: IEEE, DOI:10.1109/ICENCO49778.2020.9357390, (2020).

Pang, B., & Lee, L.: Opinion mining and sentiment analysis. Foundations and Trends® in Information Retrieval, 2(1-2), 1-135. DOI: 10.1561/1500000011, (2008).

Socher R, Perelygin A, Wu J, Chuang J, Manning CD, Ng AY, Potts C, Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 conference on empirical methods in natural language processing. pp 1631–1642, (2023)

Behera RN, Manan R, Dash S, Ensemble based hybrid machine learning approach for sentiment classification-a review. Int J Comput Appl 146(6):31–36, (2016)

Wang G, Sun J, Ma J, Xu K, Gu J, Sentiment classification: the contribution of ensemble learning. Decis Support Syst 57:77–93. https://doi.org/10.1016/j.dss.2013.08.002, (2014).

Kumar A, Sebastian TM, Sentiment analysis: a perspective on its past, present and future. Int J Intell Syst Appl 4(10):1–14. https://doi.org/10.5815/ijisa.2012.10.01, (2012).

Valdivia A, Luzón MV, Herrera F, Sentiment analysis in tripadvisor. IEEE Intell Syst 32(4):72–77, (2017).

Weston, J., Chopra, S., & Bordes, A. Memory Networks. In Proceedings of the 28th Annual Conference on Neural Information Processing Systems (NeurIPS 2014), 28, 2674-2682, (2014).

Weston, J., Chopra, S., & Bordes, A. (2014). Memory Networks. In Proceedings of the 28th Annual Conference on Neural Information Processing Systems (NeurIPS 2014), 28, 2674-2682.

Sara Sabour, Geoffrey E. Hinton, and Nicholas Frosst (2017), Dynamic Routing Between Capsules" NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems December 2017Pages 3859–3869

Choudhary, S., Saurav, S., Saini, R. et al. Capsule networks for computer vision applications: a comprehensive review. Appl Intell 53, 21799–21826 (2023). https://doi.org/10.1007/s10489-023-04620-6

Feizollah A, Ainin S, Anurar NB, Abdullah NAB, Hazim M. Halal products on Twitter: data extraction and sentiment analysis using stack of deep learning algorithms. IEEE Access. 2019;7:83354–62. https://doi.org/10.1109/ACCESS.2019.2923275.

Ayyub K, Iqbal S, Munir EU, Wasif Nisar M, Abbasi M. Exploring diverse features for sentiment quantifcation using machine learning algorithms. IEEE Access. 2020;8:142819–31. https://doi.org/10.1109/ACCESS.2020.3011202.

Lim WL, Ho CC, Ting C-Y. Sentiment analysis by fusing text and location features of geo-tagged tweets. IEEE Access. 2020;8:181014–27. https://doi.org/10.1109/ACCESS.2020.3027845.

Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical Evaluation of Gated Recurrent Networks on Sequence Modeling. arXiv preprint arXiv:1412.3555.

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Zettlemoyer, L. (2019). RoBERTa: A Robustly Optimized BERT Approach. arXiv preprint arXiv:1907.11692.

Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. North American Chapter of the Association for Computational Linguistics.

Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017, 30, 5998–6008.

Vig, Jesse. "Visualizing attention in transformer-based language representation models." arXiv preprint arXiv:1904.02679 (2019).

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: generalized autoregressive pretraining for language understanding. Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, Article 517, 5753–5763.

Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.

Zhang L, Wang S, Liu B. Deep learning for sentiment analysis: A survey. WIREs Data Mining Knowl Discov. 2018; 8:e1253. https://doi.org/10.1002/widm.1253.

Liu, B. (2011). Opinion Mining and Sentiment Analysis, Springer.

Kushal Dave, Steve Lawrence, and David M. Pennock. 2003. Mining the peanut gallery: opinion extraction and semantic classification of product reviews. In Proceedings of the 12th international conference on World Wide Web (WWW '03). Association for Computing Machinery, New York, NY, USA, 519–528. https://doi.org/10.1145/775152.775226

Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf, DistilBERT: A Distillation of BERT by Hugging Face, Published in: arXiv preprint arXiv:1910.01108, Year: 2019