Evaluating Performance of Conversational Bot Using Seq2Seq Model and Attention Mechanism
DOI:
https://doi.org/10.4108/eetsis.5457Keywords:
Seq2Seq, Attention Mechanism, Perplexity, BLEU, ROUGEAbstract
The Chat-Bot utilizes Sequence-to-Sequence Model with the Attention Mechanism, in order to interpret and address user inputs effectively. The whole model consists of Data gathering, Data preprocessing, Seq2seq Model, Training and Tuning. Data preprocessing involves cleaning of any irrelevant data, before converting them into the numerical format. The Seq2Seq Model is comprised of two components: an Encoder and a Decoder. Both Encoder and Decoder along with the Attention Mechanism allow dialogue management, which empowers the Model to answer the user in the most accurate and relevant manner. The output generated by the Bot is in the Natural Language only. Once the building of the Seq2Seq Model is completed, training of the model takes place in which the model is fed with the preprocessed data, during training it tries to minimize the loss function between the predicted output and the ground truth output. Performance is computed using metrics such as perplexity, BLEU score, and ROUGE score on a held-out validation set. In order to meet non-functional requirements, our system needs to maintain a response time of under one second with an accuracy target exceeding 90%.
References
Alireza Sadeghi Milani, Aaron Cecil-Xavier, Avinash Gupta, J. Cecil & Shelia Kennison (2022) A Systematic Review of Human–Computer Interaction (HCI) Research in Medical and Other Engineering Fields, International Journal of Human–Computer Interaction, DOI: 10.1080/10447318.2022.2116530
Khurana, D., Koli, A., Khatter, K. et al. Natural language processing: state of the art, current trends and challenges. Multimed Tools Appl 82, 3713–3744 (2023). https://doi.org/10.1007/s11042-022-13428-4
Adamopoulou, E., Moussiades, L. (2020). An Overview of Chatbot Technology. In: Maglogiannis, I., Iliadis, L., Pimenidis, E. (eds) Artificial Intelligence Applications and Innovations. AIAI 2020. IFIP Advances in Information and Communication Technology, vol 584. Springer, Cham. https://doi.org/10.1007/978-3-030-49186-4_31
Turing, A.M.: Computing machinery and intelligence. Mind 59, 433–460 (1950). https://doi. org/10.1093/mind/LIX.236.433
Brandtzaeg, P.B., Følstad, A.: Why people use chatbots. In: Kompatsiaris, I., et al. (eds.) Internet Science, pp. 377– 392. Springer, Cham (2017). https://doi.org/10.1007/978-3-319- 70284-1_30
Colby, K.M., Weber, S., Hilf, F.D.: Artificial paranoia. Artif. Intell. 2, 1–25 (1971). https:// doi.org/10.1016/0004- 3702(71)90002-6
Wallace, R.S.: The anatomy of A.L.I.C.E. In: Epstein, R., Roberts, G., Beber, G. (eds.) Parsing the Turing Test: Philosophical and Methodological Issues in the Quest for the Thinking Computer, pp. 181–210. Springer, Cham (2009). https://doi.org/10.1007/978-1-4020-6710- 5_13
Marietto, M., et al.: Artificial intelligence markup language: a brief tutorial. Int. J. Comput. Sci. Eng. Surv. 4 (2013). https://doi.org/10.5121/ijcses.2013.4301
Molnár, G., Zoltán, S.: The role of chatbots in formal education. Presented at the 15 September 2018
Colace, F., De Santo, M., Lombardi, M., Pascale, F., Pietrosanto, A., Lemma, S.: Chatbot for e-learning: a case of study. Int. J. Mech. Eng. Robot. Res. 7, 528–533 (2018). https://doi.org/ 10.18178/ijmerr.7.5.528-533
da Costa, P.C.F.: Conversing with personal digital assistants: on gender and artificial intelligence. J. Sci. Technol. Arts 10, 59–72 (2018). https://doi.org/10.7559/citarj.v10i3.563 22. Xu, A., Liu, Z., Guo, Y., Sinha, V., Akkiraju, R.: A new chatbot for customer service on social media. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 3506–3510. ACM, New York (2017)
Følstad, A., Nordheim, C.B., Bjørkli, C.A.: What makes users trust a chatbot for customer service? An exploratory interview study. In: Bodrunova, S.S. (ed.) INSCI 2018. LNCS, vol. 11193, pp. 194–208. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01437-7_16 24. Go, E., Sundar, S.S.: Humanizing chatbots: the effects of visual, identity and conversational cues on humanness perceptions. Comput. Hum. Behav. 97, 304–316 (2019). https://doi.org/ 10.1016/j.chb.2019.01.020
Sannon, S., Stoll, B., DiFranzo, D., Jung, M., Bazarova, N.N.: How personification and interactivity influence stress-related disclosures to conversational agents. In: Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing, pp. 285–288. ACM, New York (2018)
Fernandes, A.: NLP, NLU, NLG and how Chatbots work. https://chatbotslife.com/nlp-nlunlg-and-how-chatbots- work-dd7861dfc9df
Ramesh, K., Ravishankaran, S., Joshi, A., Chandrasekaran, K.: A survey of design techniques for conversational agents. In: Kaushik, S., Gupta, D., Kharb, L., Chahal, D. (eds.) ICICCT 2017. CCIS, vol. 750, pp. 336–350. Springer, Singapore (2017). https://doi.org/10.1007/978- 981-10-6544-6_31
Akma, N., Hafiz, M., Zainal, A., Fairuz, M., Adnan, Z.: Review of chatbots design techniques. Int. J. Comput. Appl. 181, 7–10 (2018). https://doi.org/10.5120/ijca2018917606
Artificial Intelligence Scripting Language - RiveScript.com. https://www.rivescript.com/ 32. Jung, S.: Semantic vector learning for natural language understanding. Comput. Speech Lang. 56, 130–145 (2019). https://doi.org/10.1016/j.csl.2018.12.008
“Chatbot Market Size, Share, Trends | CAGR of 23.91%,” Market.us, Nov. 02, 2023. [Online]. Available: https://market.us/report/chatbot-market/#overview Presented at the (2018)
Nimavat, K., Champaneria, T.: Chatbots: an overview types, architecture, tools and future possibilities. Int. J. Sci. Res. Dev. 5, 1019–1024 (2017)
Kucherbaev, P., Bozzon, A., Houben, G.-J.: Human-aided bots. IEEE Internet Comput. 22, 36–43 (2018). https://doi.org/10.1109/MIC.2018.252095348
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2015). A Diversity-Promoting Objective Function for Neural Conversation Models. ArXiv. /abs/1510.03055
Balaganesh Bojarajulu, Sarvesh Tanwar, and Thipendra Pal Singh. Parametric and Non-parametric Analysis on MAOA-based Intelligent IoT-BOTNET Attack Detection Model [J]. Int J Performability Eng, 2022, 18(10): 741-750
Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. arXiv:1409.3215. https://doi.org/10.48550/arXiv.1409.3215
Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473. https://doi.org/10.48550/arXiv.1409.0473
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Bidirectional Encoder Representations from Transformers. arXiv:1810.04805. https://doi.org/10.48550/arXiv.1810.04805
Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL) https://doi.org/10.3115/1073083.1073135
Lin, C. Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. https://aclanthology.org/W04-1013
Zhang, T., Zhao, H., & LeCun, Y. (2018). BLEU is Not a Good Metric for Legal Text Generation: The (Lack of) Correlation Between BLEU and Human Judgments. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)
Li, J., Monroe, W., & Jurafsky, D. (2016). A Simple, Fast Diverse Decoding Algorithm for Neural Generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL) https://doi.org/10.48550/arXiv.1611.08562
Zeng, D., Liu, K., Lai, S., Zhou, G., & Zhao, J. (2014). Relation classification via convolutional deep neural network. In Proceedings of COLING (ACL) https://aclanthology.org/C14-1220
Tur, G., & De Mori, R. (2011). Spoken Language Understanding: Systems for Extracting Semantic Information from Speech. John Wiley & Sons.
Open source conversational AI. (2022, October 6). Retrieved from https://rasa.community/#:~:text=Rasa%20uses%20a%20composable%20set,and%20scale%20sophisticated%20conversational%20AI. (Accessed from Dehradun, India on December 01, 23 at 16:54
Niculescu-Mizil, C., & Lee, L, (2011). “Cornell Movie Dialogs Corpus”. Cornell University. https://www.cs.cornell.edu/~cristian/Cornell_Movie_Dialogs_Corpus.html
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Karandeep Saluja, Shashwat Agarwal, Sanjeev Kumar, Tanupriya Choudhury
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
This is an open access article distributed under the terms of the CC BY-NC-SA 4.0, which permits copying, redistributing, remixing, transformation, and building upon the material in any medium so long as the original work is properly cited.
