Evaluating Performance of Conversational Bot Using Seq2Seq Model and Attention Mechanism





Seq2Seq, Attention Mechanism, Perplexity, BLEU, ROUGE


The Chat-Bot utilizes Sequence-to-Sequence Model with the Attention Mechanism, in order to interpret and address user inputs effectively. The whole model consists of Data gathering, Data preprocessing, Seq2seq Model, Training and Tuning. Data preprocessing involves cleaning of any irrelevant data, before converting them into the numerical format. The Seq2Seq Model is comprised of two components: an Encoder and a Decoder. Both Encoder and Decoder along with the Attention Mechanism allow dialogue management, which empowers the Model to answer the user in the most accurate and relevant manner. The output generated by the Bot is in the Natural Language only. Once the building of the Seq2Seq Model is completed, training of the model takes place in which the model is fed with the preprocessed data, during training it tries to minimize the loss function between the predicted output and the ground truth output. Performance is computed using metrics such as perplexity, BLEU score, and ROUGE score on a held-out validation set. In order to meet non-functional requirements, our system needs to maintain a response time of under one second with an accuracy target exceeding 90%.


Alireza Sadeghi Milani, Aaron Cecil-Xavier, Avinash Gupta, J. Cecil & Shelia Kennison (2022) A Systematic Review of Human–Computer Interaction (HCI) Research in Medical and Other Engineering Fields, International Journal of Human–Computer Interaction, DOI: 10.1080/10447318.2022.2116530

Khurana, D., Koli, A., Khatter, K. et al. Natural language processing: state of the art, current trends and challenges. Multimed Tools Appl 82, 3713–3744 (2023). https://doi.org/10.1007/s11042-022-13428-4

Adamopoulou, E., Moussiades, L. (2020). An Overview of Chatbot Technology. In: Maglogiannis, I., Iliadis, L., Pimenidis, E. (eds) Artificial Intelligence Applications and Innovations. AIAI 2020. IFIP Advances in Information and Communication Technology, vol 584. Springer, Cham. https://doi.org/10.1007/978-3-030-49186-4_31

Turing, A.M.: Computing machinery and intelligence. Mind 59, 433–460 (1950). https://doi. org/10.1093/mind/LIX.236.433

Brandtzaeg, P.B., Følstad, A.: Why people use chatbots. In: Kompatsiaris, I., et al. (eds.) Internet Science, pp. 377– 392. Springer, Cham (2017). https://doi.org/10.1007/978-3-319- 70284-1_30

Colby, K.M., Weber, S., Hilf, F.D.: Artificial paranoia. Artif. Intell. 2, 1–25 (1971). https:// doi.org/10.1016/0004- 3702(71)90002-6

Wallace, R.S.: The anatomy of A.L.I.C.E. In: Epstein, R., Roberts, G., Beber, G. (eds.) Parsing the Turing Test: Philosophical and Methodological Issues in the Quest for the Thinking Computer, pp. 181–210. Springer, Cham (2009). https://doi.org/10.1007/978-1-4020-6710- 5_13

Marietto, M., et al.: Artificial intelligence markup language: a brief tutorial. Int. J. Comput. Sci. Eng. Surv. 4 (2013). https://doi.org/10.5121/ijcses.2013.4301

Molnár, G., Zoltán, S.: The role of chatbots in formal education. Presented at the 15 September 2018

Colace, F., De Santo, M., Lombardi, M., Pascale, F., Pietrosanto, A., Lemma, S.: Chatbot for e-learning: a case of study. Int. J. Mech. Eng. Robot. Res. 7, 528–533 (2018). https://doi.org/ 10.18178/ijmerr.7.5.528-533

da Costa, P.C.F.: Conversing with personal digital assistants: on gender and artificial intelligence. J. Sci. Technol. Arts 10, 59–72 (2018). https://doi.org/10.7559/citarj.v10i3.563 22. Xu, A., Liu, Z., Guo, Y., Sinha, V., Akkiraju, R.: A new chatbot for customer service on social media. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 3506–3510. ACM, New York (2017)

Følstad, A., Nordheim, C.B., Bjørkli, C.A.: What makes users trust a chatbot for customer service? An exploratory interview study. In: Bodrunova, S.S. (ed.) INSCI 2018. LNCS, vol. 11193, pp. 194–208. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01437-7_16 24. Go, E., Sundar, S.S.: Humanizing chatbots: the effects of visual, identity and conversational cues on humanness perceptions. Comput. Hum. Behav. 97, 304–316 (2019). https://doi.org/ 10.1016/j.chb.2019.01.020

Sannon, S., Stoll, B., DiFranzo, D., Jung, M., Bazarova, N.N.: How personification and interactivity influence stress-related disclosures to conversational agents. In: Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing, pp. 285–288. ACM, New York (2018)

Fernandes, A.: NLP, NLU, NLG and how Chatbots work. https://chatbotslife.com/nlp-nlunlg-and-how-chatbots- work-dd7861dfc9df

Ramesh, K., Ravishankaran, S., Joshi, A., Chandrasekaran, K.: A survey of design techniques for conversational agents. In: Kaushik, S., Gupta, D., Kharb, L., Chahal, D. (eds.) ICICCT 2017. CCIS, vol. 750, pp. 336–350. Springer, Singapore (2017). https://doi.org/10.1007/978- 981-10-6544-6_31

Akma, N., Hafiz, M., Zainal, A., Fairuz, M., Adnan, Z.: Review of chatbots design techniques. Int. J. Comput. Appl. 181, 7–10 (2018). https://doi.org/10.5120/ijca2018917606

Artificial Intelligence Scripting Language - RiveScript.com. https://www.rivescript.com/ 32. Jung, S.: Semantic vector learning for natural language understanding. Comput. Speech Lang. 56, 130–145 (2019). https://doi.org/10.1016/j.csl.2018.12.008

“Chatbot Market Size, Share, Trends | CAGR of 23.91%,” Market.us, Nov. 02, 2023. [Online]. Available: https://market.us/report/chatbot-market/#overview Presented at the (2018)

Nimavat, K., Champaneria, T.: Chatbots: an overview types, architecture, tools and future possibilities. Int. J. Sci. Res. Dev. 5, 1019–1024 (2017)

Kucherbaev, P., Bozzon, A., Houben, G.-J.: Human-aided bots. IEEE Internet Comput. 22, 36–43 (2018). https://doi.org/10.1109/MIC.2018.252095348

Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2015). A Diversity-Promoting Objective Function for Neural Conversation Models. ArXiv. /abs/1510.03055

Balaganesh Bojarajulu, Sarvesh Tanwar, and Thipendra Pal Singh. Parametric and Non-parametric Analysis on MAOA-based Intelligent IoT-BOTNET Attack Detection Model [J]. Int J Performability Eng, 2022, 18(10): 741-750

Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. arXiv:1409.3215. https://doi.org/10.48550/arXiv.1409.3215

Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473. https://doi.org/10.48550/arXiv.1409.0473

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Bidirectional Encoder Representations from Transformers. arXiv:1810.04805. https://doi.org/10.48550/arXiv.1810.04805

Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL) https://doi.org/10.3115/1073083.1073135

Lin, C. Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. https://aclanthology.org/W04-1013

Zhang, T., Zhao, H., & LeCun, Y. (2018). BLEU is Not a Good Metric for Legal Text Generation: The (Lack of) Correlation Between BLEU and Human Judgments. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)

Li, J., Monroe, W., & Jurafsky, D. (2016). A Simple, Fast Diverse Decoding Algorithm for Neural Generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL) https://doi.org/10.48550/arXiv.1611.08562

Zeng, D., Liu, K., Lai, S., Zhou, G., & Zhao, J. (2014). Relation classification via convolutional deep neural network. In Proceedings of COLING (ACL) https://aclanthology.org/C14-1220

Tur, G., & De Mori, R. (2011). Spoken Language Understanding: Systems for Extracting Semantic Information from Speech. John Wiley & Sons.

Open source conversational AI. (2022, October 6). Retrieved from https://rasa.community/#:~:text=Rasa%20uses%20a%20composable%20set,and%20scale%20sophisticated%20conversational%20AI. (Accessed from Dehradun, India on December 01, 23 at 16:54

Niculescu-Mizil, C., & Lee, L, (2011). “Cornell Movie Dialogs Corpus”. Cornell University. https://www.cs.cornell.edu/~cristian/Cornell_Movie_Dialogs_Corpus.html




How to Cite

Saluja K, Agarwal S, Kumar S, Choudhury T. Evaluating Performance of Conversational Bot Using Seq2Seq Model and Attention Mechanism. EAI Endorsed Scal Inf Syst [Internet]. 2024 Mar. 18 [cited 2024 Apr. 18];. Available from: https://publications.eai.eu/index.php/sis/article/view/5457



Research articles