Cyberbullying Text Identification based on Deep Learning and Transformer-based Language Models

Khalid Saifullah; Muhammad Ibrahim Khan; Suhaima Jamal; Iqbal H. Sarker

doi:10.4108/eetinis.v11i1.4703

Authors

Khalid Saifullah Chittagong University of Engineering & Technology
Muhammad Ibrahim Khan Chittagong University of Engineering & Technology
Suhaima Jamal Georgia Southern University
Iqbal H. Sarker Edith Cowan University

DOI:

https://doi.org/10.4108/eetinis.v11i1.4703

Keywords:

Cyberbullying, large language modeling, deep learning, transformers models, natural language processing, NLP, fine tuning, OOV, harmful messages

Abstract

In the contemporary digital age, social media platforms like Facebook, Twitter, and YouTube serve as vital channels for individuals to express ideas and connect with others. Despite fostering increased connectivity, these platforms have inadvertently given rise to negative behaviors, particularly cyberbullying. While extensive research has been conducted on high-resource languages such as English, there is a notable scarcity of resources for low-resource languages like Bengali, Arabic, Tamil, etc., particularly in terms of language modeling. This study addresses this gap by developing a cyberbullying text identification system called BullyFilterNeT tailored for social media texts, considering Bengali as a test case. The intelligent BullyFilterNeT system devised overcomes Out-of-Vocabulary (OOV) challenges associated with non-contextual embeddings and addresses the limitations of context-aware feature representations. To facilitate a comprehensive understanding, three non-contextual embedding models GloVe, FastText, and Word2Vec are developed for feature extraction in Bengali. These embedding models are utilized in the classification models, employing three statistical models (SVM, SGD, Libsvm), and four deep learning models (CNN, VDCNN, LSTM, GRU). Additionally, the study employs six transformer-based language models: mBERT, bELECTRA, IndicBERT, XML-RoBERTa, DistilBERT, and BanglaBERT, respectively to overcome the limitations of earlier models. Remarkably, BanglaBERT-based BullyFilterNeT achieves the highest accuracy of 88.04% in our test set, underscoring its effectiveness in cyberbullying text identification in the Bengali language.

Downloads

Download data is not yet available.

References

Abdhullah-Al-Mamun and Shahin Akhter. Social media bullying detection using machine learning on bangla text. In 2018 10th International Conference on Electrical and Computer Engineering (ICECE), pages 385–388, 2018.

Sadia Afroze and Mohammed Moshiul Hoque. Sntiemd: Sentiment specific embedding model generation and EAI Endorsed Transactions Preprint evaluation for a resource constraint language. In Intelligent Computing & Optimization, pages 242–252, Cham, 2023. Springer International Publishing.

Md. Tofael Ahmed, Maqsudur Rahman, Shafayet Nur, Azm Islam, and Dipankar Das. Deployment of machine learning and deep learning algorithms in detecting cyberbullying in bangla and romanized bangla text: A comparative study. In 2021 International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT), pages 1–10, 2021.

Arnisha Akhter, Uzzal Kumar Acharjee, Md. Alamin Talukder, Md. Manowarul Islam, and Md Ashraf Uddin. A robust hybrid machine learning model for bengali cyber bullying detection in social media. Natural Language Processing Journal, 4:100027, 2023.

Sara Azmin and Kingshuk Dhar. Emotion detection from bangla text corpus using naïve bayes classifier. In 2019 4th International Conference on Electrical Information and Communication Technology (EICT), pages 1–5, 2019.

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Tran. ACL, 5:135–146, June 2017.

Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Proceedings of the international AAAI conference on web and social media, volume 11, pages 512–515, 2017.

Luis Gerardo Mojica de la Vega and Vincent Ng. Modeling trolling in social media conversations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.

Amirita Dewani, Mohsin Ali Memon, and Sania Bhatti. Cyberbullying detection: advanced preprocessing techniques & deep learning architecture for roman urdu data. Journal of Big Data, 8(1):160, December 2021.

Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. Large scale crowdsourcing and characterization of twitter abusive behavior. In Proceedings of the international AAAI conference on web and social media, volume 12, 2018.

Lei Gao and Ruihong Huang. Detecting online hate speech using context aware models. arXiv preprint arXiv:1710.07395, 2017.

Md. Rajib Hossain and Mohammed Moshiul Hoque. Coberttc: Covid-19 text classification using transformerbased language models. pages 179–186, Cham, 2023. Springer Nature Switzerland.

Md. Rajib Hossain and Mohammed Moshiul Hoque. Toward embedding hyperparameters optimization: Analyzing their impacts on deep leaning-based text classification. In The Fourth Industrial Revolution and Beyond, pages 501–512, Singapore, 2023. Springer Nature Singapore.

Md. Rajib Hossain, Mohammed Moshiul Hoque, M. Ali Akber Dewan, Nazmul Siddique, Md. Nazmul Islam, and Iqbal H. Sarker. Authorship classification in a resource constraint language using convolutional neural networks. IEEE Access, 9:100319–100338, 2021.

Md. Rajib Hossain, Mohammed Moshiul Hoque, and Nazmul Siddique. Leveraging the meta-embedding for text classification in a resource-constrained language. Engineering Applications of Artificial Intelligence, 124:106586, September 2023.

Md. Rajib Hossain, Mohammed Moshiul Hoque, Nazmul Siddique, and Iqbal H. Sarker. Bengali text document categorization based on very deep convolution neural network. Expert Systems with Applications, 184:115394, 2021.

Md. Rajib Hossain, Mohammed Moshiul Hoque, Nazmul Siddique, and Iqbal H Sarker. CovTiNet: Covid text identification network using attention-based positional embedding feature fusion. Neural Computing and Applications, 35(18):13503–13527, June 2023.

Mladen Karan and Jan Šnajder. Preemptive toxic language detection in wikipedia comments using threadlevel context. In Proceedings of the Third Workshop on Abusive Language Online, pages 129–134, 2019.

Ritesh Kumar, Atul Kr Ojha, Shervin Malmasi, and Marcos Zampieri. Benchmarking aggression identification in social media. In Proceedings of the first workshop on trolling, aggression and cyberbullying (TRAC-2018), pages 1–11, 2018.

Ritesh Kumar, Atul Kr Ojha, Shervin Malmasi, and Marcos Zampieri. Evaluating aggression identification in social media. In Proceedings of the second workshop on trolling, aggression and cyberbullying, pages 1–5, 2020.

Todor Mihaylov, Georgi Georgiev, and Preslav Nakov. Finding opinion manipulation trolls in news community forums. In Proceedings of the nineteenth conference on computational natural language learning, pages 310–314, 2015.

T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. pages 1–12, 2013.

Nishant Nikhil, Ramit Pahwa, Mehul Kumar Nirala, and Rohan Khilnani. Lstms with attention for aggression detection. arXiv preprint arXiv:1807.06151, 2018.

Endang Wahyu Pamungkas and Viviana Patti. Crossdomain and cross-lingual abusive language detection: A hybrid approach with deep learning and a multilingual lexicon. In Proceedings of the 57th annual meeting of the association for computational linguistics: Student research workshop, pages 363–370, 2019.

John Pavlopoulos, Nithum Thain, Lucas Dixon, and Ion Androutsopoulos. Convai at semeval-2019 task 6: Offensive language identification and categorization with perspective and bert. In Proceedings of the 13th international Workshop on Semantic Evaluation, pages 571–576, 2019.

J. Pennington, R. Socher, and C. Manning. Glove: Global vectors for word representation. In Proc. EMNLP, pages 1532–1543, Doha, Qatar, 2014. ACL.

Eric Rice, Robin Petering, Harmony Rhoades, Hailey Winetrobe, Jeremy Goldbach, Aaron Plant, Jorge Montoya, and Timothy Kordic. Cyberbullying perpetration and victimization among middle-school students. American journal of public health, 105(3):e66–e72, 2015.

Julian Risch and Ralf Krestel. Bagging bert models for robust aggression identification. In Proceedings EAI Endorsed Transactions Preprint of the Second Workshop on Trolling, Aggression and Cyberbullying, pages 55–61, 2020.

Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. arXiv preprint arXiv:1701.08118, 2017.

Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. Predicting the type and target of offensive posts in social media. arXiv preprint arXiv:1902.09666, 2019.