MFUIE: A Fake News Detection Model Based on Multimodal Features and User Information Enhancement

Xiulan Hao; Wenjing Xu; Xu  Huang; Zhenzhen Sheng; Huayun Yan

doi:10.4108/eetsis.7517

Authors

Xiulan Hao Huzhou University
Wenjing Xu Huzhou University
Xu Huang Huzhou College
Zhenzhen Sheng Huzhou University
Huayun Yan Huzhou University

DOI:

https://doi.org/10.4108/eetsis.7517

Keywords:

Multimodal Learning, fake news, Deep learning

Abstract

INTRODUCTION: Deep learning algorithms have advantages in extracting key features for detecting fake news. However, the existing multi-modal fake news detection models only fuse the visual and textual features after the encoder, failing to effectively utilize the multi-modal contextual relationships and resulting in insufficient feature fusion. Moreover, most fake news detection algorithms focus on mining news content and overlook the users' preferences whether to spread fake news.

OBJECTIVES: The model uses the multi-modal context relationship when extracting model features, and combines with user features to assist in mining multi-modal information to improve the performance of fake news detection.

METHODS: A fake news detection model called MFUIE (Multimodal Feature and User Information Enhancement) is proposed, which utilizes multi-modal features and user information enhancement. Firstly, for news content, we utilize the pre-trained language model BERT to encode sentences. At the same time, we use the Swin Transformer model as the main framework and introduce textual features during the early visual feature encoding to enhance semantic interactions. Additionally, we employ InceptionNetV3 as the image pattern analyser. Secondly, for user's historical posts, we use the same model as the news text to encode them, and introduce GAT (Graph Attention Network) to enhance information interaction between post nodes, capturing user-specific features. Finally, we fuse the obtained user features with the multi-modal features and validate the performance of the model.

RESULTS: The proposed model's performance is compared with those of existing methods. MFUIE model achieves an accuracy of 0.926 and 0.935 on the Weibo dataset and Weibo-21 dataset, respectively. F1 on Weibo is 0.926, 0.017 greater than SOAT model BRM; while F1 on Weibo-21 is 0.935, 0.009 greater than that of BRM.

CONCLUSION: Experimental results demonstrate that MFUIE can improve the fake news recognition in some degree.

References

[1] QI, CAO, & SHENG. ( 2021 ). Semantic-enhanced multimodal false news detection. Computer Research and Development, 58 ( 7 ), 1456-1465.

[2] WANG, Y., MA, F., JIN, Z., YUAN, Y., XUN, G., JHA, K., & Gao, J. (2018, July). Eann: Event adversarial neural networks for multi-modal fake news detection. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 849-857).

[3] ZHOU, X., WU, J., & ZAFARANI, R. (2020). Safe: similarity-aware multi-modal fake news detection. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (pp. 354-367). Cham: Springer International Publishing.

[4] ZHANG, H., FANG, Q., QIAN, S., & XU, C. (2019). Multi-modal knowledge-aware event memory network for social media rumor detection. In Proceedings of the 27th ACM International Conference on Multimedia (pp. 1942-1951).

[5] SARKI, R., AHMED, K., WANG, H., ZHANG, Y., MA, J. & WANG, K. (2021) Image preprocessing in classification and identification of diabetic eye diseases. Data Science and Engineering 6(4): 455–471.

[6] SARKI, R., AHMED, K., WANG, H., ZHANG, Y., & WANG, K. (2021) Convolutional neural network for multi-class classification of diabetic eye disease. EAI Endorsed Transactions on Scalable Information Systems 9(4). doi: https://doi.org/10.4108/eai.16-12-2021.172436.

[7] TAWHID, M.N.A., SIULY, S., WANG, K. & WANG, H. (2023) Automatic and efficient framework for identifying multiple neurological disorders from eeg signals. IEEE Transactions on Technology and Society 4(1): 76–86. doi: https://doi.org/10.1109/TTS.2023.3239526.

[8] ALVI, A.M., SIULY, S., & WANG, H. (2023) A long short term memory based framework for early detection of mild cognitive impairment from eeg signals. IEEE Transactions on Emerging Topics in Computational Intelligence 7(2): 375–388. doi: https://doi.org/10.1109/TETCI.2022.3186180.

[9] ALVI, A.M., SIULY, S., WANG, H., WANG, K., & Whittaker, F. (2022) A deep learning based framework for diagnosis of mild cognitive impairment. Knowledge-Based Systems 248: 108815. doi: https://doi.org/10.1016/j.knosys.2022.108815.

[10] TAWHID, M.N.A., SIULY, S., WANG, H., WHITTAKER, F., WANG, K., & ZHANG, Y. (2021) A spectrogram image based intelligent technique for automatic detection of autism spectrum disorder from eeg. Plos One 16(6): e0253094. doi: https://doi.org/10.1371/journal.pone.0253094.

[11] Laidi A, Ammar M, Daho MEH, Mahmoudi S. GAN Data Augmentation for Improved Automated Atherosclerosis Screening from Coronary CT Angiography. EAI Endorsed Scalable Information System [Internet]. 2022 May 17 [cited 2024 Jun. 3];10(1):e4. Available from: https://publications.eai.eu/index.php/sis/article/view/1027

[12] Hao X, Zhang C, Xu S. Fast Lung Image Segmentation Using Lightweight VAEL-Unet. EAI Endorsed Scalable Information System [Internet]. 2024 Apr. 8 [cited 2024 Jul. 13];11(6). Available from: https://publications.eai.eu/index.php/sis/article/view/4788

[13] SINGH, R., SUBRAMANI, S., DU, J., ZHANG, Y., WANG, H., MIAO, Y., & AHMED, K. (2023) Antisocial behavior identification from twitter feeds using traditional machine learning algorithms and deep learning. EAI Endorsed Transactions on Scalable Information Systems 10(4). doi: https://doi.org/10.4108/eetsis.v10i3.3184.

[14] Du, J., Rong, J., Wang, H., & Zhang, Y. (2021). Neighbor-aware review helpfulness prediction. Decision Support Systems, 148, 113581.

[15] XU, S., SONG, Y., & HAO, X. (2022) A comparative study of shallow machine learning models and deep learning models for landslide susceptibility assessment based on imbalanced data. Forests 13(11). doi: https://doi.org/10.3390/f13111908 URL https://www.mdpi.com/1999-4907/13/11/1908.

[16] CAO, Q., HAO, X., REN, H., XU, W., XU, S., & ASIEDU, C.J. (2022) Graph attention network based detection of causality for textual emotion-cause pair. World Wide Web: 1–15 doi: https://doi.org/10.1007/s11280-022-01111-5, URL https://doi.org/10.1007/s11280-022-01111-5.

[17] Sun, R. Y. (2020). Optimization for deep learning: An overview. Journal of the Operations Research Society of China, 8(2), 249-294.

[18] Ge, Y. F., Wang, H., Bertino, E., Zhan, Z. H., Cao, J., Zhang, Y., & Zhang, J. (2023). Evolutionary dynamic database partitioning optimization for privacy and utility. IEEE Transactions on Dependable and Secure Computing, 21(4), 2296-2311. doi: https://doi.org/10.1109/TDSC.2023.3302284.

[19] Ge, Y. F., Yu, W. J., Cao, J., Wang, H., Zhan, Z. H., Zhang, Y., & Zhang, J. (2020). Distributed memetic algorithm for outsourced database fragmentation. IEEE Transactions on Cybernetics, 51(10), 4808-4821.

[20] Wang, C., Sun, B., Du, K. J., Li, J. Y., Zhan, Z. H., Jeon, S. W., ... & Zhang, J. (2024). A novel evolutionary algorithm with column and sub-block local search for sudoku puzzles. IEEE Transactions on Games, 16(1), 162-172. doi: https://doi.org/10.1109/TG.2023.3236490.

[21] Shi, W., Chen, W. N., Kwong, S., Zhang, J., Wang, H., Gu, T., ... & Zhang, J. (2021). A coevolutionary estimation of distribution algorithm for group insurance portfolio. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 52(11), 6714-6728.

[22] Huang, T., Gong, Y. J., Chen, W. N., Wang, H., & Zhang, J. (2020). A probabilistic niching evolutionary computation framework based on binary space partitioning. IEEE Transactions on Cybernetics, 52(1), 51-64. doi: https://doi.org/10.1109/tcyb.2020.2972907

[23] Chen, Z.-G., Zhan, Z.-H., Wang, H., & Zhang, J. (2019). Distributed individuals for multiple peaks: a novel differential evolution for multimodal optimization problems. IEEE Transactions on Evolutionary Computation, 1–1. doi: https://doi.org/10.1109/tevc.2019.2944180

[24] Li, J. Y., Du, K. J., Zhan, Z. H., Wang, H., & Zhang, J. (2023). Distributed differential evolution with adaptive resource allocation. IEEE Transactions on Cybernetics, 53(5), 2791-2804. doi: https://doi.org/10.1109/TCYB.2022.3153964.

[25] Liu, W. L., Gong, Y. J., Chen, W. N., Liu, Z., Wang, H., & Zhang, J. (2019). Coordinated charging scheduling of electric vehicles: A mixed-variable differential evolution approach. IEEE Transactions on Intelligent Transportation Systems, 21(12), 5094-5109

[26] Huang, T., Gong, Y.-J., Kwong, S., Wang, H., & Zhang, J. (2019). A niching memetic algorithm for multi-solution traveling salesman problem. IEEE Transactions on Evolutionary Computation, 1–1. doi: https://doi.org/10.1109/tevc.2019.2936440

[27] Li, J. Y., Zhan, Z. H., Wang, H., & Zhang, J. (2020). Data-driven evolutionary algorithm with perturbation-based ensemble surrogates. IEEE Transactions on Cybernetics, 51(8), 3925-3937. doi: https://doi.org/10.1109/tcyb.2020.3008280

[28] Yang, J. Q., Yang, Q. T., Du, K. J., Chen, C. H., Wang, H., Jeon, S. W., ... & Zhan, Z. H. (2023). Bi-directional feature fixation-based particle swarm optimization for large-scale feature selection. IEEE Transactions on Big Data, 9(3), 1004-1017. doi: https://doi.org/10.1109/TBDATA.2022.3232761

[29] B. B, Rani KS, Neog A. Finding Multidimensional Constraint Reachable Paths for Attributed Graphs. EAI Endorsed Scalable Information System [Internet]. 2022 Aug. 22 [cited 2024 Jun. 3];10(1): e8. Available from: https://publications.eai.eu/index.php/sis/article/view/2581

[30] ZHANG, L., ZHANG, X., ZHOU, Z., HUANG, F., & LI, C. (2024). Reinforced Adaptive Knowledge Learning for Multimodal Fake News Detection. In Proceedings of the AAAI Conference on Artificial Intelligence 38(15): AAAI-24 Technical Tracks 15, 16777-16785

[31] SINGHAL, S., SHAH, R. R., CHAKRABORTY, T., KUMARAGURU, P., & SATOH, S. I. (2019). Spotfake: A multi-modal framework for fake news detection. In 2019 IEEE fifth International Conference on Multimedia Big Data (BigMM) (pp. 39-47). IEEE.

[32] KHATTAR, D., GOUD, J. S., GUPTA, M., & VARMA, V. (2019). Mvae: Multimodal variational autoencoder for fake news detection. In The World Wide Web Conference (WWW '19). Association for Computing Machinery, New York, NY, USA, 2915–2921. https://doi.org/10.1145/3308558.3313552

[33] QU, Z., MENG, Y., MUHAMMAD, G., & TIWARI, P. (2024). QMFND: A quantum multimodal fusion-based fake news detection model for social media. Information Fusion, 104, 102172.

[34] XUE, J., WANG, Y., TIAN, Y., LI, Y., SHI, L., & WEI, L. (2021). Detecting fake news by exploring the consistency of multimodal data. Information Processing & Management, 58(5), 102610.

[35] JIN, Z., CAO, J., GUO, H., ZHANG, Y., & LUO, J. (2017). Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In Proceedings of the 25th ACM International Conference on Multimedia (MM '17). Association for Computing Machinery, New York, NY, USA, 795–816. https://doi.org/10.1145/3123266.3123454.

[36] WU, Y., ZHAN, P., ZHANG, Y., WANG, L., & XU, Z. (2021). Multimodal fusion with co-attention networks for fake news detection. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021,. 2560-2569.

[37] YANG, F., LIU, Y., YU, X., & YANG, M. (2012). Automatic detection of rumor on sina weibo. In Proceedings of the ACM SIGKDD Workshop on Mining Data Semantics, 1-7.

[38] LIU, Y., & WU, Y. F. (2018). Early detection of fake news on social media through propagation path classification with recurrent and convolutional networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 354-361.

[39] YIN PENGBO, PAN WEIMIN, PENG CHENG, & ZHANG HAIJUN. (2020). Research on early detection of Weibo rumours based on user feature analysis. Intelligence Magazine, 39 (7), 81-86.

[40] CHEN, X., ZHOU, F., TRAJCEVSKI, G., & BONSANGUE, M. (2022). Multi-view learning with distinguishable feature fusion for rumor detection. Knowledge-Based Systems, 240, 108085.

[41] JIANG, S., CHEN, X., ZHANG, L., CHEN, S., & LIU, H. (2019). User-characteristic enhanced model for fake news detection in social media. In Natural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9–14, 2019, Proceedings, Part I 8, 634-646.

[42] JACOB DEVLIN, MING-WEI CHANG, KENTON LEE, & KRISTINA TOUTANOVA. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171-4186.

[43] LUO, G., ZHOU, Y., JI, R., SUN, X., SU, J., LIN, C. W., & TIAN, Q. (2020). Cascade grouped attention network for referring expression segmentation. In Proceedings of the 28th ACM International Conference on Multimedia ,1274-1282.

[44] MONTI, F., FRASCA, F., EYNARD, D., MANNION, D., & BRONSTEIN, M.M. (2019). Fake News Detection on Social Media using Geometric Deep Learning. ArXiv, abs/1902.06673.

[45] TUAN, N. M. D., & MINH, P. Q. N. (2021). Multimodal fusion with BERT and attention mechanism for fake news detection. In 2021 RIVF International Conference on Computing and Communication Technologies (RIVF) (pp. 1-6). IEEE.

[46] VASWANI, A., SHAZEER, N., PARMAR, N., USZKOREIT, J., JONES, L., GOMEZ, A. N., & POLOSUKHIN, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.

[47] QI, P., CAO, J., YANG, T., GUO, J., & LI, J. (2019). Exploiting multi-domain visual information for fake news detection. In 2019 IEEE International Conference on Data Mining (ICDM) (pp. 518-527). IEEE.

[48] SZEGEDY, C., VANHOUCKE, V., IOFFE, S., SHLENS, J., & WOJNA, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2818-2826).

[49] BAYAR, B., & STAMM, M. C. (2018). Constrained convolutional neural networks: A new approach towards general purpose image manipulation detection. IEEE Transactions on Information Forensics and Security, 13(11), 2691-2706.

[50] AHMAD, N., & SIDDIQUE, J. (2017). Personality assessment using Twitter tweets. Procedia computer science, 112, 1964-1973.

[51] NAN, Q., CAO, J., ZHU, Y., WANG, Y., & LI, J. (2021). MDFEND: Multi-domain fake news detection. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (pp. 3343-3347).

[52] HU, B., SHENG, Q., CAO, J., SHI, Y., LI, Y., WANG, D., & QI, P. (2024). Bad actor, good advisor: Exploring the role of large language models in fake news detection. In Proceedings of the AAAI Conference on Artificial Intelligence 38(20), 22105-22113

[53] YING, Q., HU, X., ZHOU, Y., QIAN, Z., ZENG, D., & GE, S. (2023). Bootstrapping multi-view representations for fake news detection. In Proceedings of the AAAI conference on Artificial Intelligence,5384-5392.

[54] Li, H., Huang, G., Li, Y., Zhang, X., & Wang, Y. (2023). Sentence classification based on the concept kernel attention mechanism. EAI Endorsed Transactions on Scalable Information Systems, 10(1), e3-e3.

[55] Öztürk MM. Developing a hyperparameter optimization method for classification of code snippets and questions of stack overflow: HyperSCC. EAI Endorsed Scalable Information System [Internet]. 2022 May 27 [cited 2024 Jun. 3];10(1):e5. Available from:

https://publications.eai.eu/index.php/sis/article/view/1267