An Automatic Grammar Error Correction Model Based on Encoder-decoder Structure for English Texts

Jiahao Wang; Guimin Huang; Yabing Wang

doi:10.4108/eetsis.v9i5.2011

Authors

Jiahao Wang Guilin University of Electronic Technology
Guimin Huang Guilin University of Electronic Technology
Yabing Wang Guilin University of Electronic Technology

DOI:

https://doi.org/10.4108/eetsis.v9i5.2011

Keywords:

Encoder-decoder, Grammar Error Correction(GEC), deep neural network, attention mechanism, beam search

Abstract

The role of information transmission in social life is irreplaceable, and language is a very important information carrier. Among all kinds of languages, English always occupies an important position. In the process of English learning, grammar error has become a difficult problem for most learners. In this paper, we propose an automatic grammar error correction model based on encoder-decoder structure. Different from traditional encoders, we design a dual-encoder structure to capture the information of source sentence and context sentence separately. The decoder is designed with a gated structure, it can effectively integrate output information of encoders. At the same time, the self-attention mechanism is combined to better solve the problem of long-distance information extraction. In addition, we propose a dynamic beam search algorithm to improve the accuracy of the word prediction process, and achieve dynamic extraction of the decoder output by combining kernel sampling techniques. We add a penalty factor to reduce the probability of generating repeated words, while suppressing the model's preference for generating shorter sentences. Finally, the proposed method is validated on the official English grammar error correction dataset. Experiments show that the dual encoder model in this paper has a good performance.

References

Bentley J. Report from TESOL 2014: 1.5 Billion English learners worldwide[J]. Chicago, IL: International TEFL Academy found online on December, 2014, 19: 2017.

Ranalli J, Yamashita T. Automated written corrective feedback: Error-correction performance and timing of delivery[J]. Language Learning & Technology, 2022, 26(1): 1-25.

Sakaguchi K. Robust Text Correction for Grammar and Fluency[D]. Johns Hopkins University, 2018.

Naber D, A rule-based style and grammar checker[J]. university of Bielefeld, 2003.

Gamon M, Leacock C, Brockett C, Using statistical techniques and web search to correct ESL errors[J]. Calico Journal, 2009, 26(3): 491-511.

Makarenkov V, Rokach L, Shapira B, Choosing the right word: Using bidirectional LSTM tagger for writing support systems[J]. Engineering Applications of Artificial Intelligence, 2019, 84: 1-10.

Hu L, Tang Y, Wu X, Considering optimization of English grammar error correction based on neural network[J]. Neural Computing and Applications, 2022, 34(5): 3323-3335.

Xie Z, Avati A, Arivazhagan N, Neural language correction with character-based attention[J]. arXiv preprint arXiv,1603.09727, 2016.

Hu L, Tang Y, Wu X, Considering optimization of English grammar error correction based on neural network[J]. Neural Computing and Applications, 2021.

Shi Y, Research on English Grammar Error Correction Technology Based on BLSTM Sequence An-notation[J]. Asian Conference on Artificial Intelligence Technology, 2021.

Chollampatt S, H. T. Ng, A multilayer convolutional encoder-decoder neural network for grammatical error correction[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1).

Xie Z, Genthial G, Xie S, Noising and denoising natural language: Diverse backtranslation for grammar correction[J]. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018, (Volume 1): 619-628.

Zhao W, Wang L, Shen K, Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data[J]. arXiv preprint arXiv, 1903.00138, 2019.

Vaswani A, Shazeer N, Parmar N, Attention is all you need[J]. arXiv preprint arXiv, 1706.03762, 2017 .

Cheng L, Ben P, Qiao Y, Research on Automatic Error Correction Method in English Writing Based on Deep Neural Network[J]. Computational Intelligence and Neuroscience, 2022.

Zhou S, Liu W, English Grammar Error Correction Algorithm Based on Classification Model[J]. Complexity, 2021.

Tarnavskyi M, Chernodub A, Omelianchuk K, Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction[J]. arXiv preprint arXiv, 2203.13064, 2022.

Ge Y F, Orlowska M, Cao J, et al. MDDE: multitasking distributed differential evolution for privacy-preserving database fragmentation[J]. The VLDB Journal, 2022: 1-19.

Ge Y F, Yu W J, Cao J, et al. Distributed memetic algorithm for outsourced database fragmentation[J]. IEEE Transactions on Cybernetics, 2020, 51(10): 4808-4821.

Li J Y, Zhan Z H, Wang H, et al. Data-driven evolutionary algorithm with perturbation-based ensemble surrogates[J]. IEEE Transactions on Cybernetics, 2020, 51(8): 3925-3937.

Alvi A M, Siuly S, Wang H. A long short-term memory based framework for early detection of mild cognitive impairment from EEG signals[J]. IEEE Transactions on Emerging Topics in Computational Intelligence, 2022.

Siuly S, Khare S K, Bajaj V, et al. A computerized method for automatic detection of schizophrenia using EEG signals[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(11): 2390-2400.

Shi W, Chen W N, Kwong S, et al. A coevolutionary estimation of distribution algorithm for group insurance portfolio[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2021.

Keskar N S, McCann B, Varshney L R, et al. Ctrl: A conditional transformer language model for controllable generation[J]. arXiv preprint arXiv:1909.05858, 2019.

Dahlmeier D, H. T. Ng, S. M. Wu, Building a large annotated corpus of learner English: The NUS corpus of learner English, Proceedings of the eighth workshop on innovative use of NLP for building educational applications. 2013: 22-31.

Ji J, Wang Q, Toutanova K, et al. A nested attention neural hybrid model for grammatical error correction[J]. arXiv preprint arXiv:1707.02026, 2017.

Stahlberg F, Bryant C, Byrne B. Neural grammatical error correction with finite state transducers[J]. arXiv preprint arXiv:1903.10625, 2019.

Grundkiewicz R, Junczys Dowmunt M, Heafield K. Neural grammatical error correction systems with unsupervised pre-training on synthetic data[C]//Proceedings of the Fourteenth Workshop on Innova-tive Use of NLP for Building Educational Applications. 2019: 252-263.