Journal of Soft Computing and Information Technology

Journal of Soft Computing and Information Technology

Improving the Effectiveness of Open-Domain Question-Answering Systems for Answering Multi-hop Questions in Persian Language

Document Type : Persian Original Article

Authors
Aarsh Ghafouri 1
Mobina Taji 1
Hassan Naderi 1
Behrouz Minaei Bidgoli 1
mohammad reza hasani ahnagar 2
1 Computer Engineering Department, Iran University of Science and Technology, Tehran, Iran.
2 Professor in Imam Hossein University
Abstract
Today, one of the most popular and challenging tasks in natural language processing is answering complex user questions. Question-answering systems, as a new generation of search engines, receive user questions in natural language without semantic limitations and provide precise answers. In recent years, most research in the field of question-answering systems has been focused on the English language, and not much effort has been made in languages with limited resources, such as Persian. This limitation prevents these systems from efficiently handling questions in languages like Persian.
In this regard, this article aims to enhance the efficiency of question-answering systems in the Persian language by creating a dataset for answering complex multi-turn questions. Multi-hop questions, require at least two steps of reasoning to reach an answer. This dataset, called PersianMHQA, is the first open-domain question-answering dataset in Persian and includes 7,000 multi-hop questions. It was generated using the Persian Wikipedia as a knowledge source. To evaluate and benchmark this dataset, it has been fine-tuned on the latest pre-trained language models that support the Persian language.
The best results obtained based on F1 score and exact match on this dataset are 75.92% and 71.73%, respectively. These results indicate that this dataset is a powerful starting point for improving multi-hop complex question-answering for Persian language systems.
In this regard, this article aims to enhance the efficiency of question-answering systems in the Persian language by creating a dataset for answering complex multi-turn questions. Multi-hop questions, require at least two steps of reasoning to reach an answer. This dataset, called PersianMHQA, is the first open-domain question-answering dataset in Persian and includes 7,000 multi-hop questions. It was generated using the Persian Wikipedia as a knowledge source. To evaluate and benchmark this dataset, it has been fine-tuned on the latest pre-trained language models that support the Persian language.
The best results obtained based on F1 score and exact match on this dataset are 75.92% and 71.73%, respectively. These results indicate that this dataset is a powerful starting point for improving multi-hop complex question-answering for Persian language systems.
Keywords
Open Domain Question Answering
Multi-hop Question
Dataset
Persian Languages
Persian Wikipedia Encyclopedia
[1]        M. A. Calijorne Soares and F. S. Parreiras, “A Literature Review on Question Answering Techniques, Paradigms and Systems,” J. King Saud Univ. - Comput. Inf. Sci., vol. 32, no. 6, pp. 635–646, 2020, doi: 10.1016/j.jksuci.2018.08.005.
[2]        Y. Feldman and R. El-Yaniv, “Multi-hop Paragraph Retrieval for Open-domain Question Answering,” ACL 2019 - 57th Annu. Meet. Assoc. Comput. Linguist. Proc. Conf., pp. 2296–2309, 2020, doi: 10.18653/v1/p19-1222.M.B.A. Haghighat, “Biometrics for Cybersecurity and Unconstrained Environments”, Ph.D. Thesis, University of Miami, USA, 2016.
[3]        R. Etezadi and M. Shamsfard, “PeCoQ: A Dataset for Persian Complex Question Answering over Knowledge Graph,” in 2020 11th International Conference on Information and Knowledge Technology (IKT), Dec. 2020, pp. 102–106. doi: 10.1109/IKT51791.2020.9345610.
[4]        M. B. Sajadi, B. Minaei, and A. Hadian, “Farsbase: A cross-domain farsi knowledge graph.” SEMANTICS&Posters&Demos, 2018.
[5]        H. Babaei Giglou, N. Beyranvand, R. Moradi, A. M. Salehoof, and S. Bibak, “ParsSimpleQA: The Persian Simple Question Answering Dataset and System over Knowledge Graph,” in Proceedings of the 2nd International Workshop on Natural Language Processing for Digital Humanities, Nov. 2022, pp. 59– 68. [Online]. Available: https://aclanthology.org/2022.nlp4dh-1.9
[6]        K. Darvishi, N. Shahbodagh, Z. Abbasiantaeb, and S. Momtazi, “PQuAD: A Persian Question Answering Dataset,” Feb. 2022.
[7]        N. Jamali, Y. Yaghoobzadeh, and H. Faili, “PerCQA: Persian Community Question Answering Dataset.” 2021.
[8]        S. Ayoubi, https://github.com/sajjjadayobi/PersianQA.
[9]        A. Ghafouri, H. Naderi, M. Aghajani, M. Firouzmandi, “IslamicPCQA: A Dataset for Persian Multi-hop Complex Question Answering in Islamic Text Resources.” 2023.
[10]      Kazemi, J. Mozafari, and M. A. Nematbakhsh, “PersianQuAD: The Native Question Answering Dataset for the Persian Language,” IEEE Access, vol. 10, pp. 26045–26057, 2022, doi: 10.1109/ACCESS.2022.3157289.
[11]      J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pretraining of deep bidirectional transformers for language understanding,” in NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 2019, vol. 1, pp. 4171–4186. [Online]. Available: https://github.com/tensorflow/tensor2tensor
[12]      Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.” 2020.
[13]      M. Farahani, M. Gharachorloo, M. Farahani, and M. Manthouri, “ParsBERT: Transformer-based Model for Persian Language Understanding,” Neural Process. Lett., vol. 53, no. 6, pp. 3831– 3847, 2021, doi: 10.1007/s11063-021-10528-4.
[14]      N. Abadani, J. Mozafari, A. Fatemi, M. A. Nematbakhsh, and A. Kazemi, “ParSQuAD: Machine Translated SQuAD dataset for Persian Question Answering,” in 2021 7th International Conference on Web Research (ICWR), 2021, pp. 163–168. doi: 10.1109/ICWR51868.2021.9443126.
[15]      Z. Yang et al., “Hotpotqa: A dataset for diverse, explainable multi-hop question answering,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, 2020, pp. 2369–2380. doi: 10.18653/v1/d18-1259.
[16]      A. Ghafouri, M. A. Abbasi, and H. Naderi, "AriaBERT: A pre-trained Persian BERT model for natural language understanding," preprint, Research Square, Nov. 2023. [Online]. Available: https://doi.org/10.21203/rs.3.rs-3558473/v1
[17]      Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer and V. Stoyanov, "Roberta: A robustly optimized bert pretraining approach." arXiv preprint arXiv:1907.11692, 2019.
[18]      A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” arXiv preprint arXiv:1911.02116, 2019.
[19]      F. Feng, Y. Yang, D. Cer, N. Arivazhagan and W. Wang, “Language-agnostic BERT sentence embedding,” arXiv preprint arXiv:2007.01852, 2020.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Stoyanov, V. (2019). RoERTa: A robustly optimized ERT pretraining approach. arXiv
preprint arXiv:1907.11692.
[20]      Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.” 2020.
Journal of Soft Computing and Information Technology
Volume 13, Issue 1
Spring - May
Spring 2024
Pages 1-10