NLPNING ZAMONAVIY ALGORITMLARI VA KONSEPSIYALARI
Keywords:
NLP, pipeline pipeline, Levenshtein distance, cosine similarity, Bag of words method, TF-IDF algorithm.Abstract
Natural language processing (NLP) algorithms serve to process human language data, including unstructured text data. Today, NLP algorithms are developed based on language rule-based, statistical and artificial intelligence approaches. Based on the approach based on language rules, the formation of linguistic bases for NLP tasks and the operations of classification in language corpora are performed. Statistical algorithms allow machines to read, understand and derive meaning from human languages and are based on processing large volumes of (bigdata) texts. Statistical algorithms are used in many NLP tasks such as speech recognition, machine translation, sentiment analysis, text classification and analysis. Today, deep learning models of machine learning (ML) algorithms based on CNN and RNN technologies allow to "learn" existing NLP systems and allow more accurate processing of large volumes of unstructured texts. This article discusses modern algorithms and concepts of NLP today, and methods of processing texts in the Uzbek language based on these algorithms are given.
References
Zhou, M., Duan, N., Liu, S., & Shum, H.-Y. (2020). Progress in Neural NLP: Modeling, Learning, and Reasoning. Engineering, 6(3), 275–290. https://doi.org/10.1016/j.eng.2019.12.014
Bulusu, A., & Sucharita, V. (2019). Research on machine learning techniques for POS tagging in NLP. International Journal of Recent Technology and Engineering, 8(1 Special Issue 4).
Ghorpade-Aher, J., Kontamwar, R., Kukreja, S., Karpe, T., & Kakkad, S. (2019). An overview of NLP based Chatbots. Universal Review, VIII(II).
Bhagwat, V. A. (2018). Deep Learning for ChatBots. Scholarworks.Sjsu.Edu.
Becquin, G. (2020). End-to-end NLP Pipelines in Rust. https://doi.org/10.18653/v1/2020.nlposs-1.4
Tenney, I., Das, D., & Pavlick, E. (2020). BERT rediscovers the classical NLP pipeline. ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. https://doi.org/10.18653/v1/p19-1452
Attardi, G. (2015). DeepNL: A deep learning NLP pipeline. 1st Workshop on Vector Space Modeling for Natural Language Processing, VS 2015 at the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2015. https://doi.org/10.3115/v1/w15-1515
B.Elov. Tabiiy tilni qayta ishlash (NLP)da spacy modulidan foydalanish. Илм-фан ва инновaцион ривожланиш, 2022 (4). 41-55.
N.Xudayberganov, Sh.Hasanov. Tabiiy tilni qayta ishlashda so‘zlar orasidagi masofani aniqlash algoritmidan foydalanish // O‘zbekiston: til va madaniyat. Amaliy filologiya masalalari. 2022-yil 5 (2) son. − B. 69-83.
Yujian, L., & Bo, L. (2007). A normalized Levenshtein distance metric. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6). https://doi.org/10.1109/TPAMI.2007.1078
Zhao, C., & Sahni, S. (2019). String correction using the Damerau-Levenshtein distance. BMC Bioinformatics, 20. https://doi.org/10.1186/s12859-019-2819-0
Behara, K. N. S., Bhaskar, A., & Chung, E. (2020). A novel approach for the structural comparison of origin-destination matrices: Levenshtein distance. Transportation Research Part C: Emerging Technologies, 111. https://doi.org/10.1016/j.trc.2020.01.005
Darmalaksana, W., Slamet, C., Zulfikar, W. B., Fadillah, I. F., Maylawati, D. S. adillah, & Ali, H. (2020). Latent semantic analysis and cosine similarity for hadith search engine. Telkomnika (Telecommunication Computing Electronics and Control), 18(1). https://doi.org/10.12928/TELKOMNIKA.V18I1.14874
Stecanella, B. (2019). What is TF IDF? MonkeyLearn.
Jalilifard, A., Caridá, V. F., Mansano, A. F., Cristo, R. S., & da Fonseca, F. P. C. (2021). Semantic Sensitive TF-IDF to Determine Word Relevance in Documents. Lecture Notes in Electrical Engineering, 736 LNEE. https://doi.org/10.1007/978-981-33-6987-0_27
Pietro, M. di. (2020). Text Classification with NLP: Tf-Idf vs Word2Vec vs BERT. Medium.
B. Elov, Z. Husainova, N.Xudayberganov. (2022). O‘zbek tili korpusi matnlari uchun TF-IDF statistik ko‘rsatkichni hisoblash. International scientific journal of “Science and Innovation”. Issue 8, https://doi.org/10.5281/zenodo.7440059
B.Elov, Z.Xusainova, N.Xudayberganov. (2022). Tabiiy tilni qayta ishlashda Bag of Words algoritmidan foydalanish. Til va madaniyat, Amaliy filologiya masalalari. Vol.2(5). − B. 35-50.
Zhang, Y., Jin, R., & Zhou, Z. H. (2010). Understanding bag-of-words model: A statistical framework. International Journal of Machine Learning and Cybernetics, 1(1–4). https://doi.org/10.1007/s13042-010-0001-0
Yadav, A. K., & Borgohain, S. K. (2015). Sentence generation from a bag of words using N-gram model. Proceedings of 2014 IEEE International Conference on Advanced Communication, Control and Computing Technologies, ICACCCT 2014.
https://doi.org/10.1109/ICACCCT.2014.7019414
Ariffin, S. N. A. N., & Tiun, S. (2020). Rule-based text normalization for malay social media texts. International Journal of Advanced Computer Science and Applications, 11(10).
https://doi.org/10.14569/IJACSA.2020.0111021
Webster, J. J., & Kit, C. (1992). Tokenization as the initial phase in NLP. https://doi.org/10.3115/992424.992434
Rai, A., & Borah, S. (2021). Study of various methods for tokenization. In Lecture Notes in Networks and Systems (Vol. 137). https://doi.org/10.1007/978-981-15-6198-6_18
Hafsa Jabeen. (2018). Stemming and Lemmatization in Python. Towardsdatascience.
Balakrishnan, V., & Ethel, L.-Y. (2014). Stemming and Lemmatization: A Comparison of Retrieval Performances. Lecture Notes on Software Engineering, 2(3). https://doi.org/10.7763/lnse.2014.v2.134
Khyani, D., S, S. B., M, N. N., & M, D. B. (2021). An Interpretation of Lemmatization and Stemming in Natural Language Processing. Journal of University of Shanghai for Science and Technology, 22(10).
B.Elov, Sh.Hamroyeva, D.Elova. (2022). Morfologik analizatorni yaratish usullari, O‘zbekiston: til va madaniyat (Amaliy filologiya), 2022, 5(1).
B.Elov, Sh.Hamroyeva, X.Axmedova. (2022). Methods for creating a morphological analyzer, 14th International Conference on Intellegent Human Computer Interaction, IHCI 2022, 19-23 October 2022, Tashkent.
Z.Xusainova. Tokenizatsiya algoritmlari.Ilmiy innovatsion jurnal: " Filologik tadqiqotlar:Til,adabiyot, ta'lim". 2022/№5-6. b.73-76
Rudkowsky, E., Haselmayer, M., Wastian, M., Jenny, M., Emrich, Š., & Sedlmair, M. (2018). More than Bags of Words: Sentiment Analysis with Word Embeddings. Communication Methods and Measures, 12(2–3). https://doi.org/10.1080/19312458.2018.1455817
Wang, B., Wang, A., Chen, F., Wang, Y., & Kuo, C. C. J. (2019). Evaluating word embedding models: Methods and experimental results. In APSIPA Transactions on Signal and Information Processing (Vol. 8). https://doi.org/10.1017/ATSIP.2019.12
