NLPNING ZAMONAVIY ALGORITMLARI VA KONSEPSIYALARI
##article.subject##:
NLP, pipeline konveyeri, Levenshtein masofasi, kosinus o‘xshashligi, Bag of words usuli, TF-IDF algoritmi.##article.abstract##
Tabiiy tilni qayta ishlash (NLP) algoritmlari insonning til ma’lumotlarini, shu jumladan, strukturlanmagan matn ma’lumotlarini qayta ishlashga xizmat qiladi. Bugungi kunda NLP algoritmlari til qoidalariga asoslangan, statistik va sun’iy intellektga asoslangan yondashuvlar asosida ishlab chiqiladi. Til qoidalariga asoslangan yondashuv asosida asosan NLP vazifalari uchun lingvistik bazalarni shakllantirish va til korpuslarida razmetkalash amallari bajariladi. Statistik algoritmlar mashinalarga inson tillarini o‘qish, tushunish va ma’no olish imkonini beradi hamda katta hajmdagi (bigdata) matnlarni qayta ishlashga asoslanadi. Statistik algoritmlardan nutqni tanib olish, mashina tarjimasi, hissiyotlarni tahlil qilish, matnlarni tasniflash va tahlil qilish kabi koʻplab NLP vazifalarda qo‘llaniladi. Bugungi kunda mashinali o‘rganish (ML) algoritmlarining CNN va RNN texnologiyalari asoslangan chuqur o‘rganish modellari mavjud NLP tizimlarini “o‘rganish” imkonini beradi va katta hajmdagi strukturlanmagan matnlarni yanada aniqroq qayta ishlash imkonini beradi. Ushbu maqolada bugungi kundagi NLPning zamonaviy algoritmlari va konsepsiyalari haqida fikr-mulohaza yuritiladi va o‘zbek tilidagi matnlarni ushbu algortimlar asosida qayta ishlash usullari keltiriladi.
Библиографические ссылки
Zhou, M., Duan, N., Liu, S., & Shum, H.-Y. (2020). Progress in Neural NLP: Modeling, Learning, and Reasoning. Engineering, 6(3), 275–290. https://doi.org/10.1016/j.eng.2019.12.014
Bulusu, A., & Sucharita, V. (2019). Research on machine learning techniques for POS tagging in NLP. International Journal of Recent Technology and Engineering, 8(1 Special Issue 4).
Ghorpade-Aher, J., Kontamwar, R., Kukreja, S., Karpe, T., & Kakkad, S. (2019). An overview of NLP based Chatbots. Universal Review, VIII(II).
Bhagwat, V. A. (2018). Deep Learning for ChatBots. Scholarworks.Sjsu.Edu.
Becquin, G. (2020). End-to-end NLP Pipelines in Rust. https://doi.org/10.18653/v1/2020.nlposs-1.4
Tenney, I., Das, D., & Pavlick, E. (2020). BERT rediscovers the classical NLP pipeline. ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. https://doi.org/10.18653/v1/p19-1452
Attardi, G. (2015). DeepNL: A deep learning NLP pipeline. 1st Workshop on Vector Space Modeling for Natural Language Processing, VS 2015 at the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2015. https://doi.org/10.3115/v1/w15-1515
B.Elov. Tabiiy tilni qayta ishlash (NLP)da spacy modulidan foydalanish. Илм-фан ва инновaцион ривожланиш, 2022 (4). 41-55.
N.Xudayberganov, Sh.Hasanov. Tabiiy tilni qayta ishlashda so‘zlar orasidagi masofani aniqlash algoritmidan foydalanish // O‘zbekiston: til va madaniyat. Amaliy filologiya masalalari. 2022-yil 5 (2) son. − B. 69-83.
Yujian, L., & Bo, L. (2007). A normalized Levenshtein distance metric. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6). https://doi.org/10.1109/TPAMI.2007.1078
Zhao, C., & Sahni, S. (2019). String correction using the Damerau-Levenshtein distance. BMC Bioinformatics, 20. https://doi.org/10.1186/s12859-019-2819-0
Behara, K. N. S., Bhaskar, A., & Chung, E. (2020). A novel approach for the structural comparison of origin-destination matrices: Levenshtein distance. Transportation Research Part C: Emerging Technologies, 111. https://doi.org/10.1016/j.trc.2020.01.005
Darmalaksana, W., Slamet, C., Zulfikar, W. B., Fadillah, I. F., Maylawati, D. S. adillah, & Ali, H. (2020). Latent semantic analysis and cosine similarity for hadith search engine. Telkomnika (Telecommunication Computing Electronics and Control), 18(1). https://doi.org/10.12928/TELKOMNIKA.V18I1.14874
Stecanella, B. (2019). What is TF IDF? MonkeyLearn.
Jalilifard, A., Caridá, V. F., Mansano, A. F., Cristo, R. S., & da Fonseca, F. P. C. (2021). Semantic Sensitive TF-IDF to Determine Word Relevance in Documents. Lecture Notes in Electrical Engineering, 736 LNEE. https://doi.org/10.1007/978-981-33-6987-0_27
Pietro, M. di. (2020). Text Classification with NLP: Tf-Idf vs Word2Vec vs BERT. Medium.
B. Elov, Z. Husainova, N.Xudayberganov. (2022). O‘zbek tili korpusi matnlari uchun TF-IDF statistik ko‘rsatkichni hisoblash. International scientific journal of “Science and Innovation”. Issue 8, https://doi.org/10.5281/zenodo.7440059
B.Elov, Z.Xusainova, N.Xudayberganov. (2022). Tabiiy tilni qayta ishlashda Bag of Words algoritmidan foydalanish. Til va madaniyat, Amaliy filologiya masalalari. Vol.2(5). − B. 35-50.
Zhang, Y., Jin, R., & Zhou, Z. H. (2010). Understanding bag-of-words model: A statistical framework. International Journal of Machine Learning and Cybernetics, 1(1–4). https://doi.org/10.1007/s13042-010-0001-0
Yadav, A. K., & Borgohain, S. K. (2015). Sentence generation from a bag of words using N-gram model. Proceedings of 2014 IEEE International Conference on Advanced Communication, Control and Computing Technologies, ICACCCT 2014.
https://doi.org/10.1109/ICACCCT.2014.7019414
Ariffin, S. N. A. N., & Tiun, S. (2020). Rule-based text normalization for malay social media texts. International Journal of Advanced Computer Science and Applications, 11(10).
https://doi.org/10.14569/IJACSA.2020.0111021
Webster, J. J., & Kit, C. (1992). Tokenization as the initial phase in NLP. https://doi.org/10.3115/992424.992434
Rai, A., & Borah, S. (2021). Study of various methods for tokenization. In Lecture Notes in Networks and Systems (Vol. 137). https://doi.org/10.1007/978-981-15-6198-6_18
Hafsa Jabeen. (2018). Stemming and Lemmatization in Python. Towardsdatascience.
Balakrishnan, V., & Ethel, L.-Y. (2014). Stemming and Lemmatization: A Comparison of Retrieval Performances. Lecture Notes on Software Engineering, 2(3). https://doi.org/10.7763/lnse.2014.v2.134
Khyani, D., S, S. B., M, N. N., & M, D. B. (2021). An Interpretation of Lemmatization and Stemming in Natural Language Processing. Journal of University of Shanghai for Science and Technology, 22(10).
B.Elov, Sh.Hamroyeva, D.Elova. (2022). Morfologik analizatorni yaratish usullari, O‘zbekiston: til va madaniyat (Amaliy filologiya), 2022, 5(1).
B.Elov, Sh.Hamroyeva, X.Axmedova. (2022). Methods for creating a morphological analyzer, 14th International Conference on Intellegent Human Computer Interaction, IHCI 2022, 19-23 October 2022, Tashkent.
Z.Xusainova. Tokenizatsiya algoritmlari.Ilmiy innovatsion jurnal: " Filologik tadqiqotlar:Til,adabiyot, ta'lim". 2022/№5-6. b.73-76
Rudkowsky, E., Haselmayer, M., Wastian, M., Jenny, M., Emrich, Š., & Sedlmair, M. (2018). More than Bags of Words: Sentiment Analysis with Word Embeddings. Communication Methods and Measures, 12(2–3). https://doi.org/10.1080/19312458.2018.1455817
Wang, B., Wang, A., Chen, F., Wang, Y., & Kuo, C. C. J. (2019). Evaluating word embedding models: Methods and experimental results. In APSIPA Transactions on Signal and Information Processing (Vol. 8). https://doi.org/10.1017/ATSIP.2019.12
