Авторы:
Издательство:
Дата издания
09.10.2024На каком языке издана
ISSN
03029743Область исследования
Аннотация (краткое описание)
As the volume of data continues to surge, researchers are confronted with the challenge of extracting meaningful insights from this wealth of information. Despite rapid advancements in Natural Language Processing (NLP) techniques in the AI industry, there remain gaps and opportunities for further exploration, particularly in the realm of data recombination techniques and methods. This paper proposes a novel text recombination method to facilitate the generation of recombined words from a given text. The process commences with a ’stanza’ model, which identifies and compiles Named Entity Recognitions (NERs) into a list. These NERs are then cross-referenced with Wikipedia pages to retrieve relevant information, thereby enhancing entity understanding and analysis. The ensuing step involves preprocessing the output text from the previous stage, generating a list of unique words while eliminating stop words. This preprocessing stage serves to remove noise and focus on meaningful words, laying the groundwork for more effective clustering. To enable clustering, we employ vector embeddings, representing words in a 2-dimensional space, rendering them suitable for clustering techniques. Notably, the proposed method further enhances results by re-clustering words after applying K-Means, thereby identifying the most fitting candidate words for recombination. Comparatively, this method outperforms large language models (LLMs) due to its incorporation of NER information, utilization of Wikipedia pages, and effective preprocessing techniques. Unlike LLMs, which operate as resource-intensive black boxes on static data, this method benefits from real-time information access and knowledge base updates. Furthermore, each stage of the process is visualized to control the progress correctly. Thus, due to the plots of word clusterization, the proposed text recombination approach showed positive results.DOI
10.1007/978-3-031-47640-2_1Тип публикаций
Практическое значение
Character recognition; Embeddings; K-means clustering; Knowledge based systems; Natural language processing systems; Vector spacesКвартиль
Источник квартиля
SCImago (Scopus)Данный документ располагается в коллекциях
Международные публикации ученых SU
Scopus
Файлы документа
Документ не загружен