Search

The digitization of operation and maintenance in the intelligent power grid equipment relies on a diverse array of information for smart decision-making. In the domain of intelligent decision generation, proficiency is contingent upon extensive learning from copious amounts of text. This necessitates not only robust processing capabilities but also a high level of specialization. In addressing situations where authorization is lacking, pre-trained language models (PLMs) have already provided ideas when confronted with specialized domains or tasks. In consideration of the complexity of textual content in the field of the power grid, which encompasses a multitude of specialized knowledge and involves an abundance of proprietary terminology, we have undertaken an exploration of pre-trained model specialization using the power grid domain as an example, specifically for the task of generating maintenance strategies. A two-stage fine-tuning approach (P2FT) is employed, utilizing a large-scale pre-training model specifically designed for natural language processing. The efficacy and practical value of this method were evaluated through multiple metrics, juxtaposed with other advanced approaches involving low-parameter or parameter-free fine-tuning methods. Through a meticulous analysis and validation of experimental outcomes, we have corroborated the feasibility and practical application value of employing this approach for pre-trained model specialization. Additionally, it has furnished valuable guidance for text generation within both the Chinese language domain and the power grid domain. [ABSTRACT FROM AUTHOR]

As consumer electronics evolve towards greater intelligence, their automation and complexity also increase, making it difficult for users to diagnose faults when they occur. To address the problem where users, relying solely on their own knowledge, struggle to diagnose faults in consumer electronics promptly and accurately, we propose a multimodal knowledge graph-based text generation method. Our method begins by using deep learning models like the Residual Network (ResNet) and Bidirectional Encoder Representations from Transformers (BERT) to extract features from user-provided fault information, which can include images, text, audio, and even olfactory data. These multimodal features are then combined to form a comprehensive representation. The fused features are fed into a graph convolutional network (GCN) for fault inference, identifying potential fault nodes in the electronics. These fault nodes are subsequently fed into a pre-constructed knowledge graph to determine the final diagnosis. Finally, this information is processed through the Bias-term Fine-tuning (BitFit) enhanced Chinese Pre-trained Transformer (CPT) model, which generates the final fault diagnosis text for the user. The experimental results show that our proposed method achieves a 4.4% improvement over baseline methods, reaching a fault diagnosis accuracy of 98.4%. Our approach effectively leverages multimodal fault information, addressing the challenges users face in diagnosing faults through the integration of graph convolutional network and knowledge graph technologies. [ABSTRACT FROM AUTHOR]

This research presents an approach to automatic multiple-choice question (MCQ) generation in the Spanish language, using mT5-based models. The process encompasses three crucial tasks: candidate answer extraction, answer-aware question generation, and distractor generation. A methodical pipeline is structured to seamlessly integrate these tasks, converting an input text into a systematic questionnaire. For model fine-tuning, the Stanford Question Answering Dataset is employed for the first two tasks, while a combination of three different multiple-choice question datasets, translated automatically into Spanish, is used for the distractor generation task. The efficiency of the models is then evaluated by using a triad of metrics, namely BLEU, ROUGE-L, and cosine similarity. The outcomes indicate a marginal deviation from the baseline model in the question generation task but demonstrate superior performance in the distractor generation task. Importantly, this research emphasizes the potential and effectiveness of language models for automating MCQ generation, providing a valuable contribution to the field and enhancing the understanding and application of such models in the context of the Spanish language. [ABSTRACT FROM AUTHOR]

Today's news has the characteristics of short text, frequent release, timeliness, etc. A media account releases dozens of news in a day. Developing suitable and attractive headlines for large volumes of news has become a major part of the work of media workers. Media workers need a system that automatically generates short text headlines to relieve their stress. To solve this problem, this study proposes a short text news title generation model. The model adopts sequence-to-sequence structure, using pre-trained language model and layered self-attention decoder in encoder and decoder respectively. In order to make the generated headlines contain the key information of the original news, a staged training method based on LCSTS data set and Weibo4 data set is proposed, and the model learns to extract the key news information and construct a stylized expression from the two data sets respectively, so that the generated headlines can accurately express the core content of the news and attract readers. [ABSTRACT FROM AUTHOR]

Copyright of Procesamiento del Lenguaje Natural is the property of Sociedad Espanola para el Procesamiento del Lenguaje Natural and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Journal of Frontiers of Computer Science & Technology is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Sentiment analysis is a vital tool in natural language processing (NLP), enabling the interpretation and understanding of opinions expressed in textual data. Traditional sentiment analysis methods, often limited to document or sentence-level analysis, primarily focus on identifying the sentiment without generating detailed sentiment text expressions. To address this limitation, we propose a novel Aspect-Specific Sentiment Expression Generation (ASSEG) model. Unlike traditional approaches, the ASSEG model leverages advanced text generation models, such as GPT-2 and T5, to automatically generate sentiment expressions tailored to diverse aspects of entities discussed in the text. The key innovation of our approach lies in the integration of aspect-specific attention mechanisms, which enable the model to effectively identify and prioritize aspects within the text, generating coherent and contextually relevant sentiment expressions. Our methodology includes using Recurrent Generative Adversarial Networks (RGANs) for data augmentation, addressing data imbalance issues, and enhancing the robustness of sentiment analysis models. Experimental evaluations were conducted on domain-specific datasets, including laptop and restaurant reviews. Our experimental evaluations on domain-specific datasets, including laptop and restaurant reviews, demonstrate the superior performance of our ASSEG model. The GPT-2 model achieved an accuracy of 75% and 65%, and an F1 score of 77% and 65% for restaurant and laptop datasets, respectively. Meanwhile, the T5 model outperformed GPT-2, achieving an accuracy of 85% and 75%, and an F1 score of 83% and 74% for restaurant and laptop datasets, respectively. These results highlight the potential of the ASSEG model, offering deeper insights into user opinions by generating detailed and contextually relevant sentiment expressions. [ABSTRACT FROM AUTHOR]

Objectives To enhance the performance of large language models (LLMs) in biomedical natural language processing (BioNLP) by introducing a domain-specific instruction dataset and examining its impact when combined with multi-task learning principles. Materials and Methods We created the BioInstruct , comprising 25 005 instructions to instruction-tune LLMs (LLaMA 1 and 2, 7B and 13B version). The instructions were created by prompting the GPT-4 language model with 3-seed samples randomly drawn from an 80 human curated instructions. We employed Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. We then evaluated these instruction-tuned LLMs on several BioNLP tasks, which can be grouped into 3 major categories: question answering (QA), information extraction (IE), and text generation (GEN). We also examined whether categories (eg, QA, IE, and generation) of instructions impact model performance. Results and Discussion Comparing with LLMs without instruction-tuned, our instruction-tuned LLMs demonstrated marked performance gains: 17.3% in QA on average accuracy metric, 5.7% in IE on average F1 metric, and 96% in Generation tasks on average GPT-4 score metric. Our 7B-parameter instruction-tuned LLaMA 1 model was competitive or even surpassed other LLMs in the biomedical domain that were also fine-tuned from LLaMA 1 with vast domain-specific data or a variety of tasks. Our results also show that the performance gain is significantly higher when instruction fine-tuning is conducted with closely related tasks. Our findings align with the observations of multi-task learning, suggesting the synergies between 2 tasks. Conclusion The BioInstruct dataset serves as a valuable resource and instruction tuned LLMs lead to the best performing BioNLP applications. [ABSTRACT FROM AUTHOR]

Objective To solve major clinical natural language processing (NLP) tasks using a unified text-to-text learning architecture based on a generative large language model (LLM) via prompt tuning. Methods We formulated 7 key clinical NLP tasks as text-to-text learning and solved them using one unified generative clinical LLM, GatorTronGPT, developed using GPT-3 architecture and trained with up to 20 billion parameters. We adopted soft prompts (ie, trainable vectors) with frozen LLM, where the LLM parameters were not updated (ie, frozen) and only the vectors of soft prompts were updated, known as prompt tuning. We added additional soft prompts as a prefix to the input layer, which were optimized during the prompt tuning. We evaluated the proposed method using 7 clinical NLP tasks and compared them with previous task-specific solutions based on Transformer models. Results and Conclusion The proposed approach achieved state-of-the-art performance for 5 out of 7 major clinical NLP tasks using one unified generative LLM. Our approach outperformed previous task-specific transformer models by ∼3% for concept extraction and 7% for relation extraction applied to social determinants of health, 3.4% for clinical concept normalization, 3.4%-10% for clinical abbreviation disambiguation, and 5.5%-9% for natural language inference. Our approach also outperformed a previously developed prompt-based machine reading comprehension (MRC) model, GatorTron-MRC, for clinical concept and relation extraction. The proposed approach can deliver the "one model for all" promise from training to deployment using a unified generative LLM. [ABSTRACT FROM AUTHOR]

Abstract The digitization of operation and maintenance in the intelligent power grid equipment relies on a diverse array of information for smart decision-making. In the domain of intelligent decision generation, proficiency is contingent upon extensive learning from copious amounts of text. This necessitates not only robust processing capabilities but also a high level of specialization. In addressing situations where authorization is lacking, pre-trained language models (PLMs) have already provided ideas when confronted with specialized domains or tasks. In consideration of the complexity of textual content in the field of the power grid, which encompasses a multitude of specialized knowledge and involves an abundance of proprietary terminology, we have undertaken an exploration of pre-trained model specialization using the power grid domain as an example, specifically for the task of generating maintenance strategies. A two-stage fine-tuning approach (P2FT) is employed, utilizing a large-scale pre-training model specifically designed for natural language processing. The efficacy and practical value of this method were evaluated through multiple metrics, juxtaposed with other advanced approaches involving low-parameter or parameter-free fine-tuning methods. Through a meticulous analysis and validation of experimental outcomes, we have corroborated the feasibility and practical application value of employing this approach for pre-trained model specialization. Additionally, it has furnished valuable guidance for text generation within both the Chinese language domain and the power grid domain.

In the Chinese radiology domain, radiology reports serve as a crucial basis for clinical decision-making. Therefore, utilizing natural language processing (NLP) technology to understand and learn from the textual content of radiology reports, thereby aiding radiological clinical work, has become an important research direction in this domain. However, when dealing with the natural language classification and generation tasks based on Chinese radiology reports using traditional methods, there are still challenges such as a lack of training corpora, privacy concerns, and poor model generalization capabilities, leading to insufficient overall performance. To address these issues, a solution for natural language tasks in the Chinese radiology domain based on locally efficient fine-tuning large language models is proposed. By collecting and constructing a large-scale, high-quality dataset for natural language tasks in the Chinese radiology reports, and employing the LoRA efficient fine-tuning method for supervised fine-tuning training of the open-source large language model Baichuan2, the “RadGPT” capable of solving four types of clinical tasks in the Chinese radiology domain simultaneously is proposed. A set of evaluation systems for natural language classification and generation tasks in the Chinese radiology domain is introduced. Multiple sets of experiments are conducted on three types of radiology report datasets from two centers, and comparisons are made with several typical existing methods. The results demonstrate that the proposed method performs better in terms of classification performance, text summarization and expansion capabilities, and model generalization.

A text generation based multidisciplinary problem identification method is proposed, which does not rely on a large amount of data annotation.

The text generation technique employs the transformation of the word document from the source to the targeted document based on the sequence to sequence generation. Video captioning, language identification, image captioning, recognition of speech, machine translation, and several other natural language generations are the application areas of the text generation techniques. The Electroencephalographic (EEG) signals record brain activity and are considered the source of information for using the brain-computer interface. Several kinds of research were developed for text generation. The most challenging task is more accurate text generation by considering the large contextual information and the significant features for generating the text. Hence, in this research, text generation using Folded deep learning is proposed for generating the text through text prediction and suggestion through the non-invasive technique. The EEG signal recorded from the patients is utilized for the prediction of the first letter using the proposed Folded Ensemble Deep convolutional neural network (DeepCNN), in which the hybrid ensemble activation function along with the folded concept in validating the training data to obtain the network stability and to solve the class imbalance issue. Then, the next letter suggestion is employed using the proposed Folded Ensemble Bidirectional long short-term memory (BiLSTM) approach based on the eye-blink criteria for generating the sequence-to-sequence text generation. The enhanced performance is evaluated using accuracy, precision, and recall and acquired the maximal values of 97.22%, 98.00%, and 98.00%, respectively. The proposed method can be utilized for real-time processing applications due to its non-invasive nature. [ABSTRACT FROM AUTHOR]

Copyright of Inonu University Journal of the Faculty of Education (INUJFE) is the property of Inonu University Journal of the Faculty of Education and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Automatic text summarization aims to extract the main statements from text information for the purpose of compressing information. Existing generative automatic summarization methods do not take full advantage of the pre-trained model to learn the semantics of the original text, resulting in the loss of important information in the generated content, when the data set with a small number of samples is often prone to overfitting. In order to solve such problems and obtain better fine-tuning performance, the pre-trained model mT5(multilingual T5) is used as a baseline to improve the learning ability of the model by combining R-drop(Regularized dropout) with reinforced regularity for model fine-tuning, and Sparse softmax is used to reduce the ambiguity of prediction generation to ensure the accuracy of the output. The model calculates BLEU(Bilingual Evaluation Understudy) for hyperparameter test on Chinese data sets LCSTS and CSL, and uses Rouge as evaluation index to evaluate data sets of different orders of magnitude. The experimental results show that the optimized pre-trained model can better learn the semantic representation of the original text, and the model can maintain a good fit in the small samples and generate more practical results. [ABSTRACT FROM AUTHOR]

There is a pressing need for healthcare conversational agents with domain-specific expertise to ensure the provision of accurate and reliable information tailored to specific medical contexts. Moreover, there is a notable gap in research ensuring the credibility and trustworthiness of the information provided by these healthcare agents, particularly in critical scenarios such as medical emergencies. Pilgrims come from diverse cultural and linguistic backgrounds, often facing difficulties in accessing medical advice and information. Establishing an AI-powered multilingual chatbot can bridge this gap by providing readily available medical guidance and support, contributing to the well-being and safety of pilgrims. In this paper, we present a comprehensive methodology aimed at enhancing the reliability and efficacy of healthcare conversational agents, with a specific focus on addressing the needs of Hajj pilgrims. Our approach leverages domain-specific fine-tuning techniques on a large language model, alongside synthetic data augmentation strategies, to optimize performance in delivering contextually relevant healthcare information by introducing the HajjHealthQA dataset. Additionally, we employ a retrieval-augmented generation (RAG) module as a crucial component to validate uncertain generated responses, which improves model performance by 5%. Moreover, we train a secondary AI agent on a well-known health fact-checking dataset and use it to validate medical information in the generated responses. Our approach significantly elevates the chatbot's accuracy, demonstrating its adaptability to a wide range of pilgrim queries. We evaluate the chatbot's performance using quantitative and qualitative metrics, highlighting its proficiency in generating accurate responses and achieve competitive results compared to state-of-the-art models, in addition to mitigating the risk of misinformation and providing users with trustworthy health information. [ABSTRACT FROM AUTHOR]

In the realm of the video captioning field, acquiring large amounts of high-quality aligned video–text pairs remains laborious, impeding its practical applications. Therefore, we explore the modelling techniques for unsupervised video captioning. Using text inputs similar to the video representation to generate captions has been a successful unsupervised video captioning generation strategy in the past. However, this setting relies solely on the textual data for training, neglecting vital visual cues related to the spatio-temporal appearance within the video. The absence of visual information increases the risk of generating erroneous video captions. In view of this, we propose a novel unsupervised video captioning method that introduces visual information related to text features keywords to implicitly enhance training for text generation tasks. Simultaneously, our method incorporates sentence to explicitly augment the training process. our method injects additional implicit visual features and explicit keywords into the model, Which can inject the generated captions with more accurate semantics. the experimental analysis demonstrates the merit of the proposed formulation, achieving superior performance against the state-of-the-art unsupervised studies. • Contrast learning is used to minimise the disparities between pseudo-text labels and video features. • Visual clues are aligned with the text generator for consistent semantic enhancement. • Leveraging the Found within the sentences for semantic preservation. • Outperforming existing unsupervised video captioning approaches. [ABSTRACT FROM AUTHOR]

Generating diverse and factual text is challenging and is receiving increasing attention. By sampling from the latent space, variational autoencoder-based models have recently enhanced the diversity of generated text. However, existing research predominantly depends on summarization models to offer paragraph-level semantic information for enhancing factual correctness. The challenge lies in effectively generating factual text using sentence-level variational autoencoder-based models. In this paper, a novel model called fact-aware conditional variational autoencoder is proposed to balance the factual correctness and diversity of generated text. Specifically, our model encodes the input sentences and uses them as facts to build a conditional variational autoencoder network. By training a conditional variational autoencoder network, the model is enabled to generate text based on input facts. Building upon this foundation, the input text is passed to the discriminator along with the generated text. By employing adversarial training, the model is encouraged to generate text that is indistinguishable to the discriminator, thereby enhancing the quality of the generated text. To further improve the factual correctness, inspired by the natural language inference system, the entailment recognition task is introduced to be trained together with the discriminator via multi-task learning. Moreover, based on the entailment recognition results, a penalty term is further proposed to reconstruct the loss of our model, forcing the generator to generate text consistent with the facts. Experimental results demonstrate that compared with competitive models, our model has achieved substantial improvements in both the quality and factual correctness of the text, despite only sacrificing a small amount of diversity. Furthermore, when considering a comprehensive evaluation of diversity and quality metrics, our model has also demonstrated the best performance. [ABSTRACT FROM AUTHOR]

The meteorological service texts is a window for the organizing committee, judges, and staff of the teams to obtain meteorological information about their related work, providing the necessary meteorological protection for the smooth holding of the event and the activities during the event. The existing text production requires manual writing and review, which is not efficient, while the fully automated text generation mainly relies on templates and has a fixed form. To address these problems, a knowledge graph-based texts generation method of Winter Olympics event meteorological services was proposed in combination with natural language processing technology. Focusing on content analysis and feature extraction from historical event meteorological service texts, the knowledge graph of alpine skiing events was constructed using meteorological data and historical event information. The method generated meteorological description texts based on real-time meteorological data and manuscript templates, and then obtained event impact results and generates corresponding texts based on knowledge graph query inference techniques. The experimental results show that the automatic generation of meteorological service text have good accuracy and readability, which helps the smooth promotion of the Winter Olympic events, and the text generation method has good application prospects for the special area. [ABSTRACT FROM AUTHOR]

In this study, it is aimed to produce meaningful words in accordance with character-based Turkish grammar rules by using encoder-decoder and attention architecture, which are deep learning algorithms. The results of the developed model are compared with the results of LSTM and GRU models, which are other deep learning algorithms. It is seen that the language models created with LSTM and GRU models give similar results at 100 and 200 epoch values and at different threshold values of the temperature sampling method. Among these models, the GRU model gives the highest success value with 88.40% at 200 epochs and 0.5 temperature threshold value. The encoder-decoder and attention language model developed for this study gives the highest success value of 91.90% at 100 and 200 epoch values and at different threshold values of the temperature sampling method at 200 epoch and 0.5 temperature threshold value. At the end of the experiments, the encoder-decoder and attention architecture model showed an average of 2.83% more success than the LSTM model and an average of 0.19% more success than the GRU model. [ABSTRACT FROM AUTHOR]

Deep learning is revolutionizing radiology report generation (RRG) with the adoption of vision encoder–decoder (VED) frameworks, which transform radiographs into detailed medical reports. Traditional methods, however, often generate reports of limited diversity and struggle with generalization. Our research introduces reinforcement learning and text augmentation to tackle these issues, significantly improving report quality and variability. By employing RadGraph as a reward metric and innovating in text augmentation, we surpass existing benchmarks like BLEU4, ROUGE-L, F1CheXbert, and RadGraph, setting new standards for report accuracy and diversity on MIMIC-CXR and Open-i datasets. Our VED model achieves F1-scores of 66.2 for CheXbert and 37.8 for RadGraph on the MIMIC-CXR dataset, and 54.7 and 45.6 , respectively, on Open-i. These outcomes represent a significant breakthrough in the RRG field. The findings and implementation of the proposed approach, aimed at enhancing diagnostic precision and radiological interpretations in clinical settings, are publicly available on GitHub to encourage further advancements in the field. [ABSTRACT FROM AUTHOR]

In this study, we primarily aim to address the exposure bias issue in long text generation intrinsic to statistical language models. We propose a sentence-level heuristic tree search algorithm, specially tailored for long text generation, to mitigate the problem by managing generated texts in a tree structure and curbing the compounding of biases. Our algorithm utilizes two pre-trained language models, an auto-regressive model for generating new sentences and an auto-encoder model for evaluating sentence quality. These models work in tandem to perform four critical operations: expanding the text tree with new sentences, evaluating the quality of the additions, sampling potential unfinished text fragments for further generation, and pruning leaf nodes deemed unpromising. This iterative process continues until a pre-defined number of [EOS] tokens are produced, at which point we select the highest-scoring completed text as our final output. Moreover, we pioneer two novel token-level decoding techniques—nucleus sampling with temperature and diverse beam search with sampling. These methods, integrated with our sentence-level search algorithm, aim to improve the consistency and diversity of text generation. Experimental results, both automated measures (including Jaccard similarity, Word2vec similarity, and unique word ratio) and human evaluations (assessing consistency, fluency, and rhetorical skills), conclusively demonstrate that our approach considerably enhances the quality of machine-generated long-form text. Through this research, we aim to inspire further innovations in sentence-level search-based text generation algorithms. [ABSTRACT FROM AUTHOR]

Text style transfer aims to rephrase a sentence to match the desired style while retaining the original content. As a controllable text generation task, mainstream approaches use content-independent style embedding as control variables to guide stylistic generation. Nonetheless, stylistic properties are context-sensitive even under the same style. For example, “delicious” and “helpful” convey positive sentiments, although they are more likely to describe food and people, respectively. Therefore, desired style signals must vary with the content. To this end, we propose a memory-enhanced transfer method, which learns fine-grained style representation concerning content to assist transfer. Rather than employing static style embedding or latent variables, our method abstracts linguistic characteristics from training corpora and memorizes subdivided content with the corresponding style representations. The style signal is dynamically retrieved from memory using the content as a query, providing a more expressive and flexible latent style space. To address the imbalance between quantity and quality in different content, we further introduce a calibration method to augment memory construction by modeling the relationship between candidate styles. Experimental results obtained using three benchmark datasets confirm the superior performance of our model compared to competitive approaches. The evaluation metrics and case study also indicate that our model can generate diverse stylistic phrases matching context. [ABSTRACT FROM AUTHOR]

Diffusion models have achieved tremendous success in modeling continuous data modalities, such as images, audio, and video, yet their application in discrete data domains (e.g., natural language) has been limited. Existing methods primarily represent discrete text in a continuous diffusion space, incurring significant computational overhead during training and resulting in slow sampling speeds. This paper introduces LaDiffuSeq, a latent diffusion-based text generation model incorporating an encoder–decoder structure. Specifically, it first employs a pretrained encoder to map sequences composed of attributes and corresponding text into a low-dimensional latent vector space. Then, without the guidance of a classifier, it performs the diffusion process for the sequence's corresponding latent space. Finally, a pretrained decoder is used to decode the newly generated latent vectors, producing target texts that are relevant to themes and possess multiple emotional granularities. Compared to the benchmark model, DiffuSeq, this model achieves BERTScore improvements of 0.105 and 0.009 on two public real-world datasets (ChnSentiCorp and a debate dataset), respectively; perplexity falls by 3.333 and 4.562; and it effectively quadruples the text generation sampling speed. [ABSTRACT FROM AUTHOR]