Search

Large language models (LLMs) exhibit remarkable proficiency in text generation. However, their logical reasoning capabilities require enhancement. Major strides have been achieved in reasoning techniques for LLM, such as Few-shot, Zero-shot, and Chain-of-Thought (CoT). Nevertheless, these techniques have shortcomings, particularly in addressing the representativeness heuristic (RH) phenomenon. RH is a cognitive bias that occurs when a person judges the probability of an event or the likelihood that an object belongs to a particular category based on how well it matches the prototype or stereotype of that category. In this study, we investigated the pervasive issue of RH errors in LLMs. This research surpasses the constraints of previous studies by analyzing various RH scenarios that they did not cover and by directly constructing and testing the corresponding datasets. Moreover, a novel prompt called zero-shot-RH is proposed to augment the reasoning ability of LLMs, mitigate RH errors, and thus bolster logical reasoning. This approach seeks to enable LLMs to comprehend the given information better and reduce the biases stemming from RH errors. The prompt zero-shot-RH achieved an average accuracy higher than zero-shot-CoT by 0.145 and 0.277 in the tasks of correct reasoning and correct reasonings by sex, respectively, without relying on RH. The outcomes of this research endeavor are a deeper understanding of RH errors in LLMs and novel strategies to mitigate these biases, thereby advancing the domain of logical reasoning within LLMs.

In the dynamic landscape of contemporary education, the evolution of teaching strategies such as blended learning and flipped classrooms has highlighted the need for efficient and effective generation of multiple-choice questions (MCQs). To address this, we introduce MCQGen, a novel generative artificial intelligence framework designed for the automated creation of MCQs. MCQGen uniquely integrates a large language model (LLM) with retrieval-augmented generation and advanced prompt engineering techniques, drawing from an extensive external knowledge base. This integration significantly enhances the ability of the LLM to produce educationally relevant questions that align with both the goals of educators and the diverse learning needs of students. The framework employs innovative prompt engineering, combining chain-of-thought and self-refine prompting techniques, to enhance the performance of the LLM. This process leads to the generation of questions that are not only contextually relevant and challenging but also reflective of common student misconceptions, contributing effectively to personalized learning experiences and enhancing student engagement and understanding. Our extensive evaluations showcase the effectiveness of MCQGen in producing high-quality MCQs for various educational needs and learning styles. The framework demonstrates its potential to significantly reduce the time and expertise required for MCQ creation, marking its practical utility in modern education. In essence, MCQGen offers an innovative and robust solution for the automated generation of MCQs, enhancing personalized learning in the digital era.

The challenge of balancing safety and utility in Large Language Models (LLMs) requires novel solutions that go beyond conventional methods of pre- and post-processing, red-teaming, and feedback fine-tuning. In response to this, we introduce GalaxyGPT, a framework that synergizes safety moderation services of Internet vendors with LLMs to enhance safety performance. This necessity arises from the growing complexity of online interactions and the imperative to ensure that LLMs operate within safe and ethical boundaries without compromising their utility. GalaxyGPT leverages advanced algorithms and a comprehensive dataset to significantly improve safety measures, achieving notable accuracy (95.8%) and F1-score (94.5%) through evaluations of our custom dataset comprising 500 single-round safety tests, 100 multi-round dialogue tests, and 200 open-source tests. These results starkly outperform the safety metrics of APIs from six vendors (average 40.5% accuracy) and LLMs without GalaxyGPT integration (73% accuracy). Additionally, we contribute to the community by releasing an open-source test set of 600 entries and a compact classification model for security tasks, specifically designed to challenge and enhance the robustness of APIs, thereby facilitating the efficient deployment and application of GalaxyGPT in diverse environments.

Traffic management systems have primarily relied on live traffic sensors for real-time traffic guidance. However, this dependence often results in uneven service delivery due to the limited scope of sensor coverage or potential sensor failures. This research introduces a novel approach to overcome this limitation by synergistically integrating a Physics-Informed Neural Network-based Traffic State Estimator (PINN-TSE) with a powerful Natural Language Processing model, GPT-4. The purpose of this integration is to provide a seamless and personalized user experience, while ensuring accurate traffic density prediction even in areas with limited data availability. The innovative PINN-TSE model was developed and tested, demonstrating a promising level of precision with a Mean Absolute Error of less than four vehicles per mile in traffic density estimation. This performance underlines the model’s ability to provide dependable traffic information, even in regions where conventional traffic sensors may be sparsely distributed or data communication is likely to be interrupted. Furthermore, the incorporation of GPT-4 enhances user interactions by understanding and responding to inquiries in a manner akin to human conversation. This not only provides precise traffic updates but also interprets user intentions for a tailored experience. The results of this research showcase an AI-integrated traffic guidance system that outperforms traditional methods in terms of traffic estimation, personalization, and reliability. While the study primarily focuses on a single road segment, the methodology shows promising potential for expansion to network-level traffic guidance, offering even greater accuracy and usability. This paves the way for a smarter and more efficient approach to traffic management in the future.

Behavior-driven development (BDD) is an Agile testing methodology fostering collaboration among developers, QA analysts, and stakeholders. In this manuscript, we propose a novel approach to enhance BDD practices using large language models (LLMs) to automate acceptance test generation. Our study uses zero and few-shot prompts to evaluate LLMs such as GPT-3.5, GPT-4, Llama-2-13B, and PaLM-2. The paper presents a detailed methodology that includes the dataset, prompt techniques, LLMs, and the evaluation process. The results demonstrate that GPT-3.5 and GPT-4 generate error-free BDD acceptance tests with better performance. The few-shot prompt technique highlights its ability to provide higher accuracy by incorporating examples for in-context learning. Furthermore, the study examines syntax errors, validation accuracy, and comparative analysis of LLMs, revealing their effectiveness in enhancing BDD practices. However, our study acknowledges that there are limitations to the proposed approach. We emphasize that this approach can support collaborative BDD processes and create opportunities for future research into automated BDD acceptance test generation using LLMs.

Code generation stands as a powerful technique in modern software development, improving development efficiency, reducing errors, and fostering standardization and consistency. Recently, ChatGPT has exhibited immense potential in automatic code generation. However, existing researches on code generation lack guidance for practical software development process. In this study, we utilized ChatGPT to develop a web-based code generation platform consisting of key components: User Interface, Prompt Builder, and Backend Service. Specifically, Prompt Builder dynamically generated comprehensive prompts to enhance model generation performance. We conducted experiments on 2 datasets to evaluate the performance of code generation in our approach. through 8 widely used metrics. The results demonstrate that (1) our Prompt Builder is effective, resulting in a 65.06% improvement in the exact match (EM), a 38.45% improvement in Bilingual Evaluation Understudy (BLEU), a 15.70% improvement in CodeBLEU, and a 50.64% improvement in Pass@1. (2) In real development scenarios, 98.5% of test cases can be validated through manual validation, highlighting the genuine assistance provided by the ChatGPT-based code generation approach.

Natural language processing (NLP) models find extensive applications but face vulnerabilities against adversarial inputs. Traditional defenses lean heavily on supervised detection techniques, which makes them vulnerable to issues arising from training data quality, inherent biases, noise, or adversarial inputs. This study observed common compromises in sentence fluency during aggression. On this basis, the Zero Sample Defender (ZDDR) is introduced for adversarial sample detection and recovery without relying on prior knowledge. ZDDR combines the log probability calculated by the model and the syntactic normative score of a large language model (LLM) to detect adversarial examples. Furthermore, using strategic prompts, ZDDR guides LLM in rephrasing adversarial content, maintaining clarity, structure, and meaning, thereby restoring the sentence from the attack. Benchmarking reveals a 9% improvement in area under receiver operating characteristic curve (AUROC) for adversarial detection over existing techniques. Post-restoration, model classification efficacy surges by 45% compared to the offensive inputs, setting new performance standards against other restoration techniques.

In today’s world, the development of social network services (SNS) like Facebook and Instagram has enabled consumers to acquire information about products through various channels. The acquisition of diverse information has led to a diversification in consumer preferences and requirements. As consumer preferences diversify and online channels expand, there is an increasing need for companies to provide personalized marketing. Among the means of personalized marketing, personalized marketing messages are a key tool that can enhance customer engagement. However, a limitation of personalized marketing message services is the cost issue associated with manually writing individual marketing messages for personalization. To solve this problem, when developing automated technology for personalized marketing messages, there were concerns about the complexity of model development and the quality of messages generated automatically. In this study, we propose the Persuasive Message Intelligence (PMI) service, which utilizes the recently prominent Large Language Model for automated individual personalized marketing messages. PMI generates marketing messages through prompt engineering based on the theory of persuasion in marketing and prior research on AI-generated messages, and validates the elements of prompts through surveys. The trial and error of researchers presented in this study, along with the know-how and rules of prompt engineering, will serve as guidelines for those who wish to develop services through prompts in the future.

Blockchain technology has revolutionized the financial landscape, witnessing widespread adoption of cryptocurrencies due to their decentralized and transparent nature. As sentiments expressed on social media platforms wield substantial influence over cryptocurrency market dynamics, sentiment analysis has emerged as a crucial tool for gauging public opinion and predicting market trends. This paper explores fine-tuning techniques for large language models to enhance sentiment analysis performance. Experimental results demonstrate a significant average zero-shot performance gain of 40% on unseen tasks after fine-tuning, highlighting its potential. Additionally, the impact of instruction-based fine-tuning on models of varying scales is examined, revealing that larger models benefit from instruction tuning, achieving the highest average accuracy score of 75.16%. In contrast, smaller-scale models may experience reduced generalization due to complete model capacity utilization. To gain deeper insight into instruction effectiveness, the paper presents experimental investigations under different instruction tuning setups. Results show the model achieves an average accuracy score of 72.38% for short and simple instructions, outperforming long and complex instructions by over 12%. Finally, the paper explores the relationship between fine-tuning corpus size and model performance, identifying an optimal corpus size of 6,000 data points for the highest performance across different models. Microsoft’s MiniLM, a distilled version of BERT, excels in efficient data use and performance optimization, while Google’s FLAN-T5 demonstrates consistent and reliable performance across diverse datasets.