Search

Without writing a single line of code by a human, an example Monte Carlo simulation-based application for stochastic dependence modeling with copulas is developed through pair programming involving a human partner and a large language model (LLM) fine-tuned for conversations. This process encompasses interacting with ChatGPT using both natural language and mathematical formalism. Under the careful supervision of a human expert, this interaction facilitated the creation of functioning code in MATLAB, Python, and R. The code performs a variety of tasks including sampling from a given copula model, evaluating the model's density, conducting maximum likelihood estimation, optimizing for parallel computing on CPUs and GPUs, and visualizing the computed results. In contrast to other emerging studies that assess the accuracy of LLMs like ChatGPT on tasks from a selected area, this work rather investigates ways how to achieve a successful solution of a standard statistical task in a collaboration of a human expert and artificial intelligence (AI). Particularly, through careful prompt engineering, we separate successful solutions generated by ChatGPT from unsuccessful ones, resulting in a comprehensive list of related pros and cons. It is demonstrated that if the typical pitfalls are avoided, we can substantially benefit from collaborating with an AI partner. For example, we show that if ChatGPT is not able to provide a correct solution due to a lack of or incorrect knowledge, the human-expert can feed it with the correct knowledge, e.g., in the form of mathematical theorems and formulas, and make it to apply the gained knowledge in order to provide a correct solution. Such ability presents an attractive opportunity to achieve a programmed solution even for users with rather limited knowledge of programming techniques. [ABSTRACT FROM AUTHOR]

Large Language Models (LLMs have the potential to revolutionize clinical medicine by enhancing healthcare access, diagnosis, surgical planning, and education. However, their utilization requires careful, prompt engineering to mitigate challenges like hallucinations and biases. Proper utilization of LLMs involves understanding foundational concepts such as tokenization, embeddings, and attention mechanisms, alongside strategic prompting techniques to ensure accurate outputs. For innovative healthcare solutions, it is essential to maintain ongoing collaboration between AI technology and medical professionals. Ethical considerations, including data security and bias mitigation, are critical to their application. By leveraging LLMs as supplementary resources in research and education, we can enhance learning and support knowledge-based inquiries, ultimately advancing the quality and accessibility of medical care. Continued research and development are necessary to fully realize the potential of LLMs in transforming healthcare. [ABSTRACT FROM AUTHOR]

Background Chat Generative Pre-trained Transformer (ChatGPT) and other ChatBots have emerged as tools for interacting with information in manners resembling natural human speech. Consequently, the technology is used across various disciplines, including business, education, and even in biomedical sciences. There is a need to better understand how ChatGPT can be used to advance gerontology research. Therefore, we evaluated ChatGPT responses to questions on specific topics in gerontology research, and brainstormed recommendations for its use in the field. Methods We conducted semistructured brainstorming sessions to identify uses of ChatGPT in gerontology research. We divided a team of multidisciplinary researchers into 4 topical groups: (a) gero-clinical science, (b) basic geroscience, (c) informatics as it relates to electronic health records, and (d) gero-technology. Each group prompted ChatGPT on a theory-, methods-, and interpretation-based question and rated responses for accuracy and completeness based on standardized scales. Results ChatGPT responses were rated by all groups as generally accurate. However, the completeness of responses was rated lower, except by members of the informatics group, who rated responses as highly comprehensive. Conclusions ChatGPT accurately depicts some major concepts in gerontological research. However, researchers have an important role in critically appraising the completeness of its responses. Having a single generalized resource like ChatGPT may help summarize the preponderance of evidence in the field to identify gaps in knowledge and promote cross-disciplinary collaboration. [ABSTRACT FROM AUTHOR]

Since ChatGPT exploded onto the scene in October 2022, the media has been breathless with their discussion about how it is going to impact many aspects of life and work. In order to bring this new reality of generative AI into the Introduction to Management Information Systems classroom at both the undergraduate and graduate levels, we have developed two short cases that can be completed in or out of the classroom. These cases allow students to use generative AI for an interesting and useful purpose and develop prompts to deliver the output. Students can begin to learn what generative AI can do for them, and how it can shape their future careers. We tested the perceived effectiveness of the project on students using a pre/post survey. The results are analyzed using a paired t-test and demonstrate that students increased their understanding of generative AI and prompt engineering. [ABSTRACT FROM AUTHOR]

Importance The study highlights the potential of large language models, specifically GPT-3.5 and GPT-4, in processing complex clinical data and extracting meaningful information with minimal training data. By developing and refining prompt-based strategies, we can significantly enhance the models' performance, making them viable tools for clinical NER tasks and possibly reducing the reliance on extensive annotated datasets. Objectives This study quantifies the capabilities of GPT-3.5 and GPT-4 for clinical named entity recognition (NER) tasks and proposes task-specific prompts to improve their performance. Materials and Methods We evaluated these models on 2 clinical NER tasks: (1) to extract medical problems, treatments, and tests from clinical notes in the MTSamples corpus, following the 2010 i2b2 concept extraction shared task, and (2) to identify nervous system disorder-related adverse events from safety reports in the vaccine adverse event reporting system (VAERS). To improve the GPT models' performance, we developed a clinical task-specific prompt framework that includes (1) baseline prompts with task description and format specification, (2) annotation guideline-based prompts, (3) error analysis-based instructions, and (4) annotated samples for few-shot learning. We assessed each prompt's effectiveness and compared the models to BioClinicalBERT. Results Using baseline prompts, GPT-3.5 and GPT-4 achieved relaxed F1 scores of 0.634, 0.804 for MTSamples and 0.301, 0.593 for VAERS. Additional prompt components consistently improved model performance. When all 4 components were used, GPT-3.5 and GPT-4 achieved relaxed F1 socres of 0.794, 0.861 for MTSamples and 0.676, 0.736 for VAERS, demonstrating the effectiveness of our prompt framework. Although these results trail BioClinicalBERT (F1 of 0.901 for the MTSamples dataset and 0.802 for the VAERS), it is very promising considering few training samples are needed. Discussion The study's findings suggest a promising direction in leveraging LLMs for clinical NER tasks. However, while the performance of GPT models improved with task-specific prompts, there's a need for further development and refinement. LLMs like GPT-4 show potential in achieving close performance to state-of-the-art models like BioClinicalBERT, but they still require careful prompt engineering and understanding of task-specific knowledge. The study also underscores the importance of evaluation schemas that accurately reflect the capabilities and performance of LLMs in clinical settings. Conclusion While direct application of GPT models to clinical NER tasks falls short of optimal performance, our task-specific prompt framework, incorporating medical knowledge and training samples, significantly enhances GPT models' feasibility for potential clinical applications. [ABSTRACT FROM AUTHOR]

One of the most common questions that students ask statistics advisors is 'What test should I do?' This paper explores the use of generative AI chatbots, specifically ChatGPT, as a tool to assist students, in particular those with limited experience in statistics, in selecting appropriate statistical tests for their analyses. Traditional methods, such as flowcharts and online test selectors, require at least a basic understanding of measurement scales and research design, which can be an issue for many students who have limited exposure to statistics on their courses. This research focuses on developing and refining prompts to guide ChatGPT in providing accurate and relevant statistical test recommendations. A hypothetical scenario was used to test the effectiveness of various prompts, ranging from simple, naïve questions to more sophisticated ones utilising specific prompt patterns, such as the 'context manager' and 'flipped interaction.' These patterns were selected to enhance the chatbot's responses and ensure the relevance and accuracy of the test suggestions. The findings suggest that while AI chatbots like ChatGPT can be a valuable resource for students, their effectiveness is highly dependent on the quality of the prompts used. The paper concludes with a discussion on the potential of these AI tools in educational settings, acknowledging the limitations of current technology and suggesting directions for future research and development. [ABSTRACT FROM AUTHOR]

Entity Resolution (ER) is the problem of semi-automatically determining when two entities refer to the same underlying entity, with applications ranging from healthcare to e-commerce. Traditional ER solutions required considerable manual expertise, including domain-specific feature engineering, as well as identification and curation of training data. Recently released large language models (LLMs) provide an opportunity to make ER more seamless and domain-independent. Because of LLMs' pre-trained knowledge, the matching step in ER can be made easier by just prompting. However, it is also well known that LLMs can pose risks, that the quality of their outputs can depend on how prompts are engineered, and that the cost of using LLMs can be significant. Unfortunately, a systematic experimental study on the effects of different prompting methods and their respective cost for solving domain-specific entity matching using LLMs, like ChatGPT, has been lacking thus far. This paper aims to address this gap by conducting such a study. We consider some relatively simple and cost-efficient ER prompt engineering methods and apply them to perform product matching on two real-world datasets widely used in the community. We select two well-known e-commerce datasets and provide extensive experimental results to show that an LLM like GPT3.5 is viable for high-performing product matching and, interestingly, that more complicated and detailed (and hence, expensive) prompting methods do not necessarily outperform simpler approaches. We provide brief discussions on qualitative and error analysis, including a study of the inter-consistency of different prompting methods to determine whether they yield stable outputs. Finally, we consider some limitations of LLMs when used as a product matcher in potential real-world e-commerce applications. [ABSTRACT FROM AUTHOR]

In ethnographic research, data collected through surveys, interviews, or questionnaires in the fields of sociology and anthropology often appear in diverse forms and languages. Building a powerful database system to store and process such data, as well as making good and efficient queries, is very challenging. This paper extensively investigates modern database technology to find out what the best technologies to store these varied and heterogeneous datasets are. The study examines several database categories: traditional relational databases, the NoSQL family of key-value databases, graph databases, document databases, object-oriented databases and vector databases, crucial for the latest artificial intelligence solutions. The research proves that when it comes to field data, the NoSQL lineup is the most appropriate, especially document and graph databases. Simplicity and flexibility found in document databases and advanced ability to deal with complex queries and rich data relationships attainable with graph databases make these two types of NoSQL databases the ideal choice if a large amount of data has to be processed. Advancements in vector databases that embed custom metadata offer new possibilities for detailed analysis and retrieval. However, converting contents into vector data remains challenging, especially in regions with unique oral traditions and languages. Constructing such databases is labor-intensive and requires domain experts to define metadata and relationships, posing a significant burden for research teams with extensive data collections. To this end, this paper proposes using Generative AI (GenAI) to help in the data-transformation process, a recommendation that is supported by testing where GenAI has proven itself a strong supplement to document and graph databases. It also discusses two methods of vector database support that are currently viable, although each has drawbacks and benefits. [ABSTRACT FROM AUTHOR]

Manual inspection and remediation of guideline violations (UI design smells) is a knowledge-intensive, time-consuming, and context-related task that requires a high level of expertise. This paper proposes UISGPT, a novel end-to-end approach for automatically detecting user interface (UI) design smells and explaining each violation of specific design guidelines in natural language. To avoid hallucinations in large language models (LLMs) and achieve interpretable results, UISGPT uses few-shot learning and least-to-most prompting strategies to formalize design guidelines. To prevent the model from exceeding the input window size and for the enhancement of the logic in responses, UISGPT divides design smell detection into the following three subtasks: design guideline formalization, UI component information extraction, and guideline validation. The experimental results show that UISGPT performs effectively in automatically detecting design violations (F1 score of 0.729). In comparison to the latest LLM methods, the design smell reports generated by UISGPT have higher contextual consistency and user ratings. [ABSTRACT FROM AUTHOR]

"Are we asking the right questions?" seems cliché, but for ChatGPT, it is a pivotal tool to ensure the accuracy of responses. While ChatGPT-3.5's training on the vast database promises to revolutionize STEM education and research, this investigation shows the importance of precise communication and prompt engineering in guiding ChatGPT-3.5 toward reliable and accurate responses, particularly in chemistry. For instance, emphasizing context, clearly defining symbols, and focusing on field-specific instructions can dramatically improve its performance. Furthermore, avoiding open-ended prompts and strategically using repetition can further enhance its accuracy. The iterative prompt design, demonstrated through a series of adjustments, illustrates how seemingly minor refinements, such as substituting "least" for "lowest", profoundly impact the output. This study highlights the essential role of human oversight, including the construction of well-crafted prompts, in guarding reliable information and nurturing a productive "Human–AI" (HAI) partnership. [ABSTRACT FROM AUTHOR]

Through design and development research (DDR), we aimed to create a validated automatic question generation (AQG) system using large language models (LLMs) like ChatGPT, enhanced by prompting engineering techniques. While AQG has become increasingly integral to online learning for its efficiency in generating questions, issues such as inconsistent question quality and the absence of transparent and validated evaluation methods persist. Our research focused on creating a prompt engineering protocol tailored for AQG. This protocol underwent several iterations of refinement and validation to improve its performance. By gathering validation scores and qualitative feedback on the produced questions and the system's framework, we examined the effectiveness of the system. The study findings indicate that our combined use of LLMs and prompt engineering in AQG produces questions with statistically significant validity. Our research further illuminates academic and design considerations for AQG design in English education: (a) certain question types might not be optimal for generation via ChatGPT, (b) ChatGPT sheds light on the potential for collaborative AI-teacher efforts in question generation, especially within English education. [ABSTRACT FROM AUTHOR]

Extended reality offers unique ways to create mediated spaces that enhance and help popularize experiences across several domains, including entertainment, creativity, and culture. There are still issues that hinder the widespread adoption of the medium, such as the over-reliance on scripted sequences, generalized approaches, and curated asset production. Artificial intelligence can be used to, in part, alleviate these issues, but this comes with its own set of challenges, such as factual inaccuracy or hallucinations. We delve into prompt engineering methods for the GPT API, enhancing context understanding to enable more realistic performances in historical event recreations. Specifically, we experiment with the Great Fire of Smyrna in 1922 as our historical context, situating the AI agent in the middle of chaos as a resident that has been affected by the event. Our experiments demonstrate that refined prompt engineering techniques significantly reduce factual inaccuracies and enhance the emotional resonance of AI-generated dialogues, which can lead to more immersive and engaging XR experiences. Our experiments indicate that AI can effectively support historical recreations by providing dynamic and contextually appropriate interactions. [ABSTRACT FROM AUTHOR]

The rapid advancements in artificial intelligence, particularly generative AI and large language models, have unlocked new possibilities for revolutionizing healthcare delivery. However, harnessing the full potential of these technologies requires effective prompt engineering—designing and optimizing input prompts to guide AI systems toward generating clinically relevant and accurate outputs. Despite the importance of prompt engineering, medical education has yet to fully incorporate comprehensive training on this critical skill, leading to a knowledge gap among medical clinicians. This article addresses this educational gap by providing an overview of generative AI prompt engineering, its potential applications in primary care medicine, and best practices for its effective implementation. The role of well-crafted prompts in eliciting accurate, relevant, and valuable responses from AI models is discussed, emphasizing the need for prompts grounded in medical knowledge and aligned with evidence-based guidelines. The article explores various applications of prompt engineering in primary care, including enhancing patient–provider communication, streamlining clinical documentation, supporting medical education, and facilitating personalized care and shared decision-making. Incorporating domain-specific knowledge, engaging in iterative refinement and validation of prompts, and addressing ethical considerations and potential biases are highlighted. Embracing prompt engineering as a core competency in medical education will be crucial for successfully adopting and implementing AI technologies in primary care, ultimately leading to improved patient outcomes and enhanced healthcare delivery. [ABSTRACT FROM AUTHOR]

Background and Objectives: Integrating large language models (LLMs) such as GPT-4 Turbo into diagnostic imaging faces a significant challenge, with current misdiagnosis rates ranging from 30–50%. This study evaluates how prompt engineering and confidence thresholds can improve diagnostic accuracy in neuroradiology. Methods: We analyze 751 neuroradiology cases from the American Journal of Neuroradiology using GPT-4 Turbo with customized prompts to improve diagnostic precision. Results: Initially, GPT-4 Turbo achieved a baseline diagnostic accuracy of 55.1%. By reformatting responses to list five diagnostic candidates and applying a 90% confidence threshold, the highest precision of the diagnosis increased to 72.9%, with the candidate list providing the correct diagnosis at 85.9%, reducing the misdiagnosis rate to 14.1%. However, this threshold reduced the number of cases that responded. Conclusions: Strategic prompt engineering and high confidence thresholds significantly reduce misdiagnoses and improve the precision of the LLM diagnostic in neuroradiology. More research is needed to optimize these approaches for broader clinical implementation, balancing accuracy and utility. [ABSTRACT FROM AUTHOR]

Introduction: Constructing an accurate and comprehensive knowledge graph of specific diseases is critical for practical clinical disease diagnosis and treatment, reasoning and decision support, rehabilitation, and health management. For knowledge graph construction tasks (such as named entity recognition, relation extraction), classical BERT-based methods require a large amount of training data to ensure model performance. However, real-world medical annotation data, especially disease-specific annotation samples, are very limited. In addition, existing models do not perform well in recognizing out-of-distribution entities and relations that are not seen in the training phase. Method: In this study, we present a novel and practical pipeline for constructing a heart failure knowledge graph using large language models and medical expert refinement. We apply prompt engineering to the three phases of schema design: schema design, information extraction, and knowledge completion. The best performance is achieved by designing task-specific prompt templates combined with the TwoStepChat approach. Results: Experiments on two datasets show that the TwoStepChat method outperforms the Vanillia prompt and outperforms the fine-tuned BERT-based baselines. Moreover, our method saves 65% of the time compared to manual annotation and is better suited to extract the out-of-distribution information in the real world. [ABSTRACT FROM AUTHOR]

Inferring urban functions using street view images (SVIs) has gained tremendous momentum. The recent prosperity of large-scale vision-language pretrained models sheds light on addressing some long-standing challenges in this regard, for example, heavy reliance on labeled samples and computing resources. In this paper, we present a novel prompting framework for enabling the pretrained vision-language model CLIP to effectively infer fine-grained urban functions with SVIs in a zero-shot manner, that is, without labeled samples and model training. The prompting framework UrbanCLIP comprises an urban taxonomy and several urban function prompt templates, in order to (1) bridge the abstract urban function categories and concrete urban object types that can be readily understood by CLIP, and (2) mitigate the interference in SVIs, for example, street-side trees and vehicles. We conduct extensive experiments to verify the effectiveness of UrbanCLIP. The results indicate that the zero-shot UrbanCLIP largely surpasses several competitive supervised baselines, e.g. a fine-tuned ResNet, and its advantages become more prominent in cross-city transfer tests. In addition, UrbanCLIP's zero-shot performance is considerably better than the vanilla CLIP. Overall, UrbanCLIP is a simple yet effective framework for urban function inference, and showcases the potential of foundation models for geospatial applications. [ABSTRACT FROM AUTHOR]

This paper investigates the prospective application of large language models (LLMs) in Open-Source Intelligence (OSINT), with particular emphasis on integrating information acquisition processes and the growing significance of prompt engineering for analysts. The research includes a thorough literature review, a proposed information model incorporating LLMs, and underscores the functional utilization of artificial intelligence in OSINT and the associated challenges, primarily the educational needs of security forces. In this context, the study outlines prompt engineering approaches for various OSINT tasks as a crucial skill, necessitating a deep understanding of LLMs to produce validated intelligence. Relevant general prompts are demonstrated. Additionally, the paper discusses the need for updated training in critical thinking, search techniques, and prompt engineering for intelligence professionals. The findings indicate a significant evolution in OSINT methodologies, highlighting the need for ongoing research and education to fully harness AI's potential in intelligence gathering. [ABSTRACT FROM AUTHOR]

The body of ecological literature, which informs much of our knowledge of the global loss of biodiversity, has been experiencing rapid growth in recent decades. The increasing difficulty of synthesising this literature manually has simultaneously resulted in a growing demand for automated text mining methods. Within the domain of deep learning, large language models (LLMs) have been the subject of considerable attention in recent years due to great leaps in progress and a wide range of potential applications; however, quantitative investigation into their potential in ecology has so far been lacking.In this work, we analyse the ability of GPT‐4 to extract information about invertebrate pests and pest controllers from abstracts of articles on biological pest control, using a bespoke, zero‐shot prompt.Our results show that the performance of GPT‐4 is highly competitive with other state‐of‐the‐art tools used for taxonomic named entity recognition and geographic location extraction tasks. On a held‐out test set, we show that species and geographic locations are extracted with F1‐scores of 99.8% and 95.3%, respectively, and highlight that the model can effectively distinguish between ecological roles of interest such as predators, parasitoids and pests. Moreover, we demonstrate the model's ability to effectively extract and predict taxonomic information across various taxonomic ranks. However, we do report a small number of cases of fabricated information (confabulations).Due to a lack of specialised, pre‐trained ecological language models, general‐purpose LLMs may provide a promising way forward in ecology. Combined with tailored prompt engineering, such models can be employed for a wide range of text mining tasks in ecology, with the potential to greatly reduce time spent on manual screening and labelling of the literature. [ABSTRACT FROM AUTHOR]

This study investigates the efficacy of advanced large language models, specifically GPT-4o, Claude-3.5 Sonnet, and GPT-3.5 Turbo, in detecting software vulnerabilities. Our experiment utilized vulnerable and secure code samples from the NIST Software Assurance Reference Dataset (SARD), focusing on C++, Java, and Python. We employed three distinct prompting techniques as follows: Concise, Tip Setting, and Step-by-Step. The results demonstrate that GPT-4o and Claude-3.5 Sonnet significantly outperform GPT-3.5 Turbo in vulnerability detection. GPT-4o showed the highest improvement with the Step-by-Step prompt, achieving an F1 score of 0.9072. Claude-3.5 Sonnet exhibited consistent high performance across all prompt types, with its Step-by-Step prompt yielding the best overall results (F1 score: 0.8933, AUC: 0.74). In contrast, GPT-3.5 Turbo showed minimal performance changes across prompts, with the Tip Setting prompt performing best (AUC: 0.65, F1 score: 0.6772), yet significantly lower than the other models. Our findings highlight the potential of advanced models in enhancing software security and underscore the importance of prompt engineering in optimizing their performance. [ABSTRACT FROM AUTHOR]

Since the advent of GPT, large language models (LLMs) have brought about revolutionary advancements in all walks of life. As a superior natural language processing (NLP) technology, LLMs have consistently achieved state-of-the-art performance in numerous areas. However, LLMs are considered to be general-purpose models for NLP tasks, which may encounter challenges when applied to complex tasks in specialized fields such as optical networks. In this study, we propose a framework of LLM-empowered optical networks, facilitating intelligent control of the physical layer and efficient interaction with the application layer through an LLM-driven agent (AI-Agent) deployed in the control layer. The AI-Agent can leverage external tools and extract domain knowledge from a comprehensive resource library specifically established for optical networks. This is achieved through user input and well-crafted prompts, enabling the generation of control instructions and result representations for autonomous operation and maintenance in optical networks. To improve LLM's capability in professional fields and stimulate its potential on complex tasks, the details of performing prompt engineering, establishing domain knowledge library, and implementing complex tasks are illustrated in this study. Moreover, the proposed framework is verified on two typical tasks: network alarm analysis and network performance optimization. The good response accuracies and semantic similarities of 2400 test situations exhibit the great potential of LLM in optical networks. [ABSTRACT FROM AUTHOR]

“Are we asking the right questions?” seems cliché, but for ChatGPT, it is a pivotal tool to ensure the accuracy of responses. While ChatGPT-3.5’s training on the vast database promises to revolutionize STEM education and research, this investigation shows the importance of precise communication and prompt engineering in guiding ChatGPT-3.5 toward reliable and accurate responses, particularly in chemistry. For instance, emphasizing context, clearly defining symbols, and focusing on field-specific instructions can dramatically improve its performance. Furthermore, avoiding open-ended prompts and strategically using repetition can further enhance its accuracy. The iterative prompt design, demonstrated through a series of adjustments, illustrates how seemingly minor refinements, such as substituting “least” for “lowest”, profoundly impact the output. This study highlights the essential role of human oversight, including the construction of well-crafted prompts, in guarding reliable information and nurturing a productive “Human–AI” (HAI) partnership.

Abstract Entity Resolution (ER) is the problem of semi-automatically determining when two entities refer to the same underlying entity, with applications ranging from healthcare to e-commerce. Traditional ER solutions required considerable manual expertise, including domain-specific feature engineering, as well as identification and curation of training data. Recently released large language models (LLMs) provide an opportunity to make ER more seamless and domain-independent. Because of LLMs’ pre-trained knowledge, the matching step in ER can be made easier by just prompting. However, it is also well known that LLMs can pose risks, that the quality of their outputs can depend on how prompts are engineered, and that the cost of using LLMs can be significant. Unfortunately, a systematic experimental study on the effects of different prompting methods and their respective cost for solving domain-specific entity matching using LLMs, like ChatGPT, has been lacking thus far. This paper aims to address this gap by conducting such a study. We consider some relatively simple and cost-efficient ER prompt engineering methods and apply them to perform product matching on two real-world datasets widely used in the community. We select two well-known e-commerce datasets and provide extensive experimental results to show that an LLM like GPT3.5 is viable for high-performing product matching and, interestingly, that more complicated and detailed (and hence, expensive) prompting methods do not necessarily outperform simpler approaches. We provide brief discussions on qualitative and error analysis, including a study of the inter-consistency of different prompting methods to determine whether they yield stable outputs. Finally, we consider some limitations of LLMs when used as a product matcher in potential real-world e-commerce applications.

Abstract This study was designed to assess how different prompt engineering techniques, specifically direct prompts, Chain of Thought (CoT), and a modified CoT approach, influence the ability of GPT-3.5 to answer clinical and calculation-based medical questions, particularly those styled like the USMLE Step 1 exams. To achieve this, we analyzed the responses of GPT-3.5 to two distinct sets of questions: a batch of 1000 questions generated by GPT-4, and another set comprising 95 real USMLE Step 1 questions. These questions spanned a range of medical calculations and clinical scenarios across various fields and difficulty levels. Our analysis revealed that there were no significant differences in the accuracy of GPT-3.5's responses when using direct prompts, CoT, or modified CoT methods. For instance, in the USMLE sample, the success rates were 61.7% for direct prompts, 62.8% for CoT, and 57.4% for modified CoT, with a p-value of 0.734. Similar trends were observed in the responses to GPT-4 generated questions, both clinical and calculation-based, with p-values above 0.05 indicating no significant difference between the prompt types. The conclusion drawn from this study is that the use of CoT prompt engineering does not significantly alter GPT-3.5's effectiveness in handling medical calculations or clinical scenario questions styled like those in USMLE exams. This finding is crucial as it suggests that performance of ChatGPT remains consistent regardless of whether a CoT technique is used instead of direct prompts. This consistency could be instrumental in simplifying the integration of AI tools like ChatGPT into medical education, enabling healthcare professionals to utilize these tools with ease, without the necessity for complex prompt engineering.

Abstract The body of ecological literature, which informs much of our knowledge of the global loss of biodiversity, has been experiencing rapid growth in recent decades. The increasing difficulty of synthesising this literature manually has simultaneously resulted in a growing demand for automated text mining methods. Within the domain of deep learning, large language models (LLMs) have been the subject of considerable attention in recent years due to great leaps in progress and a wide range of potential applications; however, quantitative investigation into their potential in ecology has so far been lacking. In this work, we analyse the ability of GPT‐4 to extract information about invertebrate pests and pest controllers from abstracts of articles on biological pest control, using a bespoke, zero‐shot prompt. Our results show that the performance of GPT‐4 is highly competitive with other state‐of‐the‐art tools used for taxonomic named entity recognition and geographic location extraction tasks. On a held‐out test set, we show that species and geographic locations are extracted with F1‐scores of 99.8% and 95.3%, respectively, and highlight that the model can effectively distinguish between ecological roles of interest such as predators, parasitoids and pests. Moreover, we demonstrate the model's ability to effectively extract and predict taxonomic information across various taxonomic ranks. However, we do report a small number of cases of fabricated information (confabulations). Due to a lack of specialised, pre‐trained ecological language models, general‐purpose LLMs may provide a promising way forward in ecology. Combined with tailored prompt engineering, such models can be employed for a wide range of text mining tasks in ecology, with the potential to greatly reduce time spent on manual screening and labelling of the literature.